The Unstructured UI features a no-code user interface for transforming your unstructured data into data that is ready for Retrieval Augmented Generation (RAG).
The Unstructured Workflow Endpoint, part of the Unstructured API, enables a full range of partitioning, chunking, embedding, and enrichment options for your files and data. It is designed to batch-process files and data in remote locations; send processed results to various storage, databases, and vector stores; and use the latest and highest-performing models on the market today. It has built-in logic to deliver the highest quality results at the lowest cost.
This page provides an overview of the Unstructured Workflow Endpoint. This endpoint enables Unstructured UI automation usage scenarios as well as for documentation, reporting, and recovery needs.
Choose one of the following options to get started with the Unstructured Workflow Endpoint:
curl
or Postman.This quickstart uses the Unstructured Python SDK to call the Unstructured Workflow Endpoint to get your data RAG-ready. The Python code for this quickstart is in a remote hosted Google Colab notebook. Data is processed on Unstructured-hosted compute resources.
The requirements are as follows:
Unstructured-IO/unstructured-ingest
repository in GitHub.Sign up, sign in, and get your API key
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
Create and set up the S3 bucket
This quickstart uses an Amazon S3 bucket as both the source location and the destination location. (You can use other source and destination types that are supported by Unstructured. If you use a different source or destination type, or if you use a different S3 bucket for the destination location, you will need to modify the quickstart notebook accordingly.)
Inside of the S3 bucket, a folder named input
represents the
source location. This is where your files to be processed will be stored.
The S3 URI to the source location will be s3://<your-bucket-name>/input
.
Inside of the same S3 bucket, a folder inside named output
represents the destination location. This
is where Unstructured will put the processed data.
The S3 URI to the destination location will be s3://<your-bucket-name>/output
.
Learn how to create an S3 bucket and set it up for Unstructured. (Do not run the Python SDK code or REST commands at the end of those setup instructions.)
Run the quickstart notebook
After your S3 bucket is created and set up, follow the instructions in this quickstart notebook.
View the processed data
After you run the quickstart notebook, go to your destination location to view the processed data.
Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured Workflow Endpoint to create connectors in the Unstructured UI.
Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured Workflow Endpoint to create workflows and jobs in the Unstructured UI.
Open a related notebook that covers many of the concepts that are shown in the preceding videos.
The Unstructured Python SDK, beginning with version 0.30.6, allows you to call the Unstructured Workflow Endpoint through standard Python code.
To install the Unstructured Python SDK, run the following command from within your Python virtual environment:
If you already have the Unstructured Python SDK installed, upgrade to at least version 0.30.6 by running the following command instead:
The Unstructured Python SDK code examples, shown later on this page and on related pages, use the following environment variable, which you can set as follows:
This environment variable enables you to more easily run the following Unstructured Python SDK examples and help prevent you from storing scripts that contain sensitive API keys in public source code repositories.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
Calls made by the Unstructured Python SDK’s unstructured_client
functions for creating, listing, updating,
and deleting connectors, workflows, and jobs in the Unstructured UI all use the Unstructured Workflow Endpoint URL.
This URL was provided to you when your Unstructured account was created.
If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The default URL for the Unstructured Worfklow Endpoint is https://platform.unstructuredapp.io/api/v1
.
However, you should always use the URL that was provided to you when your Unstructured account was created.
To specify an API URL in your code, set the server_url
parameter in the UnstructuredClient
constructor to the target API URL.
The Unstructured Workflow Endpoint enables you to work with connectors, workflows, and jobs in the Unstructured UI.
For general information about these objects, see:
Skip ahead to start learning about how to use the Unstructured Python SDK to work with connectors, workflows, and jobs programmatically.
The Unstructured Workflow Endpoint is callable from a set of Representational State Transfer (REST) endpoints, which you can call through standard REST-enabled
utilities, tools, programming languages, packages, and libraries. The examples, shown later on this page and on related pages, describe how to call the Unstructured Workflow Endpoint with
curl
and Postman. You can adapt this information as needed for your preferred programming languages and libraries, for example by using the
requests
library with Python.
You can also use the Unstructured Workflow Endpoint - Swagger UI to call the REST endpoints
that are available through the default Unstructured Workflow Endpoint URL: https://platform.unstructuredapp.io
. To use the Swagger UI, you must provide your Unstructured API key with each call. To
get this API key, see the quickstart, earlier on this page.
Note that you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The following curl
examples use the following environment variables, which you can set as follows:
For the API URL, this URL was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The default URL for the Unstructured Worfklow Endpoint is https://platform.unstructuredapp.io/api/v1
.
However, you should always use the URL that was provided to you when your Unstructured account was created.
These environment variables enable you to more easily run the following curl
examples and help prevent
you from storing scripts that contain sensitive URLs and API keys in public source code repositories.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The following Postman examples use variables, which you can set as follows:
In Postman, on your workspace’s sidebar, click Environments.
Click Globals.
Create two global variables with the following settings:
UNSTRUCTURED_API_URL
default
UNSTRUCTURED_API_KEY
secret
<your-unstructured-api-key>
<your-unstructured-api-key>
Click Save.
These variables enable you to more easily run the following examples in Postman and help prevent you from storing Postman collections that contain sensitive URLs and API keys in public source code repositories.
Unstructured offers a Postman collection that you can import into Postman to make Workflow Endpoint requests through a graphical user interface.
In your workspace, click Import.
In the Paste cURL, Raw text or URL box, enter the following URL, and then press Enter
:
On the sidebar, click Collections.
Expand Unstructured REST API - Workflow Endpoint.
Select the request that you want to use.
As applicable, modify the URL as needed to specify any required resource IDs for the request.
On the Headers tab, next to unstructured-api-key
, enter your Unstructured API key in the Value column.
As applicable, add, remove, or modify any other required headers for the request.
As applicable, on the Params tab, add, remove, or modify any required parameters for the request.
As applicable, on the Body tab, add, remove, or modify the required request body for the request.
Click Send.
To save the response, in the response area, click the ellipses, and then click Save response to file.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The Unstructured Workflow Endpoint enables you to work with connectors, workflows, and jobs in the Unstructured UI.
For general information about these objects, see:
Skip ahead to start learning about how to use the REST endpoints to work with connectors, workflows, and jobs programmatically.
The following Unstructured SDKs, tools, and libraries do not work with the Unstructured Workflow Endpoint:
The following Unstructured API URL is also not supported: https://api.unstructuredapp.io/general/v0/general
(the default Unstructured Partition Endpoint URL).
You can list, get, create, update, delete, and test source connectors. You can also list, get, create, update, delete, and test destination connectors.
For general information, see Connectors.
To list source connectors, use the UnstructuredClient
object’s sources.list_sources
function (for the Python SDK) or
the GET
method to call the /sources
endpoint (for curl
or Postman).
To filter the list of source connectors, use the ListSourcesRequest
object’s source_type
parameter (for the Python SDK)
or the query parameter source_type=<type>
(for curl
or Postman),
replacing <type>
with the source connector type’s unique ID
(for example, for the Amazon S3 source connector type, S3
for the Python SDK or s3
for curl
or Postman).
To get this ID, see Sources.
Python SDK
Python SDK (async)
curl
To filter the list of source connectors:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of source connectors, on the Params tab, enter the following query parameter:
source_type
, Value: <type>
Click Send.
To get information about a source connector, use the UnstructuredClient
object’s sources.get_source
function (for the Python SDK) or
the GET
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a source connector, use the UnstructuredClient
object’s sources.create_source
function (for the Python SDK) or
the POST
method to call the /sources
endpoint (for curl
or Postman).
In the CreateSourceConnector
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the connector. For the specific settings to include, which differ by connector, see
Sources.
For the Python SDK, replace <type>
with the source connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Sources.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To update information about a source connector, use the UnstructuredClient
object’s sources.update_source
function (for the Python SDK) or
the PUT
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
In the UpdateSourceConnector
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the connector. For the specific settings to include, which differ by connector, see
Sources.
For the Python SDK, replace <type>
with the source connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Sources.
You must specify all of the settings for the connector, even for settings that are not changing.
You can change any of the connector’s settings except for its name
and type
.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To delete a source connector, use the UnstructuredClient
object’s sources.delete_source
function (for the Python SDK) or
the DELETE
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To test a source connector, use the POST
method to call the /sources/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see List source connectors.
The Python SDK does not support testing source connectors.
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get information about the most recent connector check for a source connector, use the GET
method to call the /sources/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see List source connectors.
The Python SDK does not support getting information about the most recent connector check for a source connector.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To list destination connectors, use the UnstructuredClient
object’s destinations.list_destinations
function (for the Python SDK) or
the GET
method to call the /destinations
endpoint (for curl
or Postman).
To filter the list of destination connectors, use the ListDestinationsRequest
object’s destination_type
parameter (for the Python SDK) or
the query parameter destination_type=<type>
(for curl
or Postman),
replacing <type>
with the destination connector type’s unique ID
(for example, for the Amazon S3 source connector type, S3
for the Python SDK or s3
for curl
or Postman).
To get this ID, see Destinations.
Python SDK
Python SDK (async)
curl
To filter the list of destination connectors:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of destination connectors, on the Params tab, enter the following query parameter:
destination_type
, Value: <type>
Click Send.
To get information about a destination connector, use the UnstructuredClient
object’s destinations.get_destination
function (for the Python SDK) or
the GET
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a destination connectors, use the UnstructuredClient
object’s destinations.create_destination
function (for the Python SDK) or
the POST
method to call the /destinations
endpoint (for curl
or Postman).
In the CreateDestinationConnector
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the connector. For the specific settings to include, which differ by connector, see
Destinations.
For the Python SDK, replace <type>
with the destination connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Destinations.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To update information about a destination connector, use the UnstructuredClient
object’s destinations.update_destination
function (for the Python SDK) or
the PUT
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
In the UpdateDestinationConnector
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the connector. For the specific settings to include, which differ by connector, see
Destinations.
You must specify all of the settings for the connector, even for settings that are not changing.
You can change any of the connector’s settings except for its name
and type
.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To delete a destination connector, use the UnstructuredClient
object’s destinations.delete_destination
function (for the Python SDK) or
the DELETE
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To test a destination connector, use the POST
method to call the /destinations/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see
List destination connectors.
The Python SDK does not support testing destination connectors.
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get information about the most recent connector check for a destination connector, use the GET
method to call the /destinations/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see
List destination connectors.
The Python SDK does not support getting information about the most recent connector check for a destination connector.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
You can list, get, create, run, update, and delete workflows.
For general information, see Workflows.
To list workflows, use the UnstructuredClient
object’s workflows.list_workflows
function (for the Python SDK) or
the GET
method to call the /workflows
endpoint (for curl
or Postman).
To filter the list of workflows, use one or more of the following ListWorkflowsRequest
parameters (for the Python SDK) or
query parameters (for curl
or Postman):
source_id=<connector-id>
, replacing <connector-id>
with the source connector’s unique ID.
To get this ID, see List source connectors.destination_id=<connector-id>
, replacing <connector-id>
with the destination connector’s unique ID.
To get this ID, see List destination connectors.status=WorkflowState.<status>
(for the Python SDK) or status=<status>
(for curl
or Postman), replacing <status>
with one of the following workflow statuses: ACTIVE
or INACTIVE
(for the Python SDK) or active
or inactive
(for curl
or Postman).You can specify multiple query parameters, for example ?source_id=<connector-id>&status=<status>
.
Python SDK
Python SDK (async)
curl
To filter the list by source connector ID:
To filter the list by destination connector ID:
To filter the list by workflow status:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of workflows, on the Params tab, enter one or more of the following query parameter:
source_id
, Value: <connector-id>
destination_id
, Value: <connector-id>
status
, Value: <status>
Click Send.
To get information about a workflow, use the UnstructuredClient
object’s workflows.get_workflow
function (for the Python SDK) or
the GET
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a workflow, use the UnstructuredClient
object’s workflows.create_workflow
function (for the Python SDK) or
the POST
method to call the /workflows
endpoint (for curl
or Postman).
In the CreateWorkflow
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the workflow. For the specific settings to include, see
Create a workflow.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the workflow.
Click Send.
To run a workflow manually, use the UnstructuredClient
object’s workflows.run_workflow
function (for the Python SDK) or
the POST
method to call the /workflows/<workflow-id>/run
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK (remote source and remote destination)
If the target workflow was originally created programmatically by the Unstructured Python SDK or with a REST API client such as curl
or Postman,
and the workflow uses a local source connector, you can run the workflow only with a REST API client such as curl
or Postman,
as described later in this section.
You cannot run the workflow with the Python SDK or the Unstructured user interface (UI), even though the workflow is visible in the UI.
Python SDK (async) (remote source and remote destination)
If the target workflow was originally created programmatically by the Unstructured Python SDK or with a REST API client such as curl
or Postman,
and the workflow uses a local source connector, you can run the workflow only with a REST API client such as curl
or Postman,
as described later in this section.
You cannot run the workflow with the Python SDK or the Unstructured user interface (UI), even though the workflow is visible in the UI.
curl (remote source and remote destination)
curl (local source and local or remote destination)
In the following command, replace:
</full/path/to/local/filename.extension>
with the full path to the local file to upload.<filename.extension>
with the filename of the local file to upload.<local-file-media-type>
with the local file’s media type. For a list of available media types, such as application/pdf
, see Media Types.To upload multiple files, add additional --form
entries, one per file.
For a local destination, to access the processed files’ data, download a processed local file from the workflow’s job run.
Postman (remote source and remote destination)
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
Postman (local source and local or remote destination)
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
On the Body tab, select form-data, and specify the settings for the workflow run:
Key: input_files
, File, Value: Click the Value box, then click New file from local machine, and select the file to upload.
To upload multiple files, add additional input_files
entries after this one, one entry per additional file to upload.
Key: filename
, Text, Value: Type the name of the file that you just uploaded.
To upload multiple files, add additional filename
entries after this one, one entry per additional file to upload. Make sure the order of these
filename
entries matches the order of the input_files
entries, respectively.
Key: type
, Text, Value: <local-file-media-type>
To upload multiple files, add additional type
entries after this one, one entry per additional file to upload. Make sure the order of these
type
entries matches the order of the input_files
entries, respectively.
For a list of available media types, such as application/pdf
, see Media Types.
Click Send.
For a local destination, to access the processed files’ data, download a processed local file from the workflow’s job run.
To run a workflow on a schedule instead, specify the schedule
setting in the request body when you create or update a
workflow. See Create a workflow or Update a workflow.
To update information about a workflow, use the UnstructuredClient
object’s workflows.update_workflow
function (for the Python SDK) or
the PUT
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
In UpdateWorkflow
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the workflow. For the specific settings to include, see
Update a workflow.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the workflow.
Click Send.
To delete a workflow, use the UnstructuredClient
object’s workflows.delete_workflow
function (for the Python SDK) or
the DELETE
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
You can list, get, and cancel jobs.
A job is created automatically whenever a workflow runs on a schedule; see Create a workflow. A job is also created whenever you run a workflow; see Run a workflow.
For general information, see Jobs.
To list jobs, use the UnstructuredClient
object’s jobs.list_jobs
function (for the Python SDK) or
the GET
method to call the /jobs
endpoint (for curl
or Postman).
To filter the list of jobs, use one or both of the following ListJobsRequest
parameters (for the Python SDK) or
query parameters (for curl
or Postman):
workflow_id=<workflow-id>
, replacing <workflow-id>
with the workflow’s unique ID.
To get this ID, see List workflows.status=<status>
, replacing <status>
with one of the following job statuses: completed
, failed
, im progress
, scheduled
, and stopped
.For curl
or Postman, you can specify multiple query parameters as ?workflow_id=<workflow-id>&status=<status>
.
Python SDK
Python SDK (async)
curl
To filter the list by workflow ID:
To filter the list by job status:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of jobs, on the Params tab, enter one or more of the following query parameter:
workflow_id
, Value: <workflow-id>
status
, Value: <status>
Click Send.
To get basic information about a job, use the UnstructuredClient
object’s jobs.get_job
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
This function/endpoint returns basic information about the job, such as:
To get details about a job’s current processing status, see Get processing details for a job.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get current processing information about a job, use the UnstructuredClient
object’s jobs.get_job_details
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>/details
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
To get basic information about a job, see Get a job.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get the list of any failed files for a job and why those files failed, use the UnstructuredClient
object’s jobs.get_job_failed_files
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>/failed-files
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To cancel a running job, use the UnstructuredClient
object’s jobs.cancel_job
function (for the Python SDK) or
the POST
method to call the /jobs/<job-id>/cancel
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
This applies only to jobs that use a workflow with a local source and a local destination.
To download a processed local file from a completed job, use GET
to call the /jobs/<job-id>/download
endpoint, replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
You must also provide Unstructured’s IDs for the file to download and the workflow’s output node. To get these IDs, see Get a job. In the response:
output_node_files
array.output_node_files
array’s file_id
field.output_node_files
array’s node_id
field.Currently, you cannot use the Unstructured user interface (UI) or the Unstructured Python SDK to download a file from a job that uses a workflow with a local source and a local destination.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
On the Params tab, enter the following query parameters:
file_id
, Value: <file-id>
node_id
, Value: <node-id>
Click Send.
The Unstructured UI features a no-code user interface for transforming your unstructured data into data that is ready for Retrieval Augmented Generation (RAG).
The Unstructured Workflow Endpoint, part of the Unstructured API, enables a full range of partitioning, chunking, embedding, and enrichment options for your files and data. It is designed to batch-process files and data in remote locations; send processed results to various storage, databases, and vector stores; and use the latest and highest-performing models on the market today. It has built-in logic to deliver the highest quality results at the lowest cost.
This page provides an overview of the Unstructured Workflow Endpoint. This endpoint enables Unstructured UI automation usage scenarios as well as for documentation, reporting, and recovery needs.
Choose one of the following options to get started with the Unstructured Workflow Endpoint:
curl
or Postman.This quickstart uses the Unstructured Python SDK to call the Unstructured Workflow Endpoint to get your data RAG-ready. The Python code for this quickstart is in a remote hosted Google Colab notebook. Data is processed on Unstructured-hosted compute resources.
The requirements are as follows:
Unstructured-IO/unstructured-ingest
repository in GitHub.Sign up, sign in, and get your API key
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
Create and set up the S3 bucket
This quickstart uses an Amazon S3 bucket as both the source location and the destination location. (You can use other source and destination types that are supported by Unstructured. If you use a different source or destination type, or if you use a different S3 bucket for the destination location, you will need to modify the quickstart notebook accordingly.)
Inside of the S3 bucket, a folder named input
represents the
source location. This is where your files to be processed will be stored.
The S3 URI to the source location will be s3://<your-bucket-name>/input
.
Inside of the same S3 bucket, a folder inside named output
represents the destination location. This
is where Unstructured will put the processed data.
The S3 URI to the destination location will be s3://<your-bucket-name>/output
.
Learn how to create an S3 bucket and set it up for Unstructured. (Do not run the Python SDK code or REST commands at the end of those setup instructions.)
Run the quickstart notebook
After your S3 bucket is created and set up, follow the instructions in this quickstart notebook.
View the processed data
After you run the quickstart notebook, go to your destination location to view the processed data.
Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured Workflow Endpoint to create connectors in the Unstructured UI.
Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured Workflow Endpoint to create workflows and jobs in the Unstructured UI.
Open a related notebook that covers many of the concepts that are shown in the preceding videos.
The Unstructured Python SDK, beginning with version 0.30.6, allows you to call the Unstructured Workflow Endpoint through standard Python code.
To install the Unstructured Python SDK, run the following command from within your Python virtual environment:
If you already have the Unstructured Python SDK installed, upgrade to at least version 0.30.6 by running the following command instead:
The Unstructured Python SDK code examples, shown later on this page and on related pages, use the following environment variable, which you can set as follows:
This environment variable enables you to more easily run the following Unstructured Python SDK examples and help prevent you from storing scripts that contain sensitive API keys in public source code repositories.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
Calls made by the Unstructured Python SDK’s unstructured_client
functions for creating, listing, updating,
and deleting connectors, workflows, and jobs in the Unstructured UI all use the Unstructured Workflow Endpoint URL.
This URL was provided to you when your Unstructured account was created.
If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The default URL for the Unstructured Worfklow Endpoint is https://platform.unstructuredapp.io/api/v1
.
However, you should always use the URL that was provided to you when your Unstructured account was created.
To specify an API URL in your code, set the server_url
parameter in the UnstructuredClient
constructor to the target API URL.
The Unstructured Workflow Endpoint enables you to work with connectors, workflows, and jobs in the Unstructured UI.
For general information about these objects, see:
Skip ahead to start learning about how to use the Unstructured Python SDK to work with connectors, workflows, and jobs programmatically.
The Unstructured Workflow Endpoint is callable from a set of Representational State Transfer (REST) endpoints, which you can call through standard REST-enabled
utilities, tools, programming languages, packages, and libraries. The examples, shown later on this page and on related pages, describe how to call the Unstructured Workflow Endpoint with
curl
and Postman. You can adapt this information as needed for your preferred programming languages and libraries, for example by using the
requests
library with Python.
You can also use the Unstructured Workflow Endpoint - Swagger UI to call the REST endpoints
that are available through the default Unstructured Workflow Endpoint URL: https://platform.unstructuredapp.io
. To use the Swagger UI, you must provide your Unstructured API key with each call. To
get this API key, see the quickstart, earlier on this page.
Note that you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The following curl
examples use the following environment variables, which you can set as follows:
For the API URL, this URL was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.
The default URL for the Unstructured Worfklow Endpoint is https://platform.unstructuredapp.io/api/v1
.
However, you should always use the URL that was provided to you when your Unstructured account was created.
These environment variables enable you to more easily run the following curl
examples and help prevent
you from storing scripts that contain sensitive URLs and API keys in public source code repositories.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The following Postman examples use variables, which you can set as follows:
In Postman, on your workspace’s sidebar, click Environments.
Click Globals.
Create two global variables with the following settings:
UNSTRUCTURED_API_URL
default
UNSTRUCTURED_API_KEY
secret
<your-unstructured-api-key>
<your-unstructured-api-key>
Click Save.
These variables enable you to more easily run the following examples in Postman and help prevent you from storing Postman collections that contain sensitive URLs and API keys in public source code repositories.
Unstructured offers a Postman collection that you can import into Postman to make Workflow Endpoint requests through a graphical user interface.
In your workspace, click Import.
In the Paste cURL, Raw text or URL box, enter the following URL, and then press Enter
:
On the sidebar, click Collections.
Expand Unstructured REST API - Workflow Endpoint.
Select the request that you want to use.
As applicable, modify the URL as needed to specify any required resource IDs for the request.
On the Headers tab, next to unstructured-api-key
, enter your Unstructured API key in the Value column.
As applicable, add, remove, or modify any other required headers for the request.
As applicable, on the Params tab, add, remove, or modify any required parameters for the request.
As applicable, on the Body tab, add, remove, or modify the required request body for the request.
Click Send.
To save the response, in the response area, click the ellipses, and then click Save response to file.
To get your Unstructured API key, do the following:
Sign in to your Unstructured account:
Get your Unstructured API key:
a. In the Unstructured UI, click API Keys on the sidebar.
b. Click Generate API Key.
c. Follow the on-screen instructions to finish generating the key.
d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The Unstructured Workflow Endpoint enables you to work with connectors, workflows, and jobs in the Unstructured UI.
For general information about these objects, see:
Skip ahead to start learning about how to use the REST endpoints to work with connectors, workflows, and jobs programmatically.
The following Unstructured SDKs, tools, and libraries do not work with the Unstructured Workflow Endpoint:
The following Unstructured API URL is also not supported: https://api.unstructuredapp.io/general/v0/general
(the default Unstructured Partition Endpoint URL).
You can list, get, create, update, delete, and test source connectors. You can also list, get, create, update, delete, and test destination connectors.
For general information, see Connectors.
To list source connectors, use the UnstructuredClient
object’s sources.list_sources
function (for the Python SDK) or
the GET
method to call the /sources
endpoint (for curl
or Postman).
To filter the list of source connectors, use the ListSourcesRequest
object’s source_type
parameter (for the Python SDK)
or the query parameter source_type=<type>
(for curl
or Postman),
replacing <type>
with the source connector type’s unique ID
(for example, for the Amazon S3 source connector type, S3
for the Python SDK or s3
for curl
or Postman).
To get this ID, see Sources.
Python SDK
Python SDK (async)
curl
To filter the list of source connectors:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of source connectors, on the Params tab, enter the following query parameter:
source_type
, Value: <type>
Click Send.
To get information about a source connector, use the UnstructuredClient
object’s sources.get_source
function (for the Python SDK) or
the GET
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a source connector, use the UnstructuredClient
object’s sources.create_source
function (for the Python SDK) or
the POST
method to call the /sources
endpoint (for curl
or Postman).
In the CreateSourceConnector
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the connector. For the specific settings to include, which differ by connector, see
Sources.
For the Python SDK, replace <type>
with the source connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Sources.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To update information about a source connector, use the UnstructuredClient
object’s sources.update_source
function (for the Python SDK) or
the PUT
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
In the UpdateSourceConnector
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the connector. For the specific settings to include, which differ by connector, see
Sources.
For the Python SDK, replace <type>
with the source connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Sources.
You must specify all of the settings for the connector, even for settings that are not changing.
You can change any of the connector’s settings except for its name
and type
.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To delete a source connector, use the UnstructuredClient
object’s sources.delete_source
function (for the Python SDK) or
the DELETE
method to call the /sources/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the source connector’s unique ID. To get this ID, see List source connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To test a source connector, use the POST
method to call the /sources/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see List source connectors.
The Python SDK does not support testing source connectors.
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get information about the most recent connector check for a source connector, use the GET
method to call the /sources/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see List source connectors.
The Python SDK does not support getting information about the most recent connector check for a source connector.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To list destination connectors, use the UnstructuredClient
object’s destinations.list_destinations
function (for the Python SDK) or
the GET
method to call the /destinations
endpoint (for curl
or Postman).
To filter the list of destination connectors, use the ListDestinationsRequest
object’s destination_type
parameter (for the Python SDK) or
the query parameter destination_type=<type>
(for curl
or Postman),
replacing <type>
with the destination connector type’s unique ID
(for example, for the Amazon S3 source connector type, S3
for the Python SDK or s3
for curl
or Postman).
To get this ID, see Destinations.
Python SDK
Python SDK (async)
curl
To filter the list of destination connectors:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of destination connectors, on the Params tab, enter the following query parameter:
destination_type
, Value: <type>
Click Send.
To get information about a destination connector, use the UnstructuredClient
object’s destinations.get_destination
function (for the Python SDK) or
the GET
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a destination connectors, use the UnstructuredClient
object’s destinations.create_destination
function (for the Python SDK) or
the POST
method to call the /destinations
endpoint (for curl
or Postman).
In the CreateDestinationConnector
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the connector. For the specific settings to include, which differ by connector, see
Destinations.
For the Python SDK, replace <type>
with the destination connector type’s unique ID (for example, for the Amazon S3 source connector type, S3
).
To get this ID, see Destinations.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To update information about a destination connector, use the UnstructuredClient
object’s destinations.update_destination
function (for the Python SDK) or
the PUT
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
In the UpdateDestinationConnector
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the connector. For the specific settings to include, which differ by connector, see
Destinations.
You must specify all of the settings for the connector, even for settings that are not changing.
You can change any of the connector’s settings except for its name
and type
.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the connector.
Click Send.
To delete a destination connector, use the UnstructuredClient
object’s destinations.delete_destination
function (for the Python SDK) or
the DELETE
method to call the /destinations/<connector-id>
endpoint (for curl
or Postman), replacing
<connector-id>
with the destination connector’s unique ID. To get this ID, see List destination connectors.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To test a destination connector, use the POST
method to call the /destinations/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see
List destination connectors.
The Python SDK does not support testing destination connectors.
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get information about the most recent connector check for a destination connector, use the GET
method to call the /destinations/<connector-id>/connection-check
endpoint (for curl
or Postman), replacing
<connector-id>
with the connector’s unique ID. To get this ID, see
List destination connectors.
The Python SDK does not support getting information about the most recent connector check for a destination connector.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
You can list, get, create, run, update, and delete workflows.
For general information, see Workflows.
To list workflows, use the UnstructuredClient
object’s workflows.list_workflows
function (for the Python SDK) or
the GET
method to call the /workflows
endpoint (for curl
or Postman).
To filter the list of workflows, use one or more of the following ListWorkflowsRequest
parameters (for the Python SDK) or
query parameters (for curl
or Postman):
source_id=<connector-id>
, replacing <connector-id>
with the source connector’s unique ID.
To get this ID, see List source connectors.destination_id=<connector-id>
, replacing <connector-id>
with the destination connector’s unique ID.
To get this ID, see List destination connectors.status=WorkflowState.<status>
(for the Python SDK) or status=<status>
(for curl
or Postman), replacing <status>
with one of the following workflow statuses: ACTIVE
or INACTIVE
(for the Python SDK) or active
or inactive
(for curl
or Postman).You can specify multiple query parameters, for example ?source_id=<connector-id>&status=<status>
.
Python SDK
Python SDK (async)
curl
To filter the list by source connector ID:
To filter the list by destination connector ID:
To filter the list by workflow status:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of workflows, on the Params tab, enter one or more of the following query parameter:
source_id
, Value: <connector-id>
destination_id
, Value: <connector-id>
status
, Value: <status>
Click Send.
To get information about a workflow, use the UnstructuredClient
object’s workflows.get_workflow
function (for the Python SDK) or
the GET
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To create a workflow, use the UnstructuredClient
object’s workflows.create_workflow
function (for the Python SDK) or
the POST
method to call the /workflows
endpoint (for curl
or Postman).
In the CreateWorkflow
object (for the Python SDK) or
the request body (for curl
or Postman),
specify the settings for the workflow. For the specific settings to include, see
Create a workflow.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the workflow.
Click Send.
To run a workflow manually, use the UnstructuredClient
object’s workflows.run_workflow
function (for the Python SDK) or
the POST
method to call the /workflows/<workflow-id>/run
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK (remote source and remote destination)
If the target workflow was originally created programmatically by the Unstructured Python SDK or with a REST API client such as curl
or Postman,
and the workflow uses a local source connector, you can run the workflow only with a REST API client such as curl
or Postman,
as described later in this section.
You cannot run the workflow with the Python SDK or the Unstructured user interface (UI), even though the workflow is visible in the UI.
Python SDK (async) (remote source and remote destination)
If the target workflow was originally created programmatically by the Unstructured Python SDK or with a REST API client such as curl
or Postman,
and the workflow uses a local source connector, you can run the workflow only with a REST API client such as curl
or Postman,
as described later in this section.
You cannot run the workflow with the Python SDK or the Unstructured user interface (UI), even though the workflow is visible in the UI.
curl (remote source and remote destination)
curl (local source and local or remote destination)
In the following command, replace:
</full/path/to/local/filename.extension>
with the full path to the local file to upload.<filename.extension>
with the filename of the local file to upload.<local-file-media-type>
with the local file’s media type. For a list of available media types, such as application/pdf
, see Media Types.To upload multiple files, add additional --form
entries, one per file.
For a local destination, to access the processed files’ data, download a processed local file from the workflow’s job run.
Postman (remote source and remote destination)
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
Postman (local source and local or remote destination)
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
On the Body tab, select form-data, and specify the settings for the workflow run:
Key: input_files
, File, Value: Click the Value box, then click New file from local machine, and select the file to upload.
To upload multiple files, add additional input_files
entries after this one, one entry per additional file to upload.
Key: filename
, Text, Value: Type the name of the file that you just uploaded.
To upload multiple files, add additional filename
entries after this one, one entry per additional file to upload. Make sure the order of these
filename
entries matches the order of the input_files
entries, respectively.
Key: type
, Text, Value: <local-file-media-type>
To upload multiple files, add additional type
entries after this one, one entry per additional file to upload. Make sure the order of these
type
entries matches the order of the input_files
entries, respectively.
For a list of available media types, such as application/pdf
, see Media Types.
Click Send.
For a local destination, to access the processed files’ data, download a processed local file from the workflow’s job run.
To run a workflow on a schedule instead, specify the schedule
setting in the request body when you create or update a
workflow. See Create a workflow or Update a workflow.
To update information about a workflow, use the UnstructuredClient
object’s workflows.update_workflow
function (for the Python SDK) or
the PUT
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
In UpdateWorkflow
object (for the Python SDK) or
the request body (for curl
or Postman), specify the settings for the workflow. For the specific settings to include, see
Update a workflow.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select PUT.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
content-type
, Value, application/json
On the Body tab, select raw and JSON, and specify the settings for the workflow.
Click Send.
To delete a workflow, use the UnstructuredClient
object’s workflows.delete_workflow
function (for the Python SDK) or
the DELETE
method to call the /workflows/<workflow-id>
endpoint (for curl
or Postman), replacing
<workflow-id>
with the workflow’s unique ID. To get this ID, see List workflows.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select DELETE.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
You can list, get, and cancel jobs.
A job is created automatically whenever a workflow runs on a schedule; see Create a workflow. A job is also created whenever you run a workflow; see Run a workflow.
For general information, see Jobs.
To list jobs, use the UnstructuredClient
object’s jobs.list_jobs
function (for the Python SDK) or
the GET
method to call the /jobs
endpoint (for curl
or Postman).
To filter the list of jobs, use one or both of the following ListJobsRequest
parameters (for the Python SDK) or
query parameters (for curl
or Postman):
workflow_id=<workflow-id>
, replacing <workflow-id>
with the workflow’s unique ID.
To get this ID, see List workflows.status=<status>
, replacing <status>
with one of the following job statuses: completed
, failed
, im progress
, scheduled
, and stopped
.For curl
or Postman, you can specify multiple query parameters as ?workflow_id=<workflow-id>&status=<status>
.
Python SDK
Python SDK (async)
curl
To filter the list by workflow ID:
To filter the list by job status:
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
To filter the list of jobs, on the Params tab, enter one or more of the following query parameter:
workflow_id
, Value: <workflow-id>
status
, Value: <status>
Click Send.
To get basic information about a job, use the UnstructuredClient
object’s jobs.get_job
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
This function/endpoint returns basic information about the job, such as:
To get details about a job’s current processing status, see Get processing details for a job.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get current processing information about a job, use the UnstructuredClient
object’s jobs.get_job_details
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>/details
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
To get basic information about a job, see Get a job.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To get the list of any failed files for a job and why those files failed, use the UnstructuredClient
object’s jobs.get_job_failed_files
function (for the Python SDK) or
the GET
method to call the /jobs/<job-id>/failed-files
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
To cancel a running job, use the UnstructuredClient
object’s jobs.cancel_job
function (for the Python SDK) or
the POST
method to call the /jobs/<job-id>/cancel
endpoint (for curl
or Postman), replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
Python SDK
Python SDK (async)
curl
Postman
In the method drop-down list, select POST.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
Click Send.
This applies only to jobs that use a workflow with a local source and a local destination.
To download a processed local file from a completed job, use GET
to call the /jobs/<job-id>/download
endpoint, replacing
<job-id>
with the job’s unique ID. To get this ID, see List jobs.
You must also provide Unstructured’s IDs for the file to download and the workflow’s output node. To get these IDs, see Get a job. In the response:
output_node_files
array.output_node_files
array’s file_id
field.output_node_files
array’s node_id
field.Currently, you cannot use the Unstructured user interface (UI) or the Unstructured Python SDK to download a file from a job that uses a workflow with a local source and a local destination.
curl
Postman
In the method drop-down list, select GET.
In the address box, enter the following URL:
On the Headers tab, enter the following headers:
unstructured-api-key
, Value: {{UNSTRUCTURED_API_KEY}}
accept
, Value: application/json
On the Params tab, enter the following query parameters:
file_id
, Value: <file-id>
node_id
, Value: <node-id>
Click Send.