Snowflake

If you’re new to Unstructured, read this note first.

Before you can create a source connector, you must first sign in to your Unstructured account:

• If you do not already have an Unstructured account, go to https://unstructured.io/contact and fill out the online form to indicate your interest.
• If you already have an Unstructured account, sign in by using the URL of the sign in page that Unstructured provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at [e-mail].

After you sign in, the Unstructured user interface (UI) appears, which you use to get your Unstructured API key. To learn how, watch this 40-second how-to video.

After you create the source connector, add it along with a destination connector to a workflow. Then run the worklow as a job. To learn how, try out the hands-on Workflow Endpoint quickstart, go directly to the quickstart notebook, or watch the two 4-minute video tutorials for the Unstructured Python SDK.

You can also create source connectors with the Unstructured user interface (UI). Learn how.

If you need help, reach out to the community on Slack, or contact us directly.

You are now ready to start creating a source connector! Keep reading to learn how.

Ingest your files into Unstructured from Snowflake.

The requirements are as follows.

• A Snowflake account and its account identifier.

  To get the identifier for the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click Account > View account details.
  3. On the Account tab, note the value of the Account Identifier field.

  Alternatively, the following Snowflake query returns the current account’s identifier:

  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  

• The Snowflake user’s login name (not its username) and its password in the account.

  To view the login name for a user:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Admin > Users & Roles.
  3. On the Users tab, in the list of available users, click the name of the target user.
  4. In the About tile, note the Login Name for the user.

  Alternatively, the following Snowflake query returns information about the user with the username of <my-user>, including their login_name value representing their login name:

  SHOW USERS LIKE '<my-user>';
  

• The name of the Snowflake role that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  • To create a database in Snowflake, the role needs to be granted CREATE DATABASE privilege at the current account level; and USAGE privilege on the warehouse that is used to create the database.
  • To create a schema in a database in Snowflake, the role needs to be granted USAGE privilege on the database and the warehouse that is used to create the schema; and CREATE SCHEMA on the database.
  • To create a table in a schema in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to create the table; and CREATE TABLE on the schema.
  • To write to a table in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to write to the table; and INSERT on the table.
  • To read from a table in Snowflake, the role needs to be granted USAGE privilege on the database and schema and the warehouse that is used to write to the table; and SELECT on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Admin > Users & Roles.
  3. Click the Roles tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  SHOW ROLES;
  

  Grant privileges to a role. Learn more.

• The Snowflake warehouse’s hostname and its port number in the account.

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Admin > Warehouses. This view does not provide access to the warehouses’ hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a type of SNOWFLAKE_DEPLOYMENT:

  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  

• The name of the Snowflake database in the account.

  To view a list of available databases in the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Data > Databases.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  SHOW DATABASES;
  

• The name of the schema in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Data > Databases.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  SHOW SCHEMAS;
  

  The following Snowflake query returns a list of available schemas for the database named <database_name> in the current account:

  SHOW SCHEMAS IN DATABASE <database_name>;
  

• The name of the table in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to Snowsight with your Snowflake account.
  2. In Snowsight, on the navigation menu, click Data > Databases.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand Tables.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named <schema_name> in the datbase named <database_name> in the current account:

  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The recommended table schema for Unstructured is as follows. In the following CREATE TABLE statement, replace the following placeholders with the appropriate values:

  • <database_name>: The name of the target database in the Snowflake account.
  • <schema_name>: The name of the target schema in the database.
  • <number-of-dimensions>: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search, this value must match the number of dimensions that you plan to have Snowflake generate or search against.
  SQL

  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
    ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
    RECORD_ID VARCHAR,
    ELEMENT_ID VARCHAR,
    TEXT VARCHAR,
    EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
    TYPE VARCHAR,
    SYSTEM VARCHAR,
    LAYOUT_WIDTH DECIMAL,
    LAYOUT_HEIGHT DECIMAL,
    POINTS VARCHAR,
    URL VARCHAR,
    VERSION VARCHAR,
    DATE_CREATED TIMESTAMP_TZ,
    DATE_MODIFIED TIMESTAMP_TZ,
    DATE_PROCESSED TIMESTAMP_TZ,
    PERMISSIONS_DATA VARCHAR,
    RECORD_LOCATOR VARCHAR,
    CATEGORY_DEPTH INTEGER,
    PARENT_ID VARCHAR,
    ATTACHED_FILENAME VARCHAR,
    FILETYPE VARCHAR,
    LAST_MODIFIED TIMESTAMP_TZ,
    FILE_DIRECTORY VARCHAR,
    FILENAME VARCHAR,
    LANGUAGES ARRAY,
    PAGE_NUMBER VARCHAR,
    LINKS VARCHAR,
    PAGE_NAME VARCHAR,
    LINK_URLS ARRAY,
    LINK_TEXTS ARRAY,
    SENT_FROM ARRAY,
    SENT_TO ARRAY,
    SUBJECT VARCHAR,
    SECTION VARCHAR,
    HEADER_FOOTER_TYPE VARCHAR,
    EMPHASIZED_TEXT_CONTENTS ARRAY,
    EMPHASIZED_TEXT_TAGS ARRAY,
    TEXT_AS_HTML VARCHAR,
    REGEX_METADATA VARCHAR,
    DETECTION_CLASS_PROB DECIMAL,
    IMAGE_BASE64 VARCHAR,
    IMAGE_MIME_TYPE VARCHAR,
    ORIG_ELEMENTS VARCHAR,
    IS_CONTINUATION BOOLEAN
  );
  

• The name of the column in the table that uniquely identifies each record (for example, RECORD_ID).

To create a Snowflake source connector, see the following examples.

Python SDK

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import (
    CreateSourceConnector,
    SourceConnectorType,
    SnowflakeSourceConnectorConfigInput
)

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type=SourceConnectorType.SNOWFLAKE,
                config=SnowflakeSourceConnectorConfigInput(
                    account="<account>",
                    role="<role>",
                    user="<user>",
                    password="<password>",
                    host="<host",
                    port=<port>
                    database="<database>",
                    schema_="<schema>",
                    table_name="<table_name>",
                    id_column="<id-column>",
                    fields=[
                        "<field>", 
                        "<field>"
                    ],
                    batch_size=<batch-size> 
                )
            )
        )
    )

    print(response.source_connector_information)

Replace the preceding placeholders as follows:

• <name> (required) - A unique name for this connector.
• <account> (required): The target Snowflake account’s identifier.
• <role> (required): The name of the Snowflake role that the user belongs to. This role must have the appropriate access to the target Snowflake warehouse, database, schema, and table.
• <user> (required): The target Snowflake user’s login name (not their username).
• <password> (required): The user’s password.
• <host> (required): The hostname of the target Snowflake warehouse.
• <port> (required): The warehouse’s port number. The default is 443 if not otherwise specified.
• <database> (required): The name of the target Snowflake database.
• <schema> (required): The name of the target Snowflake schema within the database.
• <table_name>: The name of the target Snowflake table within the database’s schema. For the destination connector, the default is elements if not otherwise specified.
• <columns> (source connector only): A comma-separated list of columns to fetch from the table. By default, all columns are fetched unless otherwise specified.
• <id-column> (required, source connector only): The name of the column that uniquely identifies each record in the table.
• <record-id-key> (destination connector only): The name of the column that uniquely identifies each record in the table. The default is record_id if not otherwise specified.
• <batch-size> (required): The maximum number of rows to fetch for each batch. The default is 50 if not otherwise specified.

Assistant

Responses are generated using AI and may contain mistakes.