Create and Run Flows

Create a Flow

==================

Click here to watch

1. Click "Flows"

Access the Flows section.

Click 'Flows'

2. Click "New Flow"

Create a new Flow.

Click 'New Flow'

3. Enter a unique name for your flow

Enter a unique name for your flow

Enter a unique name for your flow

4. Select your Source Data Type ("Source Connection" or "Transformation")

1). Use "Source Connection" if you are trying to use data from the source connections 2). Use "Transformations" if you're trying to use transformations you created

Select your Source Data Type ('Source Connection' or 'Transformation')

5. If you select "Source Connection", the source data dropdown should list all of your previously created source connections

Select your desired connection

If you select 'Source Connection', the source data dropdown should list all of your previously created source connections

6. If your source data selection is a table based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), you should see a list of tables to select from

Select your desired table

If your source data selection is a table based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), you should see a list of tables to select from

7. Once you select your table, the "Embed Columns" field should show all columns from the selected table.

Select all columns that you are want to embed

Once you select your table, the 'Embed Columns' field should show all columns from the selected table.

8. Select all columns that you are want to embed

Select all columns that you are want to embed

Select all columns that you are want to embed

9. Use the "Replication Keys" field if you are going to create a schedule for your flow to run periodically

Select the field which will be used as the as an anchor check for new data. For example, if you select "id" as your replication key, the system will track the last ID value that was processed and will select all values above that value for subsequent flow runs

Use the 'Replication Keys' field if you are going to create a schedule for your flow to run periodically

10. If your source data selection is a file based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), the default value for "Tables", "Embed Columns" and "Replication Keys" will be "NA"

Select "NA" values

If your source data selection is a file based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), the default value for 'Tables', 'Embed Columns' and 'Replication Keys' will be 'NA'

11. Select your Target Connection

After you have selected your source data, you can now select your Target connection from the dropdown list

Select your Target Connection

12. Select a previously created Embedding Model

Select your preferred model

Select a previously created Embedding Model

13. Enter your preferred chunk size and your chunk overlap values

Default chunk size = 1024 and default overlap = 100

Enter your preferred chunk size and your chunk overlap values

14. If you want to schedule your flow to run periodically (e.g. every day or every hour), click the "Schedule Flow" collapsible section to view the options

Click the collapsible arrow to display options

If you want to schedule your flow to run periodically (e.g. every day or every hour), click the 'Schedule Flow' collapsible section to view the options

15. Click the "Start Date & Start Time" section to select the date and time that you want your flow to start

For example, if you enter your start date date and start time as December 31, 2024 at 1:00PM, that will be the first scheduled run

Click the 'Start Date & Start Time' section to select the date and time that you want your flow to start

16. Select the flow run interval

Once you select your start date & time, you will need to select how often you want the flow to run. First select the internal (days/hours/mins) and then select the value. For example, if you want your flow to run every day starting from December 31, 2024 at 1:00PM, you should select "days" as interval and enter "1" in the value field. if you want your flow to run every 6 hours starting from December 31, 2024 at 1:00PM, you should select "hours" as interval and enter "6" in the value field.

Select the flow run interval

17. Review your Schedule Configuration

The example image shows a flow which is supposed to start on April 25, 2024 at 12 AM and is scheduled to run every day

Review your Schedule Configuration

18. Click "Create"

Click "Create" button

Click 'Create'