Create and Run Flows
Create a Flow
==================
1. Click "Flows"
Access the Flows section.
2. Click "New Flow"
Create a new Flow.
3. Enter a unique name for your flow
Enter a unique name for your flow
4. Select your Source Data Type ("Source Connection" or "Transformation")
1). Use "Source Connection" if you are trying to use data from the source connections 2). Use "Transformations" if you're trying to use transformations you created
5. If you select "Source Connection", the source data dropdown should list all of your previously created source connections
Select your desired connection
6. If your source data selection is a table based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), you should see a list of tables to select from
Select your desired table
7. Once you select your table, the "Embed Columns" field should show all columns from the selected table.
Select all columns that you are want to embed
8. Select all columns that you are want to embed
Select all columns that you are want to embed
9. Use the "Replication Keys" field if you are going to create a schedule for your flow to run periodically
Select the field which will be used as the as an anchor check for new data. For example, if you select "id" as your replication key, the system will track the last ID value that was processed and will select all values above that value for subsequent flow runs
10. If your source data selection is a file based connection (i.e. PostgreSQL, MySQL, Snowflake, Salesforce), the default value for "Tables", "Embed Columns" and "Replication Keys" will be "NA"
Select "NA" values
11. Select your Target Connection
After you have selected your source data, you can now select your Target connection from the dropdown list
12. Select a previously created Embedding Model
Select your preferred model
13. Enter your preferred chunk size and your chunk overlap values
Default chunk size = 1024 and default overlap = 100
14. If you want to schedule your flow to run periodically (e.g. every day or every hour), click the "Schedule Flow" collapsible section to view the options
Click the collapsible arrow to display options
15. Click the "Start Date & Start Time" section to select the date and time that you want your flow to start
For example, if you enter your start date date and start time as December 31, 2024 at 1:00PM, that will be the first scheduled run
16. Select the flow run interval
Once you select your start date & time, you will need to select how often you want the flow to run. First select the internal (days/hours/mins) and then select the value. For example, if you want your flow to run every day starting from December 31, 2024 at 1:00PM, you should select "days" as interval and enter "1" in the value field. if you want your flow to run every 6 hours starting from December 31, 2024 at 1:00PM, you should select "hours" as interval and enter "6" in the value field.
17. Review your Schedule Configuration
The example image shows a flow which is supposed to start on April 25, 2024 at 12 AM and is scheduled to run every day
18. Click "Create"
Click "Create" button
Last updated