How to upload an SFTP file
Description
The SFTP upload route aims to provide a flexible way to programmatically update the Single Fan View. The way it works is you periodically push your data to your SFTP server, and let Cortex check for the existence of new files on the same server. Whenever Cortex finds new files that match our modules, that will trigger and update on SFV, with the new file information.
Requisites
For setting up the SFTP periodical routine, you need to set up a couple of requisites:
- SFTP Server
First you need to set up your own SFTP Server and an automated process that will periodically populate it with new files. You can decide on batch periodicity, which will also have to be set up on Cortex. The files can be of incremental nature or full refresh files - more on that below.
- Cortex Secret
You will need to share authentication details with Cortex, which will enable us to access the SFTP server. The credentials will be securely stored and only used for programmatic access, for looking for new files.
- Cortex Schedule
Cortex needs to know when to look for new files, so every batch will have a predetermined periodicity. This is also dependent on the files made available on the SFTP server, because Cortex will only try to process a file once, by file name.
Once you have these correctly set up, your Cortex instance will be set to start looking for files in your SFTP and update the information accordingly on your Fan Data Platform.
File Naming
In order to map the information on each file to the corresponding modules, file naming follows a predetermined convention. Also, through different naming, we can trigger different types of database refreshment.
The file name convention is as follows:
- Incremental loads:
<module_name>_module_<load_description>.<file_format> - Full refresh:
full_<module_name>_module_<load_description>.<file_format>
⚠️ Important: Each file will only be processed once, so <load_description> should be used to create non-repeatable names, eg: through the use of timestamps, such as user_module_20250101.csv for daily User loads
Supported Modules
Currently we support data syncing to the following modules on your Fan Data Platform, through SFTP upload:
Module Name | File name Prefix | Incremental Example | Full refresh Example |
|---|---|---|---|
User | user_module_ | user_module_20250101.csv | full_user_module_20250101.csv |
Contact Preferences | contact_preference_module_ | contact_preference_module_abc.csv | full_contact_preference_module_abc.csv |
Custom Data Fields | user_data_field_module_ | user_data_field_module_incremental_1.csv | full_user_data_field_module_refresh_all_1.csv |
Form | form_module_ | form_module_123.csv | full_form_module_123.csv |
Digital Usage | digital_usage_module_ | digital_usage_module_20250101.csv | full_digital_usage_module_20250101.csv |
email_module_ | email_module_20250101.csv | full_email_module_20250101.csv | |
Retail | retail_module_ | retail_module_load.csv | full_retail_module_load.csv |
Structured Event | structured_event_module_ | structured_event_module_xyz.csv | full_structured_event_module_xyz.csv |
Ticket | ticket_module_ | ticket_module_zzz.csv | full_ticket_module_zzz.csv |
For more information on each of the supported modules, please check the "Modules Structure" section
Refresh Type
- Incremental
Naming convention: No file prefix
Typically smaller batches of data that are added incrementally to what is already on the Fan Data Platform. This type of information is always in addition to what is already on the platform, and depends on upstream processes for its calculation.
An example would be the daily load of previous days Transactions, which are always incremental.
- Full Refresh
Naming Convention: file name starts with the full_ prefix.
Typically larger batches of data, which will entirely replace the data that is already present on the database for that particular module. This erases the old data and replaces it with the new entries coming in.
An example of this would be a monthly complete refresh of the Users database, that can be used to clear unwanted data and guarantee the consistency of existing records.
File Naming Examples
Incremental information for the User module: user_module_new_file_20250101.csv
Full refreshing Contact Preferences module: full_contact_preferences_module_trigger_full_import_20250101.csv
Supported formats
Currently the supported file extensions are:
- CSV
- Parquet
Updated about 1 month ago