How to upload an SFTP file

Description

The SFTP upload route aims to provide a flexible way to programmatically update the Single Fan View. The way it works is you periodically push your data to your SFTP server, and let Cortex check for the existence of new files on the same server. Whenever Cortex finds new files that match our modules, that will trigger and update on SFV, with the new file information.


Requisites

For setting up the SFTP periodical routine, you need to set up a couple of requisites:


  • SFTP Server

First you need to set up your own SFTP Server and an automated process that will periodically populate it with new files. You can decide on batch periodicity, which will also have to be set up on Cortex. The files can be of incremental nature or full refresh files - more on that below.


  • Cortex Secret

You will need to share authentication details with Cortex, which will enable us to access the SFTP server. The credentials will be securely stored and only used for programmatic access, for looking for new files.


  • Cortex Schedule

Cortex needs to know when to look for new files, so every batch will have a predetermined periodicity. This is also dependent on the files made available on the SFTP server, because Cortex will only try to process a file once, by file name.


Once you have these correctly set up, your Cortex instance will be set to start looking for files in your SFTP and update the information accordingly on your Fan Data Platform.


File Naming

In order to map the information on each file to the corresponding modules, file naming follows a predetermined convention. Also, through different naming, we can trigger different types of database refreshment.

The file name convention is as follows:

  • Incremental loads: <module_name>_module_<load_description>.<file_format>
  • Full refresh: full_<module_name>_module_<load_description>.<file_format>

⚠️ Important: Each file will only be processed once, so <load_description> should be used to create non-repeatable names, eg: through the use of timestamps, such as user_module_20250101.csv for daily User loads


Supported Modules

Currently we support data syncing to the following modules on your Fan Data Platform, through SFTP upload:

Module Name

File name Prefix

Incremental Example

Full refresh Example

User

user_module_

user_module_20250101.csv

full_user_module_20250101.csv

Contact Preferences

contact_preference_module_

contact_preference_module_abc.csv

full_contact_preference_module_abc.csv

Custom Data Fields

user_data_field_module_

user_data_field_module_incremental_1.csv

full_user_data_field_module_refresh_all_1.csv

Form

form_module_

form_module_123.csv

full_form_module_123.csv

Digital Usage

digital_usage_module_

digital_usage_module_20250101.csv

full_digital_usage_module_20250101.csv

Email

email_module_

email_module_20250101.csv

full_email_module_20250101.csv

Retail

retail_module_

retail_module_load.csv

full_retail_module_load.csv

Structured Event

structured_event_module_

structured_event_module_xyz.csv

full_structured_event_module_xyz.csv

Ticket

ticket_module_

ticket_module_zzz.csv

full_ticket_module_zzz.csv

For more information on each of the supported modules, please check the "Modules Structure" section


Refresh Type

  • Incremental

Naming convention: No file prefix

Typically smaller batches of data that are added incrementally to what is already on the Fan Data Platform. This type of information is always in addition to what is already on the platform, and depends on upstream processes for its calculation.

An example would be the daily load of previous days Transactions, which are always incremental.


  • Full Refresh

Naming Convention: file name starts with the full_ prefix.

Typically larger batches of data, which will entirely replace the data that is already present on the database for that particular module. This erases the old data and replaces it with the new entries coming in.

An example of this would be a monthly complete refresh of the Users database, that can be used to clear unwanted data and guarantee the consistency of existing records.


File Naming Examples

Incremental information for the User module: user_module_new_file_20250101.csv

Full refreshing Contact Preferences module: full_contact_preferences_module_trigger_full_import_20250101.csv

Supported formats

Currently the supported file extensions are:

  • CSV
  • Parquet