Description

The SFTP upload route aims to provide a flexible way to programmatically update the Single Fan View. The way it works is you periodically push your data to your SFTP server, and let Cortex check for the existence of new files on the same server. Whenever Cortex finds new files that match our modules, that will trigger and update on SFV, with the new file information.

Requisites

For setting up the SFTP periodical routine, you need to set up a couple of requisites:

• SFTP Server

First you need to set up your own SFTP Server and an automated process that will periodically populate it with new files. You can decide on batch periodicity, which will also have to be set up on Cortex. The files can be of incremental nature or full refresh files - more on that below.

• Cortex Secret

You will need to share authentication details with Cortex, which will enable us to access the SFTP server. The credentials will be securely stored and only used for programmatic access, for looking for new files.

• Cortex Schedule

Cortex needs to know when to look for new files, so every batch will have a predetermined periodicity. This is also dependent on the files made available on the SFTP server, because Cortex will only try to process a file once, by file name.

Once you have these correctly set up, your Cortex instance will be set to start looking for files in your SFTP and update the information accordingly on your Fan Data Platform.

File Naming

In order to map the information on each file to the corresponding modules, file naming follows a predetermined convention. Also, through different naming, we can trigger different types of database refreshment.

Supported Modules

Currently we support data syncing to three different modules on your Fan Data Platform, through SFTP upload:

• User

File naming start with user_module_ or users_module_

• Contact Preferences

File naming start with contact_preference_module_ or contact_preferences_module_

• Custom Data Fields

File naming start with custom_field_module_ or custom_fields_module_

For more information on the supported modules, please check the "Modules Structure" section

Refresh Type

• Incremental

Naming convention: No file prefix

Typically smaller batches of data that are added incrementally to what is already on the Fan Data Platform. This type of information is always in addition to what is already on the platform, and depends on upstream processes for its calculation.

An example would be the daily load of previous days Transactions, which are always incremental.

• Full Refresh

Naming Convention: file name starts with the full_ prefix.

Typically larger batches of data, which will entirely replace the data that is already present on the database for that particular module. This always erases the old data and replaces it with the new entries coming in.

An example of this would be a monthly complete refresh of the Users database, that can be used to clear unwanted data and guarantee the consistency of existing records.

File Naming Examples

Incremental information for the User module: user_module_new_file.csv

Full refreshing Contact Preferences module: full_contact_preferences_module_trigger_import.csv

Supported formats

Currently the supported file extensions are:

• CSV
• Parquet