How to upload an External Bucket file
Description
The External Stage upload connector lets you programmatically update the Single Fan View by dropping files into an External bucket that Cortex has been granted read access to. Cortex makes a connector available that periodically pulls new files from it into SFV.
Files are matched to a module by which folder they're placed in.
Requisites
To set up the External Stage routine, you need:
-
DWH
You set up a process ingesting source data from your different systems, modelling it to fit each module you wish to populate in the SFV.
eg: Ticketing, Retail, etc
-
Bucket
A bucket (or a dedicated prefix within an existing bucket) that you control, with an automated process that periodically populates it with new files.
-
Bucket Access
Grant read access (
roles/storage.objectViewer, or equivalent object-read permissions) on the bucket to the service account Cortex provides during onboarding. Share the bucket URL (gcs://<bucket-name>/<optional-prefix>/) with Cortex so we can create the corresponding integration. -
Cortex Schedule
How often Cortex checks the bucket for new files. This should be set to match how often you actually populate the bucket with new files, usually hourly.
Once these are set up, Cortex will start pulling files from your bucket and updating your Fan Data Platform accordingly.
Bucket Structure
Files are routed to the correct module purely by folder — place each module's files inside its corresponding subfolder in the bucket:
| Module Name | Folder |
|---|---|
| User | user/ |
| Contact Preferences | user_contact_preference/ |
| Custom Data Fields | user_data_field/ |
| Ticket | user_ticket/ |
| Retail | user_retail/ |
| Digital Usage | user_digital_usage/ |
| Structured Event | user_structured_event/ |
user_email/ | |
| Form | user_form/ |
| Streaming | user_streaming/ |
| Push | user_push/ |
| Opportunity | user_opportunity/ |
| Entitlement | user_entitlement/ |
| Hospitality | user_hospitality/ |
In order to trigger a full refresh — load behavior (incremental upsert vs. full refresh) is controlled on Cortex's side per scheduled run. Full refreshes may be triggered Ad-hoc, per request.
⚠️ Important:
-
Cortex only considers files modified within a rolling window (60 days by default) on each run.
-
Once a file has been successfully loaded, re-uploading a file with the exact same name will not be re-processed. Use a unique name per batch (e.g. include a timestamp) if you want to guarantee it's picked up as new.
Supported formats
- CSV
- Parquet
Updated 23 days ago