Skip to main content
Data Exports delivers your data to a place you own, on a schedule you choose. It is a dashboard feature under Admin > Data Exports and needs the admin.data_exports.manage permission. There is no API endpoint for it.

Choose a destination

The destination type is chosen when you create the destination and cannot be changed afterwards. Snowflake is the only warehouse we load into directly. Athena, Databricks, BigQuery and DuckDB are not destinations of their own: they read the Parquet files in an S3-compatible bucket.

S3-compatible storage

What you need

  • An S3-compatible bucket. AWS S3 works natively. Backblaze B2, Cloudflare R2, Wasabi, DigitalOcean Spaces and MinIO work with their endpoint URL.
  • An access key scoped to that bucket. The minimal policy is shown in the destination dialog and needs only s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on arn:aws:s3:::YOUR_BUCKET/*.
  • Optionally, a lifecycle rule on the bucket that aborts incomplete multipart uploads after a day. We check for it and warn if it is missing.
The secret key is encrypted at rest with envelope encryption and bound to the destination record. It is never logged and never returned by the API. You can rotate it by editing the destination and revoke it at any time in your own account.

Setting up an S3 destination

  1. Open Admin > Data Exports and choose New destination, then pick Amazon S3 or compatible storage.
  2. Enter a name, the bucket name, region, access key ID and secret. Add a prefix if you want the files under a folder, and an endpoint URL for a non-AWS provider.
  3. Pick a schedule, the datasets to export (see Datasets; a new destination starts with Messages, Projects, Opt-outs and Link clicks ticked), and, for parent organizations, which child organizations to include.
  4. Save. We write one small probe file, always at the same path _validation/connection-test.txt under your prefix, to check the key can write. It needs only s3:PutObject, and we never delete it, so you can ignore or remove it yourself. If the write fails, the destination is not created and the error from your provider is shown.

Snowflake

A Snowflake destination loads each run straight into tables in your own Snowflake account. You do not need a storage bucket. Setup is one script that you run once.

Before you start

  • A Snowflake account, and an administrator who can create roles, users and warehouses and manage grants (for example ACCOUNTADMIN).
  • A database and a schema that already exist. We create tables inside that schema but never create the database or the schema.
  • Your account identifier, such as myorg-myaccount. In Snowsight it is under the account menu, then View account details. Pasting your account URL also works.

Setting up a Snowflake destination

  1. Open Admin > Data Exports and choose New destination, then pick Snowflake.
  2. Enter a name, the account identifier, the database, the schema, and the names to use for the warehouse, role and service user. The three names start with sensible defaults (DATA_EXPORT_WH, DATA_EXPORT_ROLE and DATA_EXPORT_USER) and you can rename them. Names can contain letters, digits, underscores and dollar signs, start with a letter or underscore, and are stored in upper case. The service user must be a new user that exists only for this export: if a user with that name already exists, the script replaces its key, so its current key-pair sign-ins stop working.
  3. Pick a schedule, the datasets to export and, for parent organizations, which child organizations to include. Choose Create.
  4. We generate a key pair for the destination and show a setup script. Copy it into a Snowflake worksheet and run it as an administrator. You can open the script again at any time from the destination’s Setup script button, and it is safe to run more than once.
  5. Back in Data Exports, press Validate. Exports start only after validation passes.

What the setup script creates

  • A role that can use the database and create tables in the schema you named, and nothing else.
  • A service user with no password. It signs in only with the key we generated, and only our public key is installed in your account. The private key is encrypted at rest and is never shown or returned.
  • A small warehouse (extra small, suspends after a minute of inactivity) for the exporter to run on.
  • The grants that tie these together. The role is also granted to SYSADMIN so your own administrators can read the tables.
To revoke access at any time, drop the service user in Snowflake. Nothing else of ours remains in your account except the tables, which are yours.

What Validate checks

Validate signs in as the service user and confirms the role and warehouse are in place. It then creates a small test table in your schema under a name with a random suffix (so it can never replace a table of yours), writes one row to it, and loads one more row the way every export does: a tiny file uploaded to the service user’s own stage and copied into a temporary table. Everything it creates is removed afterwards. It proves the key is installed, the role is granted, the schema accepts new tables, the warehouse can run statements and the user’s stage works. If something is wrong, the message names the likely fix, such as running the script again or correcting an object name, and Show what the provider told us reveals the exact error.

Network policies

We have no fixed IP addresses to give you. If your Snowflake account enforces a network policy, attach a user-level network policy to the export service user that allows connections from any address. The comments in the setup script repeat this. Without it, Validate reports that your network policy blocked the connection.

Tables and how they stay current

  • There is one table per dataset in your schema, named after the dataset in upper case, for example MESSAGE_EVENTS or DONATIONS. Tables are created on the first run, and columns we add later appear on the next run.
  • Each run loads its rows and merges them on the dataset’s key, so every table holds one current row per record. A row that changes is updated in place, and a row you already have is never inserted twice. You do not need a de-duplication step.
  • The reference tables (TAGS, CONTACT_LISTS, BRANDS, CAMPAIGNS and ORGANIZATIONS) always match your current list: each run replaces their rows in one step, so an item you removed disappears on the next run, and a run that fails leaves the previous rows in place.
  • Columns marked JSON are stored as text. Read them with PARSE_JSON, for example PARSE_JSON(TAG_IDS).
  • Timestamps are stored in UTC and money columns as exact decimals in US dollars.

What a Snowflake run costs you

Each run starts your warehouse, and Snowflake bills your account for the time it runs, with a minimum of about a minute per run. A daily schedule therefore costs less than an hourly one.

What is not supported

Snowflake is the only warehouse we load into directly. We do not load into Databricks, BigQuery, Athena or DuckDB. They can all read the Parquet files from an S3-compatible destination. There is no password sign-in, and to replace the key you create a new destination.

Schedules and windows

Every window ends five minutes before the run starts so late carrier receipts are included. Each scheduled run continues from where the previous one ended, so a paused or failed run never leaves a gap; a destination that fell behind catches up seven days per run. The first run after a destination is created covers only the most recent hour or day. A Full Export walks your history in seven-day chunks, up to two years back, and can be started once every 30 days per destination. It reads what is still in your live account, so texts, replies and link clicks older than 180 days are not available. Failed chunks can be retried without restarting the batch. A Full Export finishes with a short catch-up run covering changes made while it ran. If you already export: projects are now exported in every status and opt-outs are exported again when reversed, but an existing destination only receives a project or opt-out once that row changes after November 20, 2026. Projects that are not completed and have not changed since, and START replies from before that date, reach you only through a Full Export. Run one Full Export after November 20, 2026 to pick them up.

File layout

This section and the two after it describe the files an S3-compatible destination receives. A Snowflake destination has no files: see Tables and how they stay current. Files land under your prefix in a Hive-style path that Athena, Glue, Snowflake, Databricks, BigQuery and DuckDB discover without configuration:
  • Parquet with SNAPPY compression. Files rotate every 100,000 rows (part-000, part-001, …).
  • The year, month, day and hour folders and the timestamp in the file name come from the start of the data window the file covers, in UTC, not from when the job ran. A catch-up run or a manual backfill lands under the days its rows belong to. Files written before November 20, 2026 keep their old names and folders, which were dated by when the job ran.
  • File names depend only on the job, so a retried run overwrites the files its failed attempt wrote instead of leaving a second copy. The one exception: if a retry writes fewer part files than the failed attempt did, a higher-numbered part from the failed attempt can remain. Removing duplicates makes that harmless.
  • The organization id in the file name is the organization that owns the destination and runs the export, even when the rows belong to sub-organizations. Each row carries its own organization_id.
  • A window with no rows writes nothing. The job still appears in the history with No rows.
  • Rows inside a file are not ordered.
  • Timestamps are Parquet TIMESTAMP_MILLIS in UTC. Phone numbers are E.164 (+15555550123); short-code senders appear as the short code itself (471016).
  • Money columns are Parquet DECIMAL(18,6) in US dollars. They hold the total amount charged to you.
  • Columns marked JSON hold a JSON value stored as text (an array of ids, or an object). Parse it in your warehouse.
  • The v1 prefix is the schema version. New columns are added at the end; existing columns are never renamed or removed within a version.

Removing duplicates

Some datasets re-send a row when it changes, so a window can contain a row you already have. A recently changed row can appear in more than one run (up to about an hour of overlap). Keep one row per key, the newest. The Key and Rows change columns of the dataset catalogue give the key and tell you which datasets need this. Where a row changes, keep the one with the greatest updated_at; where rows never change, any copy will do. Reference datasets (tags, contact lists, brands, campaigns and organizations) are different: every run writes the complete table. Read the files from the latest run only, instead of combining every run. A Snowflake destination does all of this for you.

Datasets

Dataset catalogue

There are eighteen datasets. A new destination starts with only the first four ticked, because the others hold names, donations and billing. You can tick more when you create the destination or edit it later. An existing destination keeps exactly the datasets it already had.

Brands and datasets without a brand

A destination can be limited to specific brands. Datasets that belong to a brand follow that limit: message_events, project_summary, opt_out_events, click_events, survey_responses, contact_lists, brands and campaigns. Datasets that belong to the whole organization have no brand, so a destination limited to specific brands does not receive them: the email datasets, people, contact_tags, donations, billing_ledger, tags and organizations. A destination that excludes some brands, rather than limiting to specific ones, still receives them.

What is not removed

Exports add and update rows. They do not carry deletions, in S3 or in Snowflake. Each of these stays in the files and tables already delivered:
  • a person removed or erased in your account, for example after a deletion request;
  • a tag removed from a person;
  • an email compliance entry that was lifted;
  • a donation that was deleted.
When a person asks you to delete their data, apply that request to your own copy as well: we cannot reach into files or tables we have already delivered. In S3 files, a reference row that was deleted stays in the files already written, and soft-deleted reference rows arrive as a changed row with a deleted_at time. In Snowflake, the reference tables are replaced on every run for the organizations the destination covers, so a deleted row disappears. If you later remove a sub-organization from a destination’s scope, its rows stay in the Snowflake reference tables, because runs no longer cover it; delete them yourself if you no longer want them. Data already delivered is yours to manage.

Which rows come again

  • message_events and email_messages are sent again when they change. An email message is re-sent whenever it changes, however long after it was sent: a delivery, open, click, reply, unsubscribe or its cost.
  • people and donations are change feeds: a row is sent again when it changes, for example when a person’s details are updated or a donation is refunded. Rows already in your account when you tick a dataset arrive through a Full Export.
  • Rows in a change feed are read from an hour before the window starts, so a recently changed row can arrive in two runs.
  • Reference datasets send every row on every run, so a Full Export writes them once instead of once per week.

message_events

One row per outbound and inbound message. A message is exported in the window that holds its last change, and again whenever its status changes later, as long as the change lands within 3 days of the message being created (for example a delivery receipt that arrives the next day). Keep the row with the greatest updated_at per message_id. url_clicked_at, url_click_count and url_unique_clicks are a snapshot from the message’s last status change. A click does not export the message again, so use click_events for clicks.

project_summary

The current state of every project, in any status: draft, scheduled, sending, paused, completed, cancelled and deleted, plus triggered projects, which never complete. A project is exported again in every run in which it changed, so counters and cost keep moving after the first row. A project that is sending right now is included with its current values, so the same project can appear in two consecutive runs. Keep the row with the greatest updated_at per project_id.

opt_out_events

One row per opt-out. A START reply marks the same row inactive, and a later opt-out marks it active again, and each change exports the row again. A change made in the last few minutes is included with its current values, so the same opt-out can appear in two consecutive runs. Keep the row with the greatest updated_at per opt_out_id. A number is opted out when that row has is_active set to true.

click_events

One row per click on a tracked link, in the window that holds the click. Rows never change. Raw IP addresses are included in ip_address, so treat this dataset as personal data in your warehouse.

survey_responses

One row per answer to a survey question, with the option it matched. Rows never change. A later answer to the same question is a new row. Use message_id and message_created_at to find the reply in message_events, which carries the phone number and contact id.

email_messages

One row per email to one recipient. A row is exported in the window that holds its last change, and again whenever it changes: a delivery, open, click, reply, unsubscribe or its cost, at any age. An email still waiting to send is exported once it leaves the queue. Keep the row with the greatest updated_at per email_message_id.

email_clicks

One row per click on a link in an email, in the window that holds the click. Rows never change. Unlike click_events, this dataset has no IP address.

email_campaigns

The current state of every email campaign, in any status, including triggered campaigns. A campaign is exported again whenever it, its content or its sender changes, so counters and cost keep moving. The email body is not included. Keep the row with the greatest updated_at per email_campaign_id.

email_suppressions

Each address your organization can no longer email, with the reason and where it applies. A changed reason is exported again. A suppression that is lifted is removed from your account but not from files already delivered. Addresses that are blocked platform-wide rather than by your organization are never included. Keep the row with the greatest updated_at per suppression_id.

people

Everyone you have texted, emailed or tagged, one row per phone number or email address in your organization. A person is exported again when their row changes. A person who is deleted from your account, for example after an erasure request, is not removed from files already delivered. Keep the row with the greatest updated_at per person_id.

contact_tags

One row per tag applied to a phone number or email address. Rows never change. A tag removed and applied again is a new row with a new tagged_at, and both are kept. A removed tag is not carried as a removal, so it stays as applied in files and tables already delivered.

donations

Each gift received through your connected fundraising pages. A donation is exported again when it changes, for example when it is refunded, when a refund is reversed, or when its donor details are erased. Keep the row with the greatest updated_at per donation_id.

billing_ledger

Every charge, payment, refund and adjustment on your account balance, as the billing ledger page shows them. Rows never change. Amounts are signed: usage is negative and payments are positive. Amounts are the total charged; there is no breakdown of fees inside a charge.

tags

Every tag, and for an activity tag the activity and period it follows. A reference dataset: every run writes every row. A deleted tag is not removed from files already delivered.

contact_lists

Every contact list, with its contact and email counts. A reference dataset: every run writes every row. A list that is deleted is exported with a deleted_at time.

brands

Every brand, with its legal name, entity type and registration status. A reference dataset: every run writes every row. Tax ids, contact people and addresses are not included.

campaigns

Every registered texting campaign, with its brand, use case and status. A reference dataset: every run writes every row.

organizations

The organizations the export covers: your own, plus the sub-organizations you included. A reference dataset: every run writes every row.

Reading the files in S3

These notes are for S3-compatible destinations. If you use Snowflake, the tables are already loaded.
  • Athena or Glue: crawl the v1/ prefix or create a table with partition projection on schema_type, year, month, day and hour.
  • Snowflake: to load the bucket yourself, create an external stage on it and use COPY INTO or Snowpipe auto-ingest. A Snowflake destination does this for you without a bucket.
  • BigQuery: create an external table over the v1/ prefix with Hive partitioning.
  • Databricks: point Auto Loader (cloudFiles) at the schema_type=... prefix.
  • DuckDB: SELECT * FROM read_parquet('s3://bucket/prefix/v1/schema_type=message_events/**/*.parquet').
Before reporting, remove duplicates on each dataset’s key, and for reference datasets read the latest run only.

Job history

The Export jobs tab lists every run with dataset, status, rows, size and duration. Jobs retry up to three times on transient failures; a job that keeps failing shows the provider’s error message. On an S3-compatible destination a failed attempt can leave part files in your bucket; the retry overwrites them, and removing duplicates covers the rare leftover. Data already delivered to your bucket or your Snowflake account is yours to manage, including honoring a deletion request from a person whose data it holds. Deleting a record in your account does not delete files or table rows we have already written.