Exports

Exports to your own storage

A scheduled export can also write every table to your own bucket, where your warehouse or BI tool picks it up.

Setting it up

  • In HailMate: Settings → Export data → Scheduled → Your storage → Add storage (owners and admins).
  • Amazon S3, Google Cloud Storage (an HMAC key), Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, or any other S3-compatible storage. The key needs to write to the bucket (s3:PutObject) and, to tidy a day that ran twice, to read and delete in its folder.
  • We write a small check file (<folder>/_hailmate_check.txt) before saving, and keep the secret where only our server reads it.
  • Then pick the storage on a scheduled export. CSV of all time is a full delivery.

The layout

The layout never changes under you: point a load job at it once.

Text
<folder>/schedule-<id>/<table>/dt=<YYYY-MM-DD>/<table>-<run>-<NNN>.csv
<folder>/schedule-<id>/_manifests/dt=<YYYY-MM-DD>.json
<folder>/schedule-<id>/_manifests/latest.json
PartWhat it is
schedule-<id>Each scheduled export has its own folder, named by its id: renaming the schedule never moves it, and two schedules (one per location) never write over each other.
<table>One folder per table, its name in lower case with dashes: jobs, payments, time-in-stage.
dt=<YYYY-MM-DD>Your company’s day the run was made: a date partition BigQuery, Athena, Snowflake and Databricks read as one.
<run>-<NNN>The run, then the file’s number: a big table comes in several files, each with the header row.

One run per day

A day’s folder holds exactly one run. When a schedule runs twice in a day (Send now, or a retry), the second run writes its own files, then removes the first run’s, after its manifest is written. A load that reads a day’s folder never reads it twice.

The manifest

Each run writes what it wrote: every file, its rows, its size and its columns. Read latest.json (or the dated one) rather than listing the bucket.

_manifests/dt=2026-11-19.json
{
  "schedule": "Nightly — every table",
  "schedule_id": "3f2a9c1e-77aa-4b2b-9e00-123456789abc",
  "run_id": "9e1b44d0-5a6b-4c7d-8e9f-0a1b2c3d4e5f",
  "made_at": "2026-11-19T07:02:11Z",
  "day": "2026-11-19",
  "format": "csv",
  "files": [
    {
      "table": "Jobs",
      "key": "hailmate/schedule-3f2a9c1e/jobs/dt=2026-11-19/jobs-9e1b44d0-001.csv",
      "part": 1,
      "rows": 6652,
      "bytes": 4183022,
      "columns": [
        "Job",
        "Job number",
        "Location",
        "Stage",
        "…"
      ]
    },
    {
      "table": "Payments",
      "key": "hailmate/schedule-3f2a9c1e/payments/dt=2026-11-19/payments-9e1b44d0-001.csv",
      "part": 1,
      "rows": 6677,
      "bytes": 1920443,
      "columns": [
        "Paid by",
        "Amount",
        "…"
      ]
    }
  ]
}

Loading it

BigQuery

Shell
bq load --source_format=CSV --skip_leading_rows=1 --autodetect \
  --hive_partitioning_mode=AUTO \
  --hive_partitioning_source_uri_prefix=gs://acme-exports/hailmate/schedule-3f2a9c1e/jobs \
  hailmate.jobs "gs://acme-exports/hailmate/schedule-3f2a9c1e/jobs/dt=2026-11-19/*.csv"

Snowflake

Text
COPY INTO hailmate.jobs
  FROM @hailmate_stage/schedule-3f2a9c1e/jobs/dt=2026-11-19/
  FILE_FORMAT = (TYPE = CSV SKIP_HEADER = 1 FIELD_OPTIONALLY_ENCLOSED_BY = '"');

Power BI and Excel

Load the files into your warehouse and point Power BI at it, or, for one report that refreshes by itself, use a live link (Getting all of it).