StarRocks backup · design note

Fast undo comes from small, targeted backups, not from rewinding the bucket

Engineers ship fast with AI, so data mistakes will happen. Keep point-in-time restore, cut the 30-day daily snapshots, and add the cheap layers that actually undo a mistake in minutes.

TrueProfit prod · AWS Backup plan read 2026-10-23 · confidence MEDIUM until the checks at the end are run

What is kept today bucket starrock-data ~0.5 TB backup storage ~10.8 TB-months · ~$650–1,185 / month Retention continuous (point-in-time) 2 days · keep daily snapshot 30 days · 31 recovery points · cut to ~7
Fig. 1StarRocks keeps rewriting files, so each daily snapshot holds a day of churn for 30 days. Most of the cost sits there, not in the 2-day point-in-time window (MEDIUM: split the cost per rule before changing it).

Why rewinding fails

Rewinding the whole bucket is not an undo button

Mismatch

Files and metadata drift apart

Data files live in S3, but table metadata lives in the FE nodes. Rewind only the files and the cluster likely no longer matches them.

Too wide

It rewinds every shop

A bug usually breaks a few tables or shops. A bucket rewind also throws away every correct write since that moment.

Slow

Hours, not minutes

The December 2025 restore into starrock-data-v2 took 7 jobs on 16–17 Dec, 2 of them failed. No record says whether StarRocks ran on it afterwards.

Fig. 2Point-in-time restore is the disaster plan. It is the wrong tool for an engineer’s mistake.

The design

Five layers, cheapest first

Prevent

No DROP/TRUNCATE rights for app and AI users. Prod DDL only through a reviewed migration. Try risky changes on a dev copy first.

mistake never lands
Dropped by mistake

StarRocks recycle bin: RECOVER TABLE …. Raise catalog_trash_expire_second from 1 day to 3–7 days.

minutes
Bad migration or backfill

BACKUP SNAPSHOT of the affected tables automatically before every prod DDL or backfill; keep 7 days; restore that table only.

10–30 minutes
Wrong numbers for some shops

Rerun the backfill from the source (Mongo, Shopify) for those shops. StarRocks data can be rebuilt from its source.

fast for a few shops · slow for all (8 at a time)
Whole cluster broken

Point-in-time restore (2 days) plus ~7 daily snapshots, with FE metadata backed up at the same moment.

hours
Fig. 3Each layer handles one kind of mistake. Bars show how fast each one brings the data back.

MongoDB is the real source of truth, so point-in-time restore matters most there. Whether Atlas continuous backup is on is not yet checked.

Side note

Moving ad-insights to StarRocks saves money only if Mongo drops a tier

Measure first

Share of Mongo

Atlas bills by cluster tier. If ad-insights is a small share of storage and working set, moving it saves nothing.

Second-order

More StarRocks churn

Facebook updates today’s row until day end. In StarRocks that means primary-key upserts, more compaction, more S3 requests and more backup churn.

Real upside

Duplicates fixed by design

A primary-key table cannot hold two rows for one key, which removes the cause of the TPCS-5375 duplicate docs.

Before changing anything

Five read-only checks, then one drill

  1. Is StarRocks in shared-data mode, which version, and does BACKUP/RESTORE work in that mode?
  2. Current value of catalog_trash_expire_second.
  3. Is FE metadata backed up anywhere?
  4. Did the December 2025 restore into starrock-data-v2 actually run StarRocks?
  5. Is Atlas continuous backup on for the production cluster?
  6. Drill: restore point-in-time to a test bucket, start a dev StarRocks on it, count rows. If it fails, today’s backup is a false sense of safety. Change retention only after it passes.

Sources

AWS Backup plan and recovery points, prod, read 2026-10-23 · AWS Cost Explorer, Jun–Oct 2026 · S3 BucketSizeBytes (CloudWatch) · vault note product-knowledge/03-engineering/investigations/2026-10-23-technical-research-starrocks-fast-undo-backup.md.