Compactor flow

Compactor flow A data-flow diagram generated by Archify. 01 / Triggers 02 / Select 03 / Merge 04 / Publish + maintain 05 / Your storage + catalog Schedule · daily 02:00 UTC · 01 / Triggers Schedule daily 02:00 UTC Backlog tick · every 15 minutes · 01 / Triggers Backlog tick every 15 minutes Detect backlog · count small files · 02 / Select Detect backlog count small files Stream merge · sort · build sidecars · 03 / Merge Stream merge sort · build sidecars Expire snapshots · every 60 minutes · 04 / Publish + maintain Expire snapshots every 60 minutes Upload + commit · files, then rewrite · 04 / Publish + maintain Upload + commit files, then rewrite Orphan cleanup · every 60 minutes · 04 / Publish + maintain Orphan cleanup every 60 minutes Iceberg catalog · new snapshot · 05 / Your storage + catalog Iceberg catalog new snapshot Azure Blob · data files + sidecars · 05 / Your storage + catalog Azure Blob data files + sidecars lower bar check flagged merged files 1. PUT 2. rewrite retire delete Legend compaction pass maintenance loop schedule trigger

Triggers

  • • Tick: partitions with more than 100 small files
  • • Schedule: partitions with 2 or more small files

Merge

  • • Small is under 32 MB; output targets 128 MB
  • • The full-text index is built here, not at ingest

Safety

  • • A file is uploaded before a commit refers to it
  • • Deleted only when unreferenced and 24 hours old