Compactor merge sequence

Compactor merge sequence A sequence diagram generated by Archify. load table, count small files GET small files, one at a time PUT merged file PUT sidecars append new files replace small files with merged committed expire old snapshots delete old files Select Merge + upload Commit Later, on own timers Ingestor · another pod · Sequence participant Ingestor another pod Compactor · one pod · Sequence participant Compactor one pod Azure Blob · your storage · Sequence participant Azure Blob your storage Iceberg catalog · your catalog · Sequence participant Iceberg catalog your catalog Legend merge step return outside the commit other writer

Order

  • • The merged file is in storage before the commit
  • • A sidecar that fails to upload stops nothing

Concurrency

  • • An append first: the commit is sent again
  • • Two compactors, same files: the second fails

Cleanup

  • • The commit deletes nothing
  • • Old files go when unreferenced and 24 hours old