Ingestor flow

Ingestor flow A data-flow diagram generated by Archify. 01 / Kafka 02 / Consume + guards 03 / Local WAL 04 / Upload + commit 05 / Your storage + catalog Backpressure · WAL disk watermark · 02 / Consume + guards Backpressure WAL disk watermark Kafka topic · one per dataset · 01 / Kafka Kafka topic one per dataset Consumer · decode · partition key · 02 / Consume + guards Consumer decode · partition key Open segment · active_*.parquet · 03 / Local WAL Open segment active_*.parquet Committer · batch per interval · 04 / Upload + commit Committer batch per interval Iceberg catalog · new snapshot · 05 / Your storage + catalog Iceberg catalog new snapshot Boot recovery · reconcile leftovers · 02 / Consume + guards Boot recovery reconcile leftovers Rotated segment · ready_*.parquet · 03 / Local WAL Rotated segment ready_*.parquet Uploader · scan every 5 s · 04 / Upload + commit Uploader scan every 5 s Azure Blob · data file + bloom · 05 / Your storage + catalog Azure Blob data file + bloom poll append row rotate commit offset scan PUT report append pause / resume at boot Legend record path guard boot only WAL file offset commit

Rotation

  • • At 64 MiB or 5 minutes, or on a rebalance
  • • The Kafka offset is committed at rotation

Durability

  • • One Iceberg append per dataset per interval
  • • The local file is deleted only after that append

Guards

  • • Consumers pause at 80 % disk, resume below 75 %
  • • Recovery deletes files that Kafka will redeliver