Ingestor shutdown sequence

Ingestor shutdown sequence A sequence diagram generated by Archify. SIGTERM rotate open segments commit offsets leave group PUT ready files + bloom sidecars append, one per dataset committed delete local files exit 0 Release partitions Upload + commit Exit Kubernetes · kubelet · Sequence participant Kubernetes kubelet Ingestor · one pod · Sequence participant Ingestor one pod Local WAL · pod volume · Sequence participant Local WAL pod volume Kafka · consumer group · Sequence participant Kafka consumer group Azure Blob · your storage · Sequence participant Azure Blob your storage Iceberg catalog · your catalog · Sequence participant Iceberg catalog your catalog Legend drain step return signal or cleanup

Release

  • • Offsets are committed before the pod leaves
  • • Kafka gives the partitions to other pods at once

Finish

  • • The pod uploads and commits what it rotated
  • • The metrics endpoint stays up during all of this

Deadline

  • • The limit is ingestion.drainTimeoutSeconds
  • • After the limit, recovery finishes the work