TraceLake architecture

TraceLake architecture An architecture diagram generated by Archify. Kafka · JSON event topics · Architecture component Kafka JSON event topics Ingestor · decode · partition · Architecture component · StatefulSet Ingestor decode · partition StatefulSet Local WAL · Parquet segments · Architecture component Local WAL Parquet segments Compactor · merge · index · expire · Architecture component · Deployment Compactor merge · index · expire Deployment Azure Blob · Iceberg tables + sidecars · Architecture component Azure Blob Iceberg tables + sidecars Iceberg REST catalog · e.g. Lakekeeper · Architecture component Iceberg REST catalog e.g. Lakekeeper Query gateway · DataFusion + pruning · Architecture component · StatefulSet Query gateway DataFusion + pruning StatefulSet OIDC provider · token issuer · Architecture component OIDC provider token issuer Clients · HTTP · PostgreSQL wire · Architecture component Clients HTTP · PostgreSQL wire External engines · Trino · Spark · DuckDB · Architecture component External engines Trino · Spark · DuckDB consume stage upload Parquet append commit rewrite + sidecars rewrite · expire poll snapshot range reads query fetch JWKS type=rest Legend Frontend Backend Database Cloud Security Message bus External

Ingest

  • • One Kafka consumer per dataset
  • • Events staged in a local Parquet WAL, then uploaded

Your data

  • • Open Iceberg tables in your own bucket
  • • Any Iceberg REST engine reads the same tables

Query

  • • HTTP (RSQL or JSON AST) and optional PostgreSQL wire
  • • Bloom and full-text sidecars prune files before scan

Security

  • • OIDC bearer token on every request
  • • Dataset-level access, deny by default