Apache Iceberg
Apache Iceberg is an Apache-licensed open table format, created at Netflix and now adopted by competing commercial platforms, that separates a table's definition from the engines that read it.
What it is
Iceberg is an open-source table format, licensed by the Apache Software Foundation and governed as an Apache project; it is not a vendor product, though the commercial platforms that read it standardised on it rather than replacing it. It originated at Netflix as a rethinking of how Hive tables were partitioned and tracked, and the specification, implementations and catalogue integrations are published openly. A table's definition lives in metadata rather than in a directory layout: hidden partitioning, partition evolution, serializable isolation, branching and tagging, and advanced filtering are properties of the table itself. Several engines read the same table without conversion, and the project lists integrations across Spark, Flink, Trino and a long tail of other systems.
What it gives the knowledge layer
Three things a knowledge layer usually has to build itself come with the format. Hidden partitioning means a query filters on an ordinary business column and the engine derives the partition value, so a reader does not need to know the physical layout; partitioning can later change without rewriting the table. Schema evolution lets columns be added, dropped, renamed or reordered as metadata changes, and the same machinery applies to partitioning. Snapshot isolation keeps concurrent writers from seeing each other's partial state. Table identity, however, lives in a catalogue — the documentation treats catalogues as a required component — which makes the catalogue an entity-resolution problem in miniature: the same logical table can be registered in two catalogues under two names.
Where it is documented
The project site is iceberg.apache.org. The latest documentation carries the introduction, concepts, catalogue properties and engine integrations; evolution and partitioning document how a table's shape is defined and changed, schemas covers column types and identifiers, and reliability explains atomic commits of table metadata. The normative table specification is published separately and is what a new implementation must satisfy.
What it does not solve
Iceberg governs a table's structure and history, not what the table means. Column names remain local, no term carries a definition, and nothing in the format maps an Iceberg column to a concept in an ontology or a term in a taxonomy. Versioned snapshots fix what the data said at a point in time; they say nothing about whether the vocabulary describing it is still current. Catalogues add identity and, in some implementations, access control, but a catalogue is a registry of tables rather than a model of the domain: it can tell an agent that a table exists, not what its rows refer to.
FAQ
No. It is an Apache-licensed open-source project. The commercial platforms that read Iceberg tables do so through their own implementations of the open specification.
Nothing visible. A query filters on an ordinary column and the engine derives the partition value, so callers do not need to know how the table is physically laid out.
The table format is open and any conforming engine can read it, but identity and access rules live in whichever catalogue registered the table, and catalogues are separate systems.
No. It gives tables a defined structure and history. Definitions, synonyms and mappings between systems belong to the vocabularies layered above the table.
Sources
- Apache Iceberg documentation — introduction, concepts, catalogue properties and the list of engine and tool integrations.
- Evolution — Apache Iceberg — schema evolution, hidden partitioning and partition evolution.
- Partitioning — Apache Iceberg — how partitioning is expressed and changed without rewriting the table.
- Reliability and performance — Apache Iceberg — atomic commit of table metadata and isolation between concurrent writers.
- Apache Iceberg table specification — the normative definition of the table format and its metadata.