Delta Lake
Delta Lake is an open-source storage framework, originating at Databricks and now governed under the Linux Foundation, that adds transactional guarantees and versioned reads to tables of Parquet files.
What it is
Delta Lake is an open-source project rather than a vendor product, although it originated at Databricks: the project site presents it as an open-source storage framework and its governance sits under Linux Foundation Projects. It works by placing a transaction log beside a directory of data files. The protocol specification defines a _delta_log of JSON commits over Parquet data files, and the log is what turns a folder of files into a table with a defined current state. On that base the documentation lists ACID transactions with serializable isolation, schema enforcement that rejects bad records at ingest, time travel to earlier versions, and merge, update and delete. Delta Universal Format lets Iceberg and Hudi clients read the same tables.
What it gives the knowledge layer
A table becomes something an agent can reason about over time. A point-in-time read asks what the table said at a given commit and returns an answer that can be reproduced later, which is the storage-layer analogue of a memento: both make an earlier state addressable by a stable reference. The parallel is not an equivalence. A memento is a representation of a resource, negotiated over HTTP and addressed by its own URI; a Delta version is a commit in a transaction log, readable only through an engine that understands the log. Schema enforcement and evolution add a second axis: the table's shape is versioned alongside its rows, so a change of column can be dated.
Where it is documented
The project site is delta.io. The Delta Lake documentation states the guarantees directly — ACID transactions, scalable metadata handling, schema enforcement, time travel, upserts and deletes. The Delta protocol specification defines the log format, the data-file layout and the mapping of Delta types to Parquet types; that is the document a second implementation has to satisfy, and therefore the one to read when interoperability is the requirement.
What it does not solve
Versioned storage does not make the meaning current. A time-travelled read of a customer table returns those rows as of that commit; it says nothing about whether the definition of an active customer changed in the same period, and the log records change to the data, not to the vocabulary describing it. Delta Lake also decides nothing about permissions: the transaction log has no model of who may read a table. Access control is imposed by whatever catalogue or engine sits above the files.
FAQ
No. It originated at Databricks and is now an open-source project governed under Linux Foundation Projects, with a published protocol specification and more than one engine implementation.
No. Both make an earlier state addressable, but a memento is a representation of a resource negotiated over HTTP with its own URI, while a Delta version is a commit in a transaction log, readable only through an engine that understands the log.
For the data, yes. A version can be re-read, which supports audits, rollbacks and repeatable analysis. It does not freeze the definitions, code or permissions that were in force at that time.
No. The transaction log carries no model of access. Permissions come from the catalogue or compute engine in front of the files, so the same table can be open in one workspace and restricted in another.
Sources
- Delta Lake — project site; describes Delta Lake as an open-source storage framework and lists its key guarantees.
- Delta Lake documentation — introduction — ACID transactions, schema enforcement, time travel, upserts and deletes.
- Delta Transaction Log Protocol — the log format, data-file layout and Delta-to-Parquet type mappings that a conforming implementation must follow.