Databricks
Databricks is a proprietary, vendor-controlled commercial data platform — a lakehouse built around open formats — with Unity Catalog as its governance and catalogue surface.
What it is
Databricks is a proprietary, vendor-controlled commercial platform, sold as a managed service and documented by its vendor; it is not an open-source project, even though it is built around open formats such as Parquet, Delta Lake and Iceberg. The vendor's documentation describes Unity Catalog as the unified governance layer for data and AI built into Databricks: when enabled for a workspace it enforces access control when a table is queried or a model is called, tracks lineage as assets are used, and logs activity for audit. Assets are modelled as securable objects in a three-level namespace, catalog.schema.object, covering tables, views, volumes, functions, models and services. The vendor also publishes an open-source implementation of Unity Catalog; the platform itself remains proprietary.
What it gives the knowledge layer
A catalogue is where an agent learns what exists and who may read it. Before choosing a table, an agent needs a directory of registered assets and a statement of what it is allowed to touch; Unity Catalog supplies both, with discovery through Catalog Explorer, an object hierarchy that gives every asset a stable address, lineage as assets are used, and audit logging. Volumes extend the same governance to files that are not tables, which matters because much enterprise knowledge sits in unstructured documents rather than rows.
Where it is documented
The vendor's documentation is the source of record: What is Unity Catalog? sets out the object model, access control, discovery, lineage and audit behaviour, and the product pages for Unity Catalog and the data lakehouse describe the positioning. Documentation is versioned per cloud — AWS, Azure and Google Cloud — and revised continuously. There is no specification to read against: the interface is the product.
What it does not solve
The governance metadata is a platform artefact. Permissions, lineage and audit records are held in the vendor's catalogue, and the work modelled there — a curated table, a registered model, a set of tags — does not leave in a portable form; the open formats underneath carry data, not the catalogue's decisions about it. Nor does a catalogue answer what data means. Unity Catalog can state that a table exists, who may read it and what it came from; it cannot say that two tables describe the same customer, or that a column name means something different from the same name elsewhere. Semantic agreement is work above the platform, in ontologies and taxonomies, and it has to be re-established wherever the data is copied.
FAQ
No. The platform is a proprietary, vendor-controlled service. It is built on open formats and the vendor publishes an open-source implementation of Unity Catalog, but the service itself is not open source.
It is the governance layer built into Databricks. It enforces access control on data and AI assets, tracks lineage as those assets are used, logs activity for audit, and registers assets in a three-level namespace of catalog, schema and object.
No. A catalogue is a registry of assets with permissions and lineage. It records what exists and who may use it, not what the assets mean or how the concepts they describe relate to one another.
The data files are readable elsewhere when they use open formats, but permissions, lineage and audit records are Unity Catalog state. They are not carried by the files.
Sources
- What is Unity Catalog? — Databricks documentation — the governance layer, securable objects, three-level namespace, access control, lineage and audit.
- Unity Catalog — Databricks — the vendor's product description of the catalogue and governance surface.
- Data lakehouse — Databricks — the vendor's description of the platform's architecture and its use of open formats.