Apache Parquet
Apache Parquet is an open-source, column-oriented data file format, licensed under the Apache License, and the storage format that commercial analytics platforms read and write by default.
What it is
Parquet is open source and Apache-licensed, maintained as an Apache Software Foundation project: it is not a database, a service or a vendor product. A Parquet file is a self-contained binary file that any conforming reader can open. The project describes it as an open-source, column-oriented data file format designed for efficient data storage and retrieval, with compression and encoding schemes for bulk data. It keeps the format specification separate from the Java implementation, which is why a second implementation can exist at all.
What it gives the knowledge layer
Structure travels with the data. Every file carries its own schema — physical types, logical types such as dates and decimals, nested groups, and column-level statistics — so a reader reconstructs the layout without consulting an external registry. Columnar storage means a query that needs three columns out of two hundred reads three, and column statistics let an engine skip row groups rather than scan them. This is the layer enterprise data actually sits in, and a column scan is the cheapest way to ground an answer in it. The schema is a type system, not a vocabulary: it says a field is a string of a given length, not that it holds a customer identifier.
Where it is documented
The format is documented at parquet.apache.org. The overview states the design and lists implementations across languages and tools; file format and metadata document the file structure and the schema and statistics a file carries. The site notes that the ecosystem is varied and that not every implementation supports the same features, so format conformance across writers and readers is not uniform — a detail that matters when two platforms claim to exchange Parquet.
What it does not solve
Parquet solves storage, not meaning. Two Parquet files can describe the same customer differently — different column names, units and identifier schemes — and neither file can detect the disagreement. The format carries no shared vocabulary, no entity identity that survives across files, no provenance, and no statement of who wrote a value or whether it is still current. A schema is enforced within a file, not between files. Reconciling names to concepts is work above the storage layer, in ontologies, taxonomies and provenance models such as PROV-O.
FAQ
No. It is a file format. A Parquet file is a self-contained binary file with an embedded schema, read by a client library rather than a server.
No. It is a physical type system: column names and primitive or logical types. It carries no concept identifiers, no definitions and no mappings to another system's terms.
Yes, and routinely. Column names, units, nullability and identifier formats are local to each file, and the format has no mechanism to compare two files' schemas for meaning.
No. Files carry metadata about layout and statistics, not lineage, ownership or currency. Provenance has to be recorded alongside the data by whatever writes it.
Sources
- Apache Parquet — project site; licence, sub-projects and the list of language and tool implementations.
- Overview — Parquet — defines Parquet as an open-source, column-oriented data file format and describes the specification repository.
- File format — Parquet — structure of the file format and its encoding, compression and page-level organisation.
- Metadata — Parquet — the embedded schema, logical types and column statistics a file carries.