HDT: RDF compressed for publishing at scale
HDT is a binary format for publishing and exchanging large RDF datasets, built so a client can read the parts it needs without processing the whole file.
What it is
HDT (Header-Dictionary-Triples) is a binary format for publishing and exchanging RDF data at large scale. A dictionary holds all the vocabulary in a highly compressed, quickly searchable form, a triples component holds the graph structure in compressed form, and a recommended header carries metadata.
What it gives the knowledge layer
It gives the knowledge layer a distribution artifact whose size does not decide the project: a compressed reference dataset can be shipped where the same graph in text would be impractical. The header carries statistics about the data, and the format permits accessing parts of the file without processing all of it.
Where the spec lives
HDT was published as a W3C Member Submission in March 2011, one of five documents covering the format, a VoID extension, an RDF Schema for header descriptions, related technologies, and an implementation. It is a submission, not a Recommendation.
What it does not solve
HDT is about distribution size, not modelling. It encodes an existing graph and says nothing about whether the model is right or entities are duplicated. It is also read-oriented: what you distribute is a snapshot, and nothing in the format keeps the artifact current.
FAQ
No. HDT was published as a W3C Member Submission in 2011, and W3C states that publication indicates no endorsement of the content. It is a submitted format, not a Recommendation.
Header, Dictionary and Triples. The dictionary holds all the vocabulary in a form permitting rapid search and high compression, the triples component holds the graph structure compressed, and the header carries metadata.
No. HDT is for publishing and exchanging RDF data at large scale, aiming at compactness and on-demand indexed access to parts of the file. The usual pattern is to archive it and load it into a query engine to query with SPARQL.
When the problem is distribution size rather than authoring. Turtle is text a person reviews and edits; HDT is a binary artifact you ship to a client that reads only the parts it needs.
Sources
- Binary RDF Representation for Publication and Exchange (HDT) (W3C Member Submission, 2011)
- RDF 1.1 Concepts and Abstract Syntax (W3C Recommendation)