> ## Documentation Index
> Fetch the complete documentation index at: /llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset layout

> How a delivered dataset is organised.

Every dataset, licensed or commissioned, is delivered in the same layout: files in object storage and one Parquet metadata row per sample.

```
datasets/<slug>/
  dataset.json            title, description, licence, column descriptions
  metadata/*.parquet      one row per sample
  clips/<city>/<id>.mp4   the files themselves
```

## Metadata

The metadata table is what you filter on before you load anything. Read a dataset's columns from the API:

```bash
curl https://api.datachad.xyz/v1/datasets/sea-helmet-cam \
  -H "X-API-Key: $DATACHAD_API_KEY"
```

The `schema` array in the response lists each column's name, Arrow type and description.

## Provenance columns

These columns ship in the metadata of the helmet-camera dataset:

| Column | What it holds |
| - | - |
| `consent_version` | Version of the consent agreement the contributor signed. |
| `faces_blurred` | Whether bystander faces were blurred. |
| `plates_blurred` | Whether number plates were blurred. |
| `driver_id` | Pseudonymous contributor ID. Split train and test on this column. |

## Querying the metadata

Parquet reads directly in DuckDB, Polars, pandas and PyArrow:

```python
import duckdb

duckdb.sql("""
    SELECT city, round(sum(duration_s) / 3600) AS hours
    FROM 'datasets/sea-helmet-cam/metadata/*.parquet'
    WHERE weather = 'rain'
    GROUP BY city
    ORDER BY hours DESC
""").show()
```
