> ## Documentation Index
> Fetch the complete documentation index at: /llms.txt
> Use this file to discover all available pages before exploring further.

# Loading into PyTorch

> Filter the metadata, then decode only the clips you picked.

The repository includes a reference `Dataset` for video datasets, `helmet_cam.py`. It filters the Parquet metadata first and decodes only the matching clips.

## Install

```bash
pip install torch pyarrow av fsspec s3fs
```

## Use

```python
import pyarrow.compute as pc
from torch.utils.data import DataLoader
from helmet_cam import HelmetCamClips

clips = HelmetCamClips(
    "s3://your-bucket/datasets/sea-helmet-cam/metadata/",
    filter=(pc.field("city") == "Jakarta") & (pc.field("weather") == "rain"),
    num_frames=16,
    size=(224, 224),
)
loader = DataLoader(clips, batch_size=8, num_workers=8, shuffle=True)

for batch in loader:
    video = batch["video"]  # uint8, [8, 16, 3, 224, 224]
```

## Parameters

| Parameter | Default | What it does |
| - | - | - |
| `metadata` | required | Path or URI of the Parquet metadata directory. |
| `filter` | `None` | PyArrow expression applied to the metadata. |
| `num_frames` | `16` | Frames sampled evenly across each clip. |
| `size` | `(224, 224)` | Height and width every frame is resized to. |
| `transform` | `None` | Callable applied to the video tensor. |
| `storage_options` | `None` | Passed to fsspec, for credentials or a custom endpoint. |
| `decode` | `True` | Set to `False` to get metadata only. |

## Splitting train and test

Split on `driver_id`, so footage from one contributor never appears on both sides:

```python
import pyarrow.compute as pc

held_out = ["drv_00017", "drv_00203"]
train = HelmetCamClips(path, filter=~pc.field("driver_id").isin(held_out))
test = HelmetCamClips(path, filter=pc.field("driver_id").isin(held_out))
```
