[ Built with Jev · Data and databases ]
jev-curate: sift training datasets
A Rust pipeline for filtering and scoring very large synthetic and pre-training datasets.
By Akash Priyadarshi (@AkashPriyadarshii on GitHub) ·
How it uses Jev
Streams Parquet and JSONL rows through Jev's typed judgements; the author reports 24 rows a second.
It's an example of a wider pattern: see Jev for research and news filtering for when it fits, the questions to ask and what it costs.
Summary written by JevHub. Figures are the author's own; we haven't reproduced them. The project belongs to its author.
