AI Dataset Deduplicator
Remove exact and near-duplicate rows from fine-tuning datasets.
Input JSONL
Drop a file here
or choose a file — it never leaves your device
Deduplicated JSONL
Removed records — 2
- Line 2 — Duplicate after normalisation: {"prompt":"what is rag?","completion":"Retrieval augmented generation."}
- Line 3 — Duplicate after normalisation: {"prompt":"What is RAG?","completion":"Retrieval augmented generation."}
About the AI Dataset Deduplicator
Load a JSONL dataset and strip duplicates by exact text, case and whitespace insensitive match, or fuzzy near-duplicate detection using word shingles with an adjustable similarity threshold. Compare on any field or the whole record, and export the cleaned file.
Examples
Dedupe prompts
4 rows, 2 duplicate promptsOutput
3 unique rows keptKeyboard shortcuts
- Copy the main outputCtrl / ⌘ + Shift + C
- Download the resultCtrl / ⌘ + S
- Share this toolCtrl / ⌘ + Shift + S
- Reset the inputsAlt + R
- Open the tool search paletteCtrl / ⌘ + K
Related tools
AI Dataset Statistics Analyzer
AI & LLM
Profile a JSONL training set: token stats, roles, duplicates and cost.
Chat Dataset Converter
AI & LLM
Convert between OpenAI chat, ShareGPT, Alpaca and prompt/completion.
AI PII & Secret Redactor
AI & LLM
Strip emails, keys, tokens and personal data before prompting an LLM.
AI Token Counter
AI & LLM
Count exact LLM tokens for GPT, Claude and Gemini prompts.
AI Tool Schema Builder
AI & LLM
Build function-calling schemas for OpenAI, Claude, Gemini, MCP and the AI SDK.
AI Tool Schema Converter
AI & LLM
Convert function-calling schemas between OpenAI, Anthropic, Gemini and MCP.
Frequently asked questions
Version 1.0.0 · Updated 2026-08-15 · Runs entirely in your browser