Remove Duplicate Lines

Why deduplicate locally instead of in a web service

The lists people most often need to dedupe are exactly the ones they should not paste into a random website that posts them to a server: email subscriber exports, customer names, order IDs, survey responses, server logs with IP addresses. This page does the whole job on your device — the logic is a small JavaScript module you can read, there are no network requests, and the Open .txt file button reads the file with the browser's own FileReader, so the bytes never leave your machine. Disconnect from the internet and it keeps working, which is the honest test of any "client-side" claim.

Worked example: how the options interact

Take this five-line list, exported from two systems that disagreed about formatting:

With no options on, only lines 1 and 5 are byte-identical, so you get 4 lines out. Turn on case-insensitive and line 2 joins the group (3 out). Add trim before compare and line 3 joins too — 2 lines out: alice@example.com and bob@example.com. Note what the options do not do: the kept line is emitted with its original text. Keep-first emits line 1's exact casing; switch to keep last and the survivor is line 5 (same text here, but in a list where later rows were corrected, keep-last is how you prefer the newer entry).

Line endings: the silent diff-wrecker

Windows tools end lines with CRLF (\r\n); Unix tools with LF (\n). A dedupe tool that silently rewrites one into the other makes every single line of your file "changed" in the next diff or version-control commit. This tool counts both styles in your input, keeps the dominant one on output, and preserves a trailing line break if the input had one — the label under the counters tells you which style it detected. Mixed-ending files (common after copy-pasting between platforms) are still compared correctly line by line.

One tool for the dedupe-and-sort family

Dedupe, trim, drop-blank-lines and sort are usually wanted together, so they live on one page instead of four thin ones. Deduplication always runs first; the optional sort then orders the survivors A→Z or Z→A, comparing case-insensitively with a deterministic tie-break so repeated runs give identical output. Leave sort off for logs and playlists where order is information. The counters above the result update on every keystroke: lines in, unique lines out, and how many were removed.

Honest limits

Everything happens in a textarea, so the practical ceiling is your browser's memory — tens of thousands of lines are instant (each line is hashed once, not compared against every other line), but a multi-hundred-megabyte log file belongs in a command-line tool like sort -u or awk '!seen[$0]++'. "Empty" for the ignore-empty option means blank or whitespace-only; a line containing a single dot is not empty. And duplicates are detected per line — the tool does not find near-duplicates, fuzzy matches, or repeated words inside one line (for word-level analysis, see the word counter's keyword-density table).