fix‑encoding

The one bug it fixes — and what it refuses to touch

What it does. Almost all mojibake comes from a single mistake: text correctly stored as UTF-8 (é is the bytes C3 A9) gets read by some program as Windows-1252, so é renders as é — then the wrong text is re-saved as UTF-8, baking the damage in. This tool reverses exactly that chain, including text mangled two or three times over (café), whatever alphabet the original was written in: the same reversal that turns Café back into Café turns 日本語 back into 日本語, Привет into Привет and 😀 into 😀, and it handles Greek, Arabic, Thai and Hangul the same way. It also undoes the same mistake through the legacy CJK code pages: Japanese garbled via Shift_JIS (東京 → 譚ｱ莠ｬ), Chinese via GBK or Big5, Korean via EUC-KR.

When you'd use it. Legacy CSV/JSON/SQL dumps where the accents arrived broken, old mail archives, scraped pages, and log pipelines that keep printing ’ where an apostrophe belongs.

Opening a file, and saving the repaired one. Those are files, so you can hand one straight to the page: Open a .csv or .txt file in the header above the paste box, or drop the file anywhere on it. The browser's own FileReader reads it on your device, decodes it as UTF-8 and drops the text into the same box you would have pasted into — the same repair, the same confidence rating, the same step list, byte for byte. Nothing is uploaded; reading a local file makes no request at all. Then Save file beside Copy writes back exactly what the Repaired output box shows: those code points, encoded UTF-8, with no BOM added and no trailing newline added, and a repaired invoices.csv comes back as invoices-fixed.csv with the extension kept. Text you pasted has no name to borrow, so it saves as repaired.txt. Save is dead until there is a repair to take away, and if the browser refuses to write the file the button says Save failed and the line under it says why — it never pretends quietly. One honest catch, about line endings: the paste box is an ordinary browser textarea, and every browser normalises CR LF to LF inside one. So a Windows-style CSV opened here comes back with Unix line endings and is 1 byte shorter per line. Nothing in the save step rewrites anything — the change happens on the way in, to a paste exactly as much as to a file — but if CRLF matters to whatever reads the file next, convert it back afterwards.

What a file has to be. The ceiling is 128 KB (131,072 bytes), roughly 3,000 lines of an ordinary CSV export. The repair re-runs on every keystroke, and measured in Node on this machine a full 128 KB of mojibake repairs in about 47 ms while 128 KB of already-clean text — the slowest case, because every candidate pass is tried and rejected — takes about 220 ms. A file over the ceiling is refused by name, with its own size and the ceiling's, rather than silently cut short: half a repaired CSV is worse than none. A binary file is refused by name too — a PNG, PDF, ZIP or SQLite signature, or a NUL byte anywhere in it — because a binary rendered as text is mojibake, and this page would then be repairing damage it had just caused. And one refusal matters more here than anywhere else: mojibake is valid UTF-8 — that is precisely what makes it mojibake, because é is two perfectly legal UTF-8 characters standing where é belongs. So a file that is not valid UTF-8 is a different problem entirely: it is a file still saved in Windows-1252 (or Latin-1, or UTF-16), never re-encoded, and the fix is to re-save it as UTF-8, not to run it through here. Such a file is refused by name with the offending byte and its offset — 0x93 at offset 6, say, the Windows-1252 opening smart quote — instead of being decoded with replacement characters and handed to a repair engine as fresh, invented mojibake.

It refuses clean text. A decoding pass is applied only when it strictly reduces mojibake evidence, so correctly-encoded prose — café, 日本語, Привет, Ελληνικά, 😀, curly quotes, a lone em dash or euro sign — is returned verbatim, and running the tool on its own output is a guaranteed no-op. That caution has a flip side: ambiguous input that no repair provably improves is left alone rather than guessed at. Exactly one guess is allowed, and only inside text a decoding pass has already proven is mojibake: Ã followed by an ordinary space is read back as the no-break space it used to be. à is the bytes C3 A0, Windows-1252 renders A0 as a no-break space, and HTML forms, spreadsheets and copy-and-paste all flatten that A0 to an ordinary 20 — so without the guess, à la carte is the one everyday accent the repair leaves broken. Because it is a guess it gets its own line in the step list, it can never earn the high rating, and the tick box under the result switches it off.

Honest limits. It is a heuristic, not a proof: it handles the dominant cp1252 chain in every script and the four CJK chains above, but corruption from other code pages (KOI8-R, the wider ISO-8859 family, EUC-JP) is out of scope and left untouched. Inside the cp1252 chain one deliberate blind spot remains: é (a garbled é) and É… (a clean all-caps word followed by an ellipsis) are the same shape — one lead character, one continuation character, both perfectly valid UTF-8 when re-encoded — so only the result can tell them apart. So a two-character sequence is trusted only when its second character is one that Windows-1252 leaves undefined — a provable test rather than a judgement call. Six of the 57 candidate letters pass it; 51 do not, and the cost of that is specific enough to name plainly: when the only damage in your text is Romanian ș or ț, Vietnamese ơ or ư, Azerbaijani ə, or Hausa ɓ or ƙ, the text is handed back untouched instead of guessed at. Orașul București este frumos și mare is a real, ordinary sentence this tool declines. What usually rescues it is that one accent anywhere in the passage is enough: append în to that same sentence and it restores completely, ș included. The honest rule is therefore not "prose always works" — it is that the passage needs at least one Windows-1252-encodable character somewhere in it (â î ă ö é, a curly apostrophe, an em dash). Damage decoding into Syriac, Thaana or N'Ko is left alone for the same reason. The payoff for all this caution is that «ÉTÉ», CAFÉ… and „groß“ come back untouched. The replacement character � means a byte was already destroyed upstream — the text around it is still repaired, but the lost byte is never invented, and such input can never earn the "high" confidence rating (that rating requires a byte-exact round-trip proof). Repairs deeper than 3 encoding passes stop at the ceiling and report "medium" honestly. And CJK garble that happens to look like ordinary ideographs carries no evidence, so it is conservatively left alone.