fix‑encoding

The one bug it fixes — and what it refuses to touch

What it does. Almost all mojibake comes from a single mistake: text correctly stored as UTF-8 (é is the bytes C3 A9) gets read by some program as Windows-1252, so é renders as é — then the wrong text is re-saved as UTF-8, baking the damage in. This tool reverses exactly that chain, including text mangled two or three times over (café), whatever alphabet the original was written in: the same reversal that turns Café back into Café turns 日本語 back into 日本語, Привет into Привет and 😀 into 😀, and it handles Greek, Arabic, Thai and Hangul the same way. It also undoes the same mistake through the legacy CJK code pages: Japanese garbled via Shift_JIS (東京 → 譚ｱ莠ｬ), Chinese via GBK or Big5, Korean via EUC-KR.

When you'd use it. Legacy CSV/JSON/SQL dumps where the accents arrived broken, old mail archives, scraped pages, and log pipelines that keep printing ’ where an apostrophe belongs.

It refuses clean text. A repair is applied only when it strictly reduces mojibake evidence, so correctly-encoded prose — café, 日本語, Привет, Ελληνικά, 😀, curly quotes, a lone em dash or euro sign — is returned verbatim, and running the tool on its own output is a guaranteed no-op. That caution has a flip side: ambiguous input that no repair provably improves is left alone rather than guessed at.

Honest limits. It is a heuristic, not a proof: it handles the dominant cp1252 chain in every script and the four CJK chains above, but corruption from other code pages (KOI8-R, the wider ISO-8859 family, EUC-JP) is out of scope and left untouched. Inside the cp1252 chain one deliberate blind spot remains: é (a garbled é) and É… (a clean all-caps word followed by an ellipsis) are the same shape — one lead character, one continuation character, both perfectly valid UTF-8 when re-encoded — so only the result can tell them apart. So a two-character sequence is trusted only when its second character is one that Windows-1252 leaves undefined — a provable test rather than a judgement call. Six of the 57 candidate letters pass it; 51 do not, and the cost of that is specific enough to name plainly: when the only damage in your text is Romanian ș or ț, Vietnamese ơ or ư, Azerbaijani ə, or Hausa ɓ or ƙ, the text is handed back untouched instead of guessed at. Orașul București este frumos și mare is a real, ordinary sentence this tool declines. What usually rescues it is that one accent anywhere in the passage is enough: append în to that same sentence and it restores completely, ș included. The honest rule is therefore not "prose always works" — it is that the passage needs at least one Windows-1252-encodable character somewhere in it (â î ă ö é, a curly apostrophe, an em dash). Damage decoding into Syriac, Thaana or N'Ko is left alone for the same reason. The payoff for all this caution is that «ÉTÉ», CAFÉ… and „groß“ come back untouched. The replacement character � means a byte was already destroyed upstream — the text around it is still repaired, but the lost byte is never invented, and such input can never earn the "high" confidence rating (that rating requires a byte-exact round-trip proof). Repairs deeper than 3 encoding passes stop at the ceiling and report "medium" honestly. And CJK garble that happens to look like ordinary ideographs carries no evidence, so it is conservatively left alone.