What each option does
Strip HTML removes tags and keeps the text between them.
Normalise converts curly quotes, long dashes and other typographic characters back to plain ASCII.
Collapse spaces turns runs of spaces and tabs into a single space. Line breaks flattens broken lines into flowing text.
Trim strips leading and trailing whitespace from every line, Remove blank lines drops the empty ones, and Remove duplicates deletes lines that repeat.
Tick as many as you need. They all run in one pass.
The characters you cannot see
Pasted text carries passengers, and the ones causing trouble are usually invisible.
A non-breaking space, U+00A0, looks just like a normal space. It is not one. Word inserts them all the time.
They survive copy and paste, then quietly break searches, sorts and comparisons. “John Smith” with a non-breaking space does not equal “John Smith” with a plain one, and nothing on screen tells you why.
A zero-width space, U+200B, is worse. It has no width at all. Nothing shows, nothing moves, and it still counts as a character.
Text copied from web pages and rich editors picks these up all the time. That field rejecting your input for being 1 character too long? Often this.
More live in the same family. Zero-width non-joiner at U+200C. Byte order mark at U+FEFF. Unicode defines at least 17 different space characters of varying widths, and every one of them is not the ordinary space at U+0020.
Normalising catches the common ones. If a string still refuses to match something it obviously equals, count its characters and compare against what you can see. A string that reads as 10 characters and reports 12 has 2 passengers.
Smart quotes break code
Word and Google Docs helpfully convert straight quotes to curly ones as you type. Helpful in prose. Ruinous in anything else.
Straight quotes make a valid string in every programming language. Curly quotes make a syntax error in all of them, and the message rarely says why, because to a parser those are just 2 odd characters.
Apostrophes go the same way. So do hyphens, where a typed double hyphen turns into a long dash.
This is why a command copied from a blog post or a Word file sometimes fails in the terminal for no visible reason. Run it through Normalise first.
Duplicate removal keeps the first
Worth knowing before you run it on something order-sensitive.
Removing duplicates keeps the first occurrence of each line and drops every later one. The surviving order matches the order lines first appeared, so it is stable rather than sorted.
Comparison is exact. Two lines differing only by trailing whitespace count as different lines, which is why running Trim before Remove duplicates catches far more than running it alone. Case matters too, so Apple and apple both survive. This trips people up on exported lists constantly, because a spreadsheet or a database dump will happily give you 3 copies of the same email address where 1 has a stray tab, 1 has a capital letter, and only 1 is what you were expecting. Run Trim and then check whether case matters for your job before you trust the count. If it does not, lowercase the whole list first and the duplicates collapse properly.
Frequently asked questions
Why does my text still not match after cleaning?
Probably an invisible character the normaliser did not cover. Compare the character length against what you can see, and if they disagree, something zero-width is in there.
What is a non-breaking space?
A space that stops a line from breaking at that point. It looks identical to a normal space and is a different character, which is why it quietly breaks comparisons and searches.
Why did my pasted command fail in the terminal?
Almost certainly smart quotes. Word and Google Docs turn straight quotes into curly ones, and curly quotes are a syntax error everywhere. Normalise converts them back.
Does Remove duplicates keep the first or last copy?
The first. Every later occurrence is dropped and the original order is preserved.
Should I trim before removing duplicates?
Yes. Lines differing only by trailing whitespace count as different lines, so trimming first catches far more of them.
Can I use several options at once?
Yes, that is the intended way. Tick everything you need and it all applies in a single pass.
Related: remove line breaks · remove duplicate lines · find and replace · change case