How the Remove Duplicate Lines works
The tool reads your text line by line and remembers each line it has already seen. The first time a line appears it is kept in its original position; later copies are dropped. The order of the remaining lines never changes, and the whole list is processed in a single pass, so lists with hundreds of thousands of lines finish almost instantly.
Lines are compared exactly unless you choose otherwise. Ignore case treats Apple and
apple as the same line. Trim spaces before comparing ignores spaces and tabs at the start
and end of a line. In both cases the first copy is output exactly as you wrote it. Empty lines (including lines with only
spaces) are never treated as duplicates, so paragraph breaks stay where they are, unless you tick
Remove empty lines. Choose "Only lines that were repeated" to list each duplicated line once instead.
How to use the Remove Duplicate Lines
- Paste your list into the Lines box, one item per line, open a text file, or select Load example.
- Choose the options: Ignore case, Trim spaces before comparing, Remove empty lines, and whether the Output shows each line once or only the lines that were repeated.
- The result updates as you type. You can also select Remove duplicates or press Ctrl + Enter.
- Check the counts of lines in, lines out and duplicates removed, then copy the result or download it as a .txt file.
Example
This list:
apple
banana
Apple
cherry
banana
cherrywith Ignore case and Trim spaces before comparing ticked becomes:
apple
banana
cherryThe statistics show 6 lines in, 3 lines out and 3 duplicates removed.
Common use cases
- Cleaning a list of email addresses, customer IDs or product codes before an import.
- Removing repeated entries from log files, keyword lists or URL lists.
- Finding which values in an export appear more than once.
- Merging two lists and keeping only one copy of each item.
Common errors and how to fix them
- Obvious duplicates were not removed
- The lines differ in a way you cannot see: a trailing space, a tab where the other has spaces, or a different line ending. Clean the text with Remove Extra Spaces first, then remove duplicates.
- Lines that differ only in capitalisation were both kept
- Comparison is case-sensitive by default. Turn on case-insensitive matching to treat Value and value as the same line.
- The remaining lines came back in a different order
- Sorting is switched on. Turn it off to keep the lines in their original order.
- Blank lines were treated as duplicates of each other
- An empty line is a line like any other, so all but one is removed. If the blank lines are meaningful separators, keep a copy of the original before running the tool.
Frequently asked questions
Does removing duplicates change the order of my lines?
No. The first copy of each line stays exactly where it was, and later copies are removed. If you want the lines sorted as well, use the Text Sorter, which can also remove duplicates.
What do Ignore case and Trim spaces do?
Ignore case treats lines that differ only in upper and lower case, such as Apple and apple, as duplicates. Trim spaces before comparing ignores spaces and tabs at the start and end of each line. In both cases the first copy is kept exactly as you wrote it.
What happens to empty lines?
Empty lines, including lines that contain only spaces, are never treated as duplicates, so blank lines between groups stay in place. Tick Remove empty lines to delete all of them.
Can I see only the duplicated lines?
Yes. Set Output to Only lines that were repeated. The result lists each line that appeared more than once, a single time, in the order of its first appearance.
How many lines can it handle?
The tool checks each line once using a lookup table, so lists with hundreds of thousands of lines are processed in well under a second on a typical computer. Input up to 20 MB is accepted.
Is my list uploaded?
No. Everything runs in your browser and your text is not sent to our server.