What the Duplicate Line Remover does
This tool removes duplicate lines from any list or block of text. Paste the lines (or drop a .txt file) and you get the list with every repeat taken out, a count of how many copies were removed, and a table showing which lines repeated, how many times and on which line numbers.
You decide what counts as a duplicate. Keep the first copy or the last one, and optionally ignore upper and lower case, stray spaces, punctuation or accents, so that "[email protected]" and "[email protected] " are treated as the same line. It all runs in your browser, so lists of emails, customers or keywords never leave your computer.
How to use it
- Paste your text into the box, one item per line, or drop a .txt or .csv file onto the drop area. Up to 200,000 lines or 5 MB.
- Choose whether to keep the first or the last copy of each repeated line. Keeping the last copy is useful when later lines are newer versions of earlier ones.
- Tick the loose-matching options you need: ignore case, spaces at the ends, extra inner spaces, punctuation or accents. The original text of the kept copy is never changed - the options only affect matching.
- Pick what to show: the unique list, only the lines that were repeated, or only the lines that never repeated, in the original order, A to Z, or most repeated first.
- Copy the result, copy it with a count before each line, download it as a .txt file, or download a CSV of every line with how often it appeared.
Reading the results
"Duplicates removed" is the number of extra copies taken out. A list of 8 lines with 4 left had 4 extra copies, which may belong to just 3 different repeated lines - the table shows which.
"Spellings merged" lists the different ways a line was written when loose matching joined them, such as "Yes", "yes" and "YES". Check it before trusting the result: ignoring punctuation can join lines you meant to keep apart, such as two product codes that differ only by a hyphen.
Blank lines are kept where they were by default, so paragraph spacing survives. Remove them, or treat them like other lines so only one blank line is left.
Worked example: cleaning an email list
The list has 8 lines: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], the same Tom address with a space at each end, and [email protected] again.
With exact matching only [email protected] and the second [email protected] are exact repeats, so 2 lines go and 6 are left.
Ticking "ignore case" and "ignore spaces at the start and end" also joins [email protected] and the padded Tom address: 4 copies are removed and 4 addresses remain - anna, raj, li.wei and tom. The table shows Anna three times (lines 1, 3 and 8, written two ways), Raj twice and Tom twice.
How this differs from removing duplicate rows in a CSV
The CSV Duplicate Remover works on the rows and columns of a spreadsheet file. This page treats each line as plain text, whatever it contains - a list of names, keywords, URLs, log lines or words copied from a document - and adds the controls that matter for lists: keeping the first or last copy, loose matching on case, spacing, punctuation and accents, and a count of how many times each line appeared before it was removed.
Doing the same in an editor or the command line
On Linux and macOS, sort -u file.txt removes duplicates but also sorts the lines, and awk '!seen[$0]++' file.txt keeps the first copy in the original order. In Vim, :sort u sorts and removes duplicates. In Notepad++, Edit > Line Operations > Remove Duplicate Lines keeps the first copy. None of those ignore punctuation or accents, or tell you how many copies each line had, which is what this tool adds.
Limitations: what the result does not prove
- Matching is line by line. Two lines that say the same thing in different words, or one item split over two lines, are not detected as duplicates.
- Loose matching can merge lines you wanted to keep apart. "Ignore punctuation" treats "A-1" and "A1" as the same; check the "Spellings merged" column before you use the result.
- The kept line is copied exactly as it was, including any leading spaces or odd capitals, because the options only change how lines are compared, not the output.
- Very large files are limited to 200,000 lines or 5 MB so the page stays responsive; split bigger files and run them in parts.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- GNU coreutils - uniq and sort -u
- Unicode Standard Annex #15 - Unicode Normalization Forms (used for accent-insensitive matching)
- Vim documentation - :sort
Frequently asked questions
How do I remove duplicate lines from a list?
Paste the list into the box. Duplicates are removed straight away, keeping the first copy of each line in its original position. Copy the cleaned list or download it as a text file; the table underneath shows which lines repeated and how many times.
Can I remove duplicates without changing the order of the lines?
Yes - the default keeps the original order, with each line at the position of its first copy. Choose "last copy" to keep each line where it last appeared instead, or choose A to Z or most repeated first if you do want the list sorted.
How do I remove duplicate lines ignoring upper and lower case?
Tick "Ignore upper and lower case". "Apple", "apple" and "APPLE" are then treated as one line and the first spelling is kept. The "Spellings merged" column lists the variants that were joined so you can check them.
How do I remove duplicate lines in Notepad++ or Vim?
In Notepad++ use Edit > Line Operations > Remove Duplicate Lines; in Vim, :sort u removes duplicates but sorts the file. Both match lines exactly. Paste the text here if you need to ignore case, spaces or punctuation, keep the last copy, or see how many times each line appeared.
Can I see how many times each line was repeated?
Yes. The table lists every line that appeared more than once, how many times, and the line numbers of each copy. "Copy with counts" puts the count before each line, and the duplicates CSV includes every line with its count.
Can I keep only the lines that appear once?
Yes. Set "Show" to "Only lines that never repeated" to remove every line that has a duplicate, including its first copy. "Only lines that were repeated" does the opposite and lists one copy of each repeated line.
Is it safe to paste a customer or email list here?
The text is processed by JavaScript in your browser tab and is not uploaded, logged or stored. Dropped files are read locally too. Closing the tab clears everything; nothing is remembered between visits.
Last reviewed by the A2Z.Tools team against the sources listed above.
