What the Spreadsheet Duplicate Finder does
This tool finds duplicate and near-duplicate records in an Excel workbook or CSV file - the "Acme Trading Ltd" and "ACME Trading Limited", "Bluebell Florists" and "Bluebell Florist" kind - by matching on a key built from several columns, normalising each column the way that suits it, and grouping rows whose similarity reaches a threshold you set. You review every cluster, choose which row to keep, and only then download a clean file.
It is built for the messy case. If your rows are byte-for-byte identical, the simpler CSV Duplicate Remover is quicker; this tool exists for lists typed by different people, merged from different systems, or exported with inconsistent spacing, capitals, accents and company suffixes. It runs entirely in your browser.
How to use it
- Drop an .xlsx or .csv file (or paste CSV). For a workbook, pick the sheet.
- Tick the columns that together identify a record - for customers typically name plus city, or email alone. Each ticked column gets a normalisation rule: company names ignore Ltd, Inc and GmbH; phones compare the last ten digits; emails are trimmed and lower-cased; general text ignores case, spacing, punctuation and accents.
- Set the similarity threshold. 1.00 finds rows that are identical after normalising; 0.90-0.95 also catches typos and small spelling differences. Every key column must reach the threshold, so a matching name in a different city is not merged.
- Choose Find duplicates and review the clusters. Change the kept row in any cluster if the default (first, last or most complete) is wrong.
- Download the deduplicated sheet as CSV or XLSX, plus a cluster report showing every decision.
Reading the results
A cluster is a group of rows linked by matches. Matching is transitive: if A matches B and B matches C, all three are one cluster even when A and C are less similar - the cluster shows its lowest pairwise similarity so you can spot a chain that went too far.
Identical after normalising means the key values became the same text once the rules were applied - these are safe to merge. Near-duplicate clusters are suggestions: the similarity score says how alike the text is, not whether the records are the same business or person.
For up to 2,500 rows every pair is compared. Larger sheets use sorted-neighbourhood blocking (each row is compared with its 24 nearest neighbours in the sorted order of each key column), which is fast but can miss a pair whose values sort far apart; the statistics say which method ran.
Worked example: a customer list merged from two systems
The example workbook's Customers sheet has 10 rows. With Name (company rule) and City as the key at a threshold of 0.92, it produces four clusters.
ACME Trading Limited and Acme Trading Ltd are identical after normalising: lower-casing gives "acme trading limited" and "acme trading ltd", and the company rule drops both suffixes to leave "acme trading". The two Northwind Foods rows are identical outright, and Café Olé and Cafe Ole become "cafe ole" once accents are folded.
Bluebell Florists and Bluebell Florist are not identical: after normalising they differ by one final letter. Their Jaro-Winkler similarity is 0.988, above 0.92, so they form a near-duplicate cluster. Keep-first retains sheet rows 2, 3, 5 and 7, removes rows 4, 6, 8 and 10, and the clean file has 6 customers.
How this differs from the CSV Duplicate Remover
The CSV Duplicate Remover deletes rows that are exactly the same in a CSV. This finder adds what that workflow cannot do: it reads XLSX workbooks as well as CSV, matches on a composite key rather than the whole row, normalises each key column by type, finds near-duplicates above a similarity threshold, and shows grouped clusters for review before anything is removed. Use the remover for exact copies; use this when the duplicates were typed differently.
Choosing a similarity measure
Jaro-Winkler suits short strings such as names, and rewards a shared beginning. Levenshtein similarity counts single-character edits relative to length and suits longer text with typos. Dice bigrams compare pairs of letters regardless of order, which helps when words are swapped. All three are standard string metrics; the thresholds are yours to judge, and a quick look at the clusters at two or three thresholds is the best calibration.
Limitations: what the result does not prove
- Similarity is about spelling, not meaning. Two different companies with similar names in the same city can cluster, and the same company recorded under a trading name will not - review before you delete.
- The tool does not merge the contents of duplicate rows into one; it keeps one row per cluster as it is. Copy any missing details across in your spreadsheet if the kept row lacks them.
- Large sheets use blocking, which can miss some pairs; very large files are limited by browser memory (the limit here is 200,000 rows and 40 MB).
- The clean XLSX stores every value as text so nothing is re-typed. Formulas, formatting and other sheets from the original workbook are not carried over.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- Winkler (1990) - String comparator metrics and enhanced decision rules (US Census Bureau)
- Hernández and Stolfo (1995) - The merge/purge problem for large databases (sorted neighbourhood)
- ICO - Accuracy principle (UK GDPR)
Frequently asked questions
How do I find near-duplicate rows in Excel without a plugin?
Drop the workbook here, tick the columns that identify a record, pick a normalisation rule for each and set a threshold below 1.00. Rows whose key columns are at least that similar are grouped into clusters you can review and download without them.
What threshold should I use for fuzzy matching names?
Start at 0.92 with Jaro-Winkler. Lower values (0.85-0.90) catch more spelling variants but also more false matches between genuinely different names; 1.00 catches only rows that are identical once normalised. Compare the clusters at two thresholds before choosing.
Why does a composite key matter?
A single column is rarely enough to say two rows are the same record. Two branches of the same chain share a name but not a city; two people share a surname but not an email. Requiring every key column to match keeps those apart.
Does it treat Ltd and Limited as the same?
With the company rule, yes: it ignores common legal suffixes such as Ltd, Limited, Inc, LLC, Corp, PLC, GmbH, AG, SA, BV and Pty, as well as punctuation, capitals, accents and the word "and", so "Smith & Sons Ltd" and "SMITH SONS LIMITED" compare as equal.
Can it compare phone numbers written in different formats?
The phone rule keeps digits only and compares the last ten, so +44 20 7946 0001, (020) 7946 0001 and 020-7946-0001 all match. It does not validate numbers or know country codes, so check clusters where numbers are short or unusual.
Is my customer list uploaded anywhere?
No. The workbook is unzipped and read in your browser, the comparisons run in this tab, and the clean files are generated locally. Nothing is sent to a server, which matters for personal data covered by data-protection law.
What happens to rows with blank key columns?
A blank on one side scores zero for that column, so it will not match a filled value. Two rows that are blank in every key column are never treated as duplicates of each other, because an empty key says nothing about identity.
Last reviewed by the A2Z.Tools team against the sources listed above.
