What the Bibliography Deduplicator does
This tool finds duplicate references in BibTeX and RIS libraries - the kind you get after exporting from Scopus, Web of Science, PubMed and Google Scholar and pasting it all together. It groups records by DOI and by normalised title with the same first author and a close year, explains why each group matched, lets you choose which record to keep, and exports a clean library with a change log.
Matching runs in your browser. Records that carry different DOIs are never merged, and every match shows its reason, so nothing is removed on a hunch.
How to use it
- Paste one or more BibTeX or RIS exports into the box, or drop up to ten .bib and .ris files. You can mix both formats.
- Leave the title similarity at 92% to begin with. Lower it to catch looser matches such as small typing differences; raise it towards 100% to accept only exact title matches.
- Choose Find duplicates. Each cluster lists its records, marks the most complete one and says why they matched.
- Pick the record to keep in each cluster. With the fill option ticked, blank fields in the kept record - a missing DOI or page range, say - are filled from its duplicates; existing values are never overwritten.
- Download the clean library as BibTeX or RIS, and the CSV change log for your records or a systematic-review flow diagram.
Reading the results
Same DOI is the strongest evidence: two records with one DOI describe the same work, whatever their titles look like. DOIs are compared without regard to case, as the DOI system specifies.
Title matches are normalised first - case, accents, punctuation and the & sign are ignored - and must also share the first author's surname and be within a year of each other, which allows for online-first and print dates.
Most complete means the record with the most filled fields; it is a sensible default, not a judgement of which export is correct.
Worked example: a merged export with three duplicates
The example pastes a three-entry BibTeX file and a four-record RIS file - seven records in total.
Ray (2004) appears in both. The BibTeX DOI is 10.1037/0003-066X.59.1.29 and the RIS link is https://doi.org/10.1037/0003-066x.59.1.29; after removing the link and ignoring case they are the same DOI, so they form cluster 1.
Dunlosky et al. (2013) appears in title case in BibTeX and sentence case in RIS. After normalising they are identical, with the same first author and year, so they form cluster 2. The BibTeX record is more complete (five authors, journal, pages), but only the RIS record has the DOI; with fill ticked the kept record gains DOI 10.1177/1529100612453266.
Brown (2014) is not matched: one record says "Make It Stick" and the other adds the subtitle, and the titles are only about 27% similar. That is a case to check by hand - the tool reports what it can show, and the clean library keeps five records.
Using it for a systematic review
PRISMA-style reviews report how many records were removed as duplicates before screening. The change log gives you that number and an audit trail of each removal with its reason, which you can keep with your review protocol.
Because different databases format titles differently, run the check once at the default threshold, then once lower, and review the extra clusters rather than accepting them all.
Limitations: what the result does not prove
- Titles that differ by a subtitle, a translation or an abbreviation fall below any sensible similarity threshold and are not matched. Check likely pairs by hand.
- Preprints and the published version of the same study usually have different DOIs and are deliberately kept apart; decide for yourself whether to treat them as one study.
- Only the fields the tool understands are carried into the export: authors, editors, title, container, volume, issue, pages, dates, publisher, place, edition, DOI, URL and access date. Notes, keywords, abstracts and attachments from your reference manager are not kept.
- Comparing every pair is quadratic, so the tool stops at 3,000 records; split larger libraries.
Privacy: where your data goes
Everything you paste, type or drop is processed in this browser tab. It is not uploaded, logged, stored or sent to analytics. Session recording and tag-manager scripts are switched off on this page.
Standards and sources
- DOI Handbook - DOI names are case-insensitive (International DOI Foundation)
- BibTeX on CTAN (the Comprehensive TeX Archive Network)
Frequently asked questions
How do I remove duplicates from a BibTeX file?
Paste or drop the .bib file, choose Find duplicates, review each cluster and download the clean BibTeX. The citation keys of kept BibTeX records are preserved, so existing \cite commands in your LaTeX document keep working.
Can I combine exports from several databases?
Yes. Drop several .bib and .ris files together, or paste them one after another. The records are read as one library, compared with each other, and exported as a single file in either format.
Why were two versions of the same paper not merged?
Either their DOIs differ, which usually means a preprint and a journal version, or their titles are too different, for example when one includes a subtitle. Both are deliberate safety rules; review such pairs by hand.
Does it change my records?
Only when you tick the fill option, and then only blank fields in the record you keep are filled from its duplicates. Existing values are never overwritten, and the change log lists every field that was filled.
Is Zotero's or EndNote's duplicate check not enough?
Reference managers work well inside one library. This tool is useful before import, for comparing exports from different databases, for keeping a documented log of removals, and when you only have plain .bib or .ris files.
Last reviewed by the A2Z.Tools team against the sources listed above.
