Data Deduplication
Data deduplication removes repeated text sequences to prevent the model from memorizing duplicates and to improve generalization. Both exact-match and fuzzy (near-duplicate) deduplication are applied across and within documents.
What is Data Deduplication?
Data deduplication removes repeated text sequences to prevent the model from memorizing duplicates and to improve generalization. Both exact-match and fuzzy (near-duplicate) deduplication are applied across and within documents.
Data deduplication removes repeated text sequences to prevent the model from memorizing duplicates and to improve generalization. Both exact-match and fuzzy (near-duplicate) deduplication are applied across and within documents.
Where is it used?
RefinedWeb and LLaMA use MinHash and LSH for near-duplicate deduplication across trillions of tokens. Studies show deduplication improves benchmark performance and reduces memorization of training data. The Pile applied exact deduplication.
How to build it
Exact: hash each document and drop duplicates. Near-duplicate: use MinHash + LSH via `datasketch.MinHashLSH`. For n-gram dedup: compute shingles, hash them, and cluster by Jaccard similarity threshold (e.g., 0.8). Remove all but one per cluster.