用 LSH 做数据去重

Web crawls are full of copies: mirrored sites, boilerplate footers, syndicated news, the same recipe reformatted by 300 blogs. Train on that raw and the loss is dominated by whichever text is most repeated.

• Memorisation: the model can recite duplicated passages verbatim instead of generalising.
• Benchmark leakage: if test data leaked into training, your "great results" are an illusion.
• Wasted compute: every duplicated token is a training token you paid for twice.
• Distorted distribution: a duplicated topic silently becomes over-weighted.

Exact matching misses near-duplicates. Instead, break each document into overlapping shingles (every 5-word window) and compare the Jaccard similarity of the two shingle sets: the size of the intersection divided by the size of the union.

Computing Jaccard for every pair is O(n²) — impossible at a billion documents. MinHash fixes it with a trick: hash every shingle, keep the minimum hash value. Do that with 128 different hash functions and a document becomes a 128-number signature. The beautiful part: the fraction of positions where two signatures agree is an unbiased estimate of their Jaccard similarity.

Comparing every pair of documents is O(n²): a billion documents would need 5×10¹⁷ comparisons. Locality Sensitive Hashing cuts this to near-linear by exploiting one idea: similar items should collide more often than dissimilar ones.

1. 1Compute a 128-dimension MinHash signature for every document.
2. 2Split the signature into bands — say 16 bands of 8 rows each.
3. 3Hash every band independently; documents landing in the same bucket for any band become candidates.
4. 4Only compute exact Jaccard similarity for those candidate pairs, then drop the near-duplicates.

# 1. Exact: normalise and hash the full document
key = sha256(normalise(text).encode()).hexdigest()

# 2. Near-duplicate: MinHash + LSH bands
sig = minhash(normalise(text), num_perm=128)
if lsh.insert(sig) is not None:      # collided with an existing bucket
    drop(text)

# 3. Semantic: paraphrase detection on the survivors
emb = embed(normalise(text))
if any(cosine(emb, e) > 0.95 for e in kept_embeddings):
    drop(text)

• Normalise first: lowercase, strip HTML, collapse whitespace, drop boilerplate.
• Deduplicate across train/validation/test splits, not inside each one.
• Log what you dropped — a sudden 60% drop usually means a pipeline bug, not dirty data.
• Track the duplicate rate over crawl dates to catch template regressions.

Deduplication is not hygiene for its own sake. It changes what the model can do: fewer verbatim copies means more gradient signal spent on generalisable patterns, and less capacity burned on memorised strings.

• Less memorisation: regurgitating training text is a legal and reputational risk, and it grows with duplication.
• Better generalisation: the effective dataset is more diverse, so the model sees more of the language.
• Cheaper training: fewer tokens to process for the same effective coverage.
• Fairer evaluation: no leaked test data inflating your benchmark numbers.