Updated on 20 October 2026.
A duplicate is another copy of the same observation, not simply the same answer text. Independent runs may return identical text. Before deleting rows, establish their provenance and record unit. This procedure produces a working copy, a decision log and unresolved records for review.
Establish what one row represents
Determine whether a row represents an answer, cited URL, mention or entire run. An answer can have several sources and mentions. A shared answer ID therefore does not justify deleting rows from a source table. Read the provider's field definitions first.
Check without losing evidence
Preserve the original files. Add filename, source row, retrieval scope and filters to a working copy. These describe provenance, not a fresh measurement.
Identify a stable observation reference within its provider and project namespace. A batch or run ID may cover multiple questions. Question text, calendar date and an answer hash alone cannot guarantee identity.
Compare records whose shared identity is established. If content and configuration agree, keep one analytical representation and every provenance reference. Conflicting content requires review, not an automatic keep-first rule.
Keep separate runs even when their text is identical. Without evidence of shared identity, mark a pair unresolved rather than inferring it from a time gap.
When joining export pages, check filters, page counts and retrieval boundaries. If ordering can change, file length alone does not prove completeness.
A worked classification example
Example data explain the method and do not represent any brand's results.
| Records in the same answer table | Decision | Preserve |
|---|---|---|
| OBS-A in two files, same project and identical payload | Confirmed copy | One analytical record and both provenance references |
| OBS-A and OBS-B, confirmed separate runs, identical text | Two observations | Both records |
| OBS-C with conflicting payloads | Conflict | Both versions for review |
| No ID, similar timestamp and text | Unresolved identity | Review records without automatic deletion |
When the dataset is ready
After checking, distinguish retained observations, confirmed copies and unresolved records. Record the rule, reviewer and reason. Do not present a recalculated metric without explaining how ambiguous records were treated.
Deduplication is not the selection of a better model answer. Do not remove an unfavourable observation because another run looks better. There is no universal time window that automatically separates copies from independent measurements.
Working with Semly data
Before joining Semly material, confirm the scope and granularity available in your view or export. If observation identity is missing, request clarification instead of inventing a reliable-looking key. This protocol does not promise a particular Semly export format or internal database schema.
