ReplyDrift
Local evidence workflow
Open workspace

Operator field guide · Stale-answer incident

A five-step checklist for finding who may have seen an outdated AI answer.

A Help Center correction does not tell you which earlier support conversations cited the old source. This checklist keeps the evidence small, deterministic, and reviewable before anyone considers customer outreach.

Use this when
A Help Center article changed after a reported wrong or outdated AI-support answer.
Leave with
A bounded list of exact conversation IDs, a separate ambiguity queue, and an explicit record of what the export could not prove.
Data boundary
IDs and source references only. Remove message bodies, names, emails, attachments, secrets, and internal notes before import.

The workflow

A reviewable incident path, not an automated customer action.

  1. 01

    Freeze the changed-source boundary

    Record the changed article's canonical URL, stable article ID, and exact effective timestamp. Keep the previous URL when a rename or consolidation occurred.

    Operator checkUse one timestamp and one timezone across the incident. Do not substitute the export time for the content-change time.

  2. 02

    Prepare a minimized conversation export

    Keep the composite conversation ID, platform conversation ID when available, bot ID, first-message time, test state, and explicit knowledge-source references.

    Operator checkDelete message text, customer identity, contact details, attachments, internal notes, payment data, credentials, and unknown fields.

  3. 03

    Map only deterministic source evidence

    Match canonical source references to the changed article. Add an explicit alias only when a known old URL belongs to that one article.

    Operator checkNever use title similarity, keywords, or message content to manufacture a match.

  4. 04

    Separate exact evidence from ambiguity

    Put source-and-timing matches in the exact lane. Keep collisions, malformed references, and uncertain candidates in review required with their reason codes.

    Operator checkA review-required row is not a weaker exact match; it is a deliberate stop for human judgment.

  5. 05

    Approve customer action outside the tool

    Review the article change, the original response in the system of record, policy obligations, and current customer context before deciding whether to contact anyone.

    Operator checkReplyDrift does not read messages, judge answer correctness, edit Zendesk, or send a correction.

Decision boundary

What belongs in the queue—and what does not.

Observed signalQueue treatmentHuman action
One known source reference maps to the article, and the conversation began before the changeEXACTOpen the original conversation and verify whether the changed fact affected the answer.
A deliberately supplied old URL maps to exactly one changed articleEXACT · explicit aliasConfirm the redirect or rename history before using the row.
A reference collides, is malformed, or cannot be bound to one articleREVIEW_REQUIREDResolve the source identity manually; do not batch it with exact rows.
The conversation started after the article changeOutside queueDo not treat it as a pre-change exposure from this run.
The export contains no explicit knowledge sourceOutside queueRecord the coverage gap. Do not infer a source from the response text.