← All articles

Respond when an AI answer is wrong

Start with whether the answer has been used or an automated action is still running, then contain the problem, preserve evidence and check the fix.

First establish whether the answer is still a draft, has already been used or is part of an automated process that keeps running. That determines what needs attention immediately.

Situation

First response

A draft is wrong and nobody has used it

Correct or discard it; investigate repeated errors

Someone has acted on the answer

Tell the people who may still rely on it and correct affected work

An automated process is spreading the error

Pause the affected actions and involve whoever supports the system

Prioritise stopping further harm. Preserve the original input, output and relevant records as soon as practical. Do not delay necessary containment while collecting a perfect record.

Find out how far it went

Suppose a fictional service sends an incorrect delivery estimate to several customers. Find which messages went out, whether anyone has acted on them and whether the same source error affects other drafts. Correct the information and contact the people who may still rely on it.

The seriousness depends on consequences, sensitivity and reversibility. An internal error can affect money or records. A customer-facing typo may be minor. Assess the actual case.

If personal information was exposed, lost or otherwise affected, involve the person responsible for data protection promptly. The ICO’s breach guidance explains assessment and reporting. Where notification is required, time limits matter; do not wait for a complete technical investigation before seeking advice.

Keep enough evidence to explain the failure

Save the original input and output, the time, the source material available and the instructions or settings in use. Limit access if the record contains confidential or personal information.

Look for contributing causes: an old document, a missing fact, a faulty connection, unsupported model output or a check that did not find the error. There may be several. Asking the same question again can produce a different result, so it does not replace the original evidence.

Check the repair before restarting

Test the failed case, related cases and some that worked before. If the fix changes instructions, keep a record of the change. Agree who can authorise a restart for a connected process, and monitor the affected outputs closely afterwards.

If the cause remains uncertain, say so. Narrowing the task or keeping it manual may be appropriate while you investigate. The failed-trial guide helps compare those options once the immediate problem is contained.

Jay Janes

About Jay

I’m Jay Janes, a Practical AI Advisor and founder of Reformat Labs. I write about choosing useful work for AI, checking the results and supporting the people using it.

Contact

Talk to Jay

Considering AI for a task in your business? Tell me what takes too long, causes problems or needs to improve.

What to send

For advice or a workshop, describe the work and your question. For a talk or podcast, send the audience, subject and date.