AI Customer Research
Menu

Kit

The Insights Audit System

A repeatable way to check AI-generated analysis before you act on it, whichever LLM you use.

AI gave you an insight. How do you know it’s actually there?

AI is getting very good at analyzing customer research.

It can read hundreds of interviews, pull patterns out of open-text responses, compare segments, and turn a pile of research into a neat set of findings in minutes.

But there’s a catch. The better the output sounds, the harder it can be to spot when it’s wrong.

  • Maybe the evidence doesn’t actually support the claim.
  • Maybe AI has mistaken something common for something important.
  • Maybe there’s contradictory evidence buried in the data.
  • Maybe it has quietly turned an interpretation into a fact.

And checking all of that manually defeats some of the point of using AI in the first place.

The Insights Audit System gives you a repeatable way to check AI-generated analysis before you act on it. You can use it on your own AI outputs, regardless of which LLM you’re using.

Four ways to audit your insights

  1. Qualitative audit

    For interviews, open-text responses, support conversations and other qualitative data. Check whether AI has:

    • represented what people actually said
    • found real patterns rather than superficial similarities
    • kept the relevant context
    • confused frequency with importance
    • missed outliers or contradictory evidence
    • presented interpretation as fact
    • made recommendations that go beyond the evidence

  2. Quantitative audit

    For surveys, experiments and product or behavioral data. Check:

    • sample and base sizes
    • denominators
    • missing data
    • significance
    • correlation vs. causation
    • segmentation
    • measurement quality
    • generalizability
    • whether the conclusion actually follows from the numbers

    AI can be very convincing when it explains a dataset. That doesn’t mean the analysis is right.

  3. Mixed-methods audit

    When you have both qualitative and quantitative evidence, things get more interesting. The system helps you check:

    • where the evidence agrees
    • where it doesn’t
    • where one method adds something the other can’t
    • whether you’re actually comparing like with like
    • whether AI is forcing two pieces of evidence into a conclusion they don’t support

  4. Multi-source audit

    This is for the messy stuff most customer-insights teams actually deal with: interviews + surveys + support tickets + reviews + analytics + sales calls + existing research.

    Instead of asking AI for one giant summary, you build an Evidence Map.

The Evidence Map

ClaimSupporting evidenceContradictory evidenceRelevant segmentsEvidence strengthLimitationsConfidenceWhat would change our mind?Next action

This gives you a much better way to work with AI when the evidence doesn’t live in one neat dataset.

The TRACE framework

The audits are built around five questions:

TTraceability
Can we find the evidence?
RRepresentativeness
Does the evidence support the broader claim?
AAlternatives
What else could explain this?
CContradictions
What evidence disagrees?
EExtrapolation
Has AI gone beyond what the evidence actually tells us?

You can use TRACE to check a single insight, an entire analysis, or a research report.

And there’s a bonus: audit the workflow itself

Sometimes the problem isn’t the final answer. It’s the way you asked AI to get there.

  • Maybe you gave it the wrong context.
  • Maybe it retrieved the wrong evidence.
  • Maybe your instructions are creating a predictable failure.
  • Maybe a different model or workflow would do the job better.

The Workflow Evaluation bonus gives you a way to test this.

You run your workflow against a known dataset, compare the output with expected findings, identify where it failed, change the workflow, and run it again. Over time, you can build your own benchmark and regression tests.

So instead of: “This version seems better.”

You can ask: “Did this change actually improve the analysis?”

What’s included

The Insights Audit System
The complete audit workflow and methodology.
4 audit workflows
Qualitative, quantitative, mixed-methods and multi-source.
Evidence Manifest
A structured way to keep track of where evidence came from and which claims it supports.
Evidence Map
A claim-level view of supporting and contradictory evidence.
MCP Retrieval Protocol
A practical approach to retrieving the evidence you need from connected research tools without feeding an entire research repository to an LLM.
Workflow Evaluation
The bonus system for testing and improving your own AI analysis workflow.
5 worked examples
Complete examples showing the audits in practice.
Templates
  • Audit report
  • Evidence table
  • Evidence map
  • Stakeholder summary
  • Decision-readiness report
  • Workflow evaluation benchmark

This isn’t a prompt pack

You’re not getting 50 prompts and being told to hope for the best. You’re getting a way of working.

And you don’t need to use a particular AI tool. Use it with Claude, ChatGPT, Gemini, Claude Code, Codex, or whatever you end up using next.

The point is the audit process, not the model.

Who is this for?

  • Researchers and customer-insights people who are already using AI and want a better way to check its work.
  • PMs and designers who are increasingly analyzing customer evidence themselves.
  • Marketing and customer teams using AI to work with feedback, research and customer language.

You don’t need to be an AI expert. You just need to be responsible for making decisions based on customer evidence.

The goal

You shouldn’t have to choose between “I’ll check everything myself” and “The AI said it, so it must be right.”

There’s a better middle ground. Let AI do more of the analysis. Put a proper audit around the parts that matter.

Kit or course?

A kit is a set of ready-made skill files. Download it and run it on your own data the same day, without knowing how it works inside or how to build it from scratch.

A course teaches the whole method, with recorded tutorials, workflows and files, so you can build and adapt systems like these yourself. See the courses.