All articles
AI readiness8 min read

Your AI pilot worked. Why did it break at scale?

When an AI pilot works and then starts giving wrong answers after a company-wide rollout, the model usually hasn't changed. The rollout has. It extended the assistant into documents the pilot never touched: other departments' policies, old process pages and content nobody owns. The fix is to check those documents before each new phase of the rollout, not after the complaints start.

Anchoris teamNotes from the people building Anchoris

A single team, often IT or customer support, runs a contained trial for a few months. The assistant answers questions accurately, people like it, and the numbers look good enough to take to leadership. The company-wide rollout is approved, expectations rise, and within a few weeks of going live the same tool starts getting things wrong. Not occasionally. Repeatedly, with complete certainty, across departments that had never used it before.

The instinct is to assume something broke. The model got worse, the vendor changed something, or the rollout was rushed. In most cases none of that is true. The rollout extended the AI's reach into knowledge the pilot never had to deal with, and that knowledge was never as solid as the pilot made it look.

A pilot is a curated environment, whether anyone planned it or not

Pilots succeed for a reason that has little to do with the technology. They're small. One team, a handful of folders, a scope narrow enough that someone could keep track of what the AI was reading, even informally. That isn't a flaw in how pilots are run. It's the point of a pilot. But it means a successful pilot proves that one team's slice of the company's knowledge held up, not that the company's knowledge holds up in general.

Some teams do this on purpose, limiting the AI to a few folders they already rely on, precisely because they know the rest of the documentation isn't in the same shape. That's a sensible way to run a pilot. The trouble starts when its success is read as proof of readiness, rather than proof that a carefully chosen set of content was ready. Scaling the same assistant to every department means scaling it into every gap, contradiction and forgotten document those departments have built up over the years, most of which the pilot team never had reason to look at.

What actually changes between the pilot and the rollout

The model doesn't change. The infrastructure mostly doesn't either. What changes is how much the AI is expected to read, and that grows much faster than most rollout plans allow for.

A pilot confined to one team might touch a few dozen documents that someone personally reviewed in the past year. A company-wide rollout touches everything: HR policy, process pages from a department that reorganised eighteen months ago, pages written by people who have left, and the contradicting versions of a policy that different teams have worked around for years without ever settling. None of that was visible during the pilot, because the pilot was never pointed at it. It becomes visible the moment the rollout starts.

Why the model gets blamed anyway

Blaming the model is the easiest explanation, because it's the newest and least familiar part of the system. Everyone can point to it. Almost nobody can point to the HR policy that has been wrong since a reorganisation two years ago, because almost nobody has read it recently enough to know.

That imbalance matters. Switching vendor or adjusting a prompt feels like a concrete fix. Checking years of documentation across every department that now expects the AI to work feels like a huge, unglamorous job with no clear owner, which is why it tends not to happen until something goes wrong publicly enough to force it. By then, trust in the rollout has usually taken a knock that's harder to repair than the documents ever were. And the stakes are real: Gartner expects 60% of AI projects without AI-ready data to be abandoned through 2026 (Gartner, 2025).

There's also a scale effect that manual processes never had to deal with. One outdated policy in an obscure folder was a minor risk when only the few people who knew where to look might find it. An assistant with broad access will serve that same document to anyone who asks a related question, in exactly the same tone it uses for information that's completely accurate. The rollout doesn't just expose more content. It exposes it more evenly and more often than search ever did.

What a rollout needs before it happens

The lesson from a pilot that worked isn't that the organisation is ready to scale. It's that the slice of knowledge the pilot touched was ready. Treating those as the same thing is where most rollouts get into trouble, because it skips checking whether the rest of the company's knowledge is in similar shape.

That doesn't mean re-running the pilot everywhere at once. It means applying the same standard the pilot team used, even informally, to every folder and department about to come into scope:

  • Are there questions people will ask that have no documented answer?
  • Are there contradicting versions of policies that different teams have never reconciled?
  • Is there outdated content in departments that haven't been actively maintained?
  • Does the newly included content have an owner who would notice if it needed fixing?

Answering those questions before the rollout, rather than after the complaints start, is the difference between a launch that holds and one that loses trust department by department.

How Anchoris fits into a rollout

This is where Anchoris earns its place, because the problem isn't really about the pilot. It's about what happens when an organisation extends an AI's reach beyond the tidy environment the pilot ran in.

Anchoris connects to the shared drives you choose in Google Drive, or the sites you choose in SharePoint, and checks the documents your assistant answers from for contradictions between documents, outdated and expired content, people named as owners or contacts who have left, duplicates, unfinished placeholders and broken links. As each new department comes into scope, you add its drives and get first results within an hour. Every issue is ranked by how many people can open the document, Anchoris suggests who should fix it, and approved workflows fix issues at the source, with every run logged. A quality score from 0 to 10 shows where each set of documents stands before you open it up.

You choose which drives Anchoris can reach, and nothing changes without your approval. It works alongside the assistant you rolled out, whether that's Copilot, Gemini, Claude or Glean: fix the source once, and every assistant that reads it answers correctly.

The takeaway

A successful pilot proves that a small, carefully bounded slice of an organisation's knowledge holds up in real use. It says very little about the rest, because the rest was never tested. The problems that appear at scale aren't a sign the technology got worse. They're a sign the knowledge underneath was never evenly ready, and the rollout was the first thing to go looking.

Check your documents before the next phase of your rollout.

Book a demo

Frequently asked questions

Why does an AI pilot succeed but then give wrong answers after a company-wide rollout?

A pilot is usually limited to one team and a small set of folders people already rely on, so its success only proves that slice of knowledge held up. A company-wide rollout extends the AI into HR policy, old process pages and content nobody has reviewed in years, none of which the pilot had to answer from.

Does an AI model get worse when it's rolled out to more people?

No. The model and infrastructure usually don't change between a pilot and a full rollout. What changes is how much documentation the AI answers from, and that usually contains far more gaps, contradictions and outdated content than the set the pilot used.

Why do teams blame the model when a rollout starts giving wrong answers?

The model is the newest and most visible part of the system, so it's the easiest thing to point to. Almost nobody can name the outdated document behind a given wrong answer. Switching vendor feels like a concrete fix, while checking years of documentation feels like a job with no owner, so it gets skipped.

Why is one outdated document riskier once an AI tool can read it?

An outdated document in an obscure folder was a minor risk when only a few people knew where to find it. An AI assistant with broad access serves that document to anyone who asks a related question, in the same tone it uses for accurate information.

What should you check before expanding an AI pilot to the rest of the company?

Apply the pilot's standard to every folder and department coming into scope: questions with no documented answer, contradicting versions of policies, outdated content in departments nobody maintains, and new content with no owner who would notice if it needed fixing.

Does a successful AI pilot mean the rest of the company's knowledge is ready too?

No. A successful pilot shows that the narrow slice of knowledge it used was in good shape. It says little about the rest of the organisation, which was never part of the test.

How does Anchoris help during a rollout?

As each department comes into scope, you connect its drives and Anchoris checks the documents for contradictions, outdated content, people who have left, duplicates, placeholders and broken links, with first results within an hour. Issues are ranked by how many people they reach, with a suggested owner and approved workflows to fix them at the source.

Next article

Why does AI give wrong answers? Often it is the documents

Read next

More from the blog

AI readiness
AI readiness12 min read

AI readiness checklist: 8 signs your company's knowledge is ready for an AI assistant

AI doesn't create bad information, it scales it. Eight checks that show whether your documents are ready for an assistant.

Guides
Guides9 min read

How to find duplicate files in Google Drive

Google Drive has no one-click duplicate finder. Four manual methods, where they break down, and what it takes to stop copies coming back.

Guides
Guides11 min read

The anatomy of a documentation audit: what to look for

Most audits end as a 400-row spreadsheet. The four things a useful audit looks for, and how to keep its findings from decaying.