Your AI Moat Is the Learning Loop

Every day, employees teach AI systems how their company actually works.

A support lead corrects the escalation policy the model misread. A product manager rejects a plausible roadmap because it ignores a promise made to one enterprise customer. A finance analyst shows the agent which revenue exception belongs in the forecast. An engineer explains why the clean-looking fix would fail in production.

Much of that instruction never becomes reusable company knowledge.

Satya Nadella has given the risk a useful name: the Reverse Information Paradox. His argument starts with economist Kenneth Arrow's 1962 account of the information problem. A buyer cannot know what information is worth until the seller reveals it, at which point the buyer already has it. AI reverses the exposure. The customer pays for a model and then supplies the proprietary context required to make it useful.

Nadella writes that firms can end up paying for intelligence twice: once with money, then again with their own knowledge.

The warning lands. But it also leaves a more common failure half-explained.

Even if the provider never uses our conversations to improve its model, our own company can still lose the lesson. A useful correction may stay inside one chat, so the next employee has to teach the AI the same thing again. The provider can respect every enterprise privacy promise while the company keeps forgetting what its people already learned.

That is the operating problem product leaders need to solve.

This week in AI News

Three developments this week matter for product leaders:

• Meta AI Adds Human Review Before Parent Alerts — Meta will manually review conversations flagged for possible teen suicide or self-harm before alerting a supervising parent.
• Claude for Teachers Turns Curriculum Into Product Infrastructure — Anthropic has connected Claude to teaching standards, established curricula, classroom tools, and K-12-specific privacy terms.
• Australia Puts AI Infrastructure and Rights Into One Framework — Australia's proposed rules connect data-centre power and water use with creative rights and AI training.

The knowledge companies keep losing

Nadella calls the traces created around AI work “intelligence exhaust.” The name makes them sound disposable. They are often a record of how the company actually works.

A manager's correction may contain a pricing rule that never reached the handbook. A rejected answer may show what compliance will approve. A failed agent run may expose a broken process. These small interactions hold operating knowledge that would otherwise remain scattered across people and chats.

Most major enterprise AI products say they do not use business customer content to train their general models by default. The exact terms still matter and should be checked for the product being used. But a privacy promise solves only part of the problem. It may stop the provider from learning from our work. It does not help our own company remember the correction.

Leakage risk and learning-loss risk

Consider a support team using an enterprise assistant.

An agent drafts a reply to a customer asking for a refund after a delayed infrastructure project. The draft follows the published policy. A senior support manager changes it because this customer is part of a strategic renewal, the delay involved a known integration defect, and a narrow concession has already been approved by finance.

The correction contains more value than the first draft. It combines account context, commercial judgment, product history, and an exception rule.

Several things can now happen.

The corrected exchange might remain in a private chat until retention expires. Another agent may make the same mistake next week. The manager may paste the lesson into a document nobody maintains. Or the company can turn the case into a governed artifact: an eval example, an exception rule with an owner and expiry date, a trace linked to the outcome, and a reusable piece of context available to the next authorised workflow.

The vendor can follow a strict zero-training commitment in all three cases. The difference is whether the company learns.

This gives us two separate risks:

• Leakage risk: knowledge crosses the intended trust boundary or improves an external system without the firm's consent.
• Learning-loss risk: useful feedback occurs inside the firm but never becomes reusable institutional capability.

Security teams naturally focus on the first. Product and operating leaders need to own the second as well.

Learning loss is easy to miss because employees still get local productivity gains. The support manager finishes the reply faster. The PM gets a better draft after three corrections. The engineer closes the ticket. Each person may feel more productive while the organization repeatedly pays for the same lesson.

That is an expensive form of amnesia.

What the company should keep

Companies do not need to build a foundation model. They should keep the material that makes any model work better for them.

Private evals define what good means

Public benchmarks tell us whether a model can code, reason, retrieve facts, or operate a computer under standard conditions. They cannot tell us whether a pricing recommendation fits our margin rules, whether a generated PBI meets our engineering team's acceptance standard, or whether a support response protects a particular customer relationship.