Why Governed AI Needs More Than Provenance

This post summarizes a working paper by Ryan Lewis and Ramon Martin.

A lot of AI systems can be drawn as a neat pipeline: take in some records, run a model, generate a response, update another system, and feed the result back in.

The diagram is simple. The authority inside it usually is not.

An authenticated record is not the same thing as a model output. A model output is not an approval. A drafted notice is not necessarily cleared for release. A successful API call does not prove that the intended real-world outcome occurred. And a readback from a system should not quietly become accepted evidence just because the workflow produced it.

That is the problem we examine in our working paper, Separating Evidence, Inference, Communication, and Effect in Governed AI Systems.

The basic idea is that AI-enabled systems should track more than where an artifact came from. They should also track what kind of artifact it is, what it is allowed to become, and who or what is allowed to move it from one state to another.

Provenance is only half the job

Most engineering teams already understand the value of provenance.

When a system produces a score, recommendation, summary, or forecast, we want to know which records went into it, which model or rule set produced it, which version was running, and what parameters or context were used.

That is necessary, but it does not answer a different question:

What standing does the result actually have?

A system can preserve perfect lineage and still make a bad leap. It might treat a calculated score as an accepted fact. It might send a generated draft before anyone or anything has authorized its release. It might let the same component recommend, approve, and execute an action. It might take an observed result and feed it back into the knowledge base as though it had already been reviewed and accepted.

Provenance tells you where something came from. It does not, by itself, tell you what the system is allowed to do with it.

What a separated workflow looks like

Consider a system that reviews an application.

The submitted application may be accepted as evidence for a specific purpose. A rule or model then evaluates that evidence and produces a derived result. Another component uses the application and result to draft a notice.

So far, nothing has been approved or executed.

The derived result may support a proposed status change, but the result should not authorize that change by itself. A separate policy or adjudication step decides whether the proposed effect is allowed. An executor then applies the authorized change and records what happened.

The notice follows a similar path. Drafting text and releasing it are different operations.

That distinction matters because sending a communication can itself have consequences. A notice may change someone’s obligations, start a deadline, disclose sensitive information, affect access, or create a public record. In those cases, release is not just a formatting step. It is an effect that may need its own authorization.

After execution, the system may read the updated status back from a database or external service. That readback is an observed outcome. It should not automatically become accepted evidence for the next decision. An explicit acceptance step records that the observation has been accepted, by whom or under which policy, and for what purpose.

The workflow therefore keeps several things separate:

• accepted source material and calculated results;
• results and approvals;
• draft communications and released communications;
• proposed actions, authorized actions, and executed actions;
• execution records and observed outcomes;
• observed outcomes and accepted evidence.

The exact names are less important than the boundary. An artifact should not silently acquire another artifact class’s authority.

This is not an argument for more microservices

Keeping these states separate does not mean every box needs its own service, database, deployment, and queue.

A single application can preserve the distinctions with typed records, explicit transition methods, separate roles, scoped credentials, and authorization checks.

The reverse is also true. A system can be split across ten services and still collapse authority if one caller can relabel outputs, bypass approval checks, or send an analytical result directly to an operational tool.

The important boundary is logical, not physical.

In practice, this can mean giving each artifact a declared class and lifecycle state, separating an action request from its authorization, requiring tools to accept an authorized effect rather than an arbitrary model payload, and recording each consequential transition in an attributable receipt.

It can also mean avoiding silent overwrites. When accepted information changes, the system can create a successor record and mark the prior one as superseded instead of making the history disappear.

None of these controls is particularly exotic. The harder part is applying them consistently across the entire workflow.

Making the rules executable

The paper includes a small deterministic Python implementation of the model.

The implementation completed 250 seeded valid workflows while preserving the expected artifact sequence and receipt chain. It also blocked or detected 2,250 attempts across nine classes of prohibited transitions. When individual guards were removed, the corresponding prohibited paths became reachable.

The implementation is intentionally compact, synthetic, and single-process. It does not prove that the taxonomy is complete, that an organization’s policy is legitimate, or that an authorized decision is a good decision.

What it does show is that these boundaries do not have to remain boxes in a governance diagram. They can be expressed as code, exercised in tests, and deliberately weakened to see which guarantees disappear.

That is a more useful starting point than relying on documentation that says a human is “in the loop” without defining what the human is authorizing, when authorization occurs, or whether the system can bypass it.

Questions worth asking during a design review

For an AI-enabled workflow, a few questions expose a surprising amount of ambiguity:

1. Is this artifact accepted source material, a derived result, or a communication?
2. Who or what is allowed to change its status?
3. What turns a proposed action into an authorized action?
4. Is this communication still a draft, or has its release been approved and recorded?
5. Is this feedback merely observed, or has it been explicitly accepted for future use?

If the system cannot answer those questions, adding another audit log probably will not solve the underlying problem.

The larger point is not that every organization should adopt our exact labels. It is that standing and authority should be visible before AI-generated material starts changing records, notifying people, moving resources, or steering operations.

The full working paper develops the state model, transition rules, implementation, controlled evaluation, and limitations in more detail.

Read the working paper on Zenodo.

The reference implementation and reproduction materials are also available.