How to Pilot Governed Manufacturing AI Agents Across Existing MES and ERP Workflows

TL;DR

• A useful pilot depends on a bounded decision, a clear evidence contract, and staged autonomy gates. Model selection and a platform rebuild can wait.

• Choose one recurring decision and record its current outcome and decision time before the agent enters the workflow.

• Specify which MES and ERP records the agent may use, who approves its recommendations, and who handles missing or conflicting evidence.

• Run the loop in shadow mode first, then move to assisted decisions and bounded autonomy only when its audit record and outcomes meet agreed gates.

• This runbook covers how to operate the first loop. The decision-authority buyer’s guide covers who controls it.

How to scope a manufacturing AI agent pilot

A manufacturing AI agent pilot should test one operational decision against plant records and approval rules before it gains authority to act. That requires a measurable baseline, defined evidence and a person authorized to approve or resolve each proposed action.

A promising scheduling recommendation can stall when the pilot has no agreed rule for conflicting MES and ERP records. If the MES shows a machine down while the ERP still shows an open production order, the agent needs a defined source for each fact, a way to detect stale data, and a person authorized to resolve the conflict. Name who may resolve the conflict and authorize an action. Otherwise, a demonstration cannot show whether the agent can operate within the plant’s rules.

Governance belongs in the pilot design rather than in a review after the model works. Start with one bounded decision, its required evidence, and its approval and exception paths. Those limits give you a basis for testing the agent in shadow mode before considering actions in MES or ERP.

Choosing the one decision loop to pilot

Choose a decision that occurs often enough to test and stays within a clear operational boundary. A pilot could recommend how one plant handles a late material delivery on a defined production line. It should not attempt to optimize every scheduling decision at once. Specify what triggers the decision, which responses are allowed, and which cases remain outside the pilot.

A viable candidate recurs often enough to test, has a measurable outcome and has a named decision owner. If people must resolve the same ambiguity differently on every occurrence, narrow the candidate before testing an agent against it.

A scheduling exception can provide a recorded trigger and a decision that planners can compare with the agent’s proposal. For an example of the inputs a planner needs to weigh, see how AI can resolve scheduling conflicts across multiple lines. An on-time delivery intervention or an inventory exception may also work if you can identify the permitted response. Shift performance and frontline knowledge are possible entry points when you can isolate a decision, such as when to escalate a recurring downtime reason or apply an approved procedure. The shift handoff workflow guide shows where observation ends and verification and procedure approval begin. None of these categories defines a pilot on its own.

Write a decision statement before building the loop. For the late material example, name the shortage flag that starts it, the schedule changes an agent may recommend, the person who approves them, and the conditions that require escalation. You could measure the time between the flag and an approved change while excluding orders under a quality hold. If existing records cannot show when the decision began and ended, choose a narrower candidate with an observable result.

How to establish a baseline for the agent pilot

Before you run a manufacturing AI agent, define the current decision metric, its source, and the period you will use for comparison. For a scheduling exception, you might measure the time between an MES event marking a machine unavailable and the approved schedule change recorded in ERP. Agree on the start and end events before reviewing the data so different interpretations cannot change the result later.

Record the baseline over a window that captures the conditions the pilot will face. Keep the underlying event IDs, timestamps, and any excluded cases so you can check the calculation. If approvals happen outside ERP or MES, record their timestamps separately and identify who maintains that record. A missing approval time cannot support a decision latency measure.

Set the pilot’s outcome gates against that baseline before the agent makes recommendations. Specify what improvement would justify moving forward and which checks, such as missed commitments or an increase in manual corrections, would require a pause. At review, compare like conditions where possible. A faster decision time during a quiet production week does not establish that the loop improved decisions under the workload it was chosen to handle.

What evidence and system access does the agent need?

The evidence contract should specify which records the agent may read for one decision, what each record means, and when it becomes too old to use. For a scheduling exception, you might permit reads of ERP order priorities and MES job status while excluding employee records and unrelated production lines. Start with access that permits reads but no writes. Document the approved fields and source systems, and limit queries to the selected decision.

ERP and MES remain the systems of record. A downstream decision layer can compare their data and prepare a recommendation, but it should not silently correct either system. Require every cited fact to carry its source record identifier, the time the source recorded or updated it, and the time the agent retrieved it. If a work order changed after the agent retrieved it, the recommendation needs a fresh check before anyone acts.

Set ownership rules for facts that appear in both systems. For example, ERP might own the promised delivery date while MES owns the current job status. Record that assignment in the contract rather than letting the agent choose whichever value appears newer. When records conflict, the agent should identify both values and route the discrepancy to the named data owner. The same rule should apply when a required value exceeds its agreed freshness limit.

The contract should also define what the agent must produce when evidence falls short. A recommendation can cite the usable records, identify the missing or disputed fact, and request a human check. It should not present an action as ready for approval when its required evidence is stale or unresolved. Before granting write access in a later pilot phase, specify which approved action may change which MES or ERP field and require the action to recheck the evidence it relied on.

When does the pilot need an industrial ontology or semantic model?

A pilot needs a semantic model when its evidence contract cannot reliably identify the same equipment, process step, or event across systems. For example, MES may record a line stop under a work center, while SCADA records alarms under individual machine tags. If the pilot cannot determine which alarms belong to that stop, it should not use them to recommend an action.

Test the mapping with cases drawn from the decision loop. Can you trace each cited event to one asset, one production context, and a timestamp without relying on an operator to interpret conflicting identifiers? Check what happens when equipment names repeat across plants, an asset gets replaced, or ERP and MES group process steps differently. If a small, verified lookup table resolves the pilot’s limited set of identifiers, a direct mapping may be enough.

When those relationships change or span multiple systems, test whether the plant’s existing industrial ontology or semantic model can resolve them. This shop-floor decision example follows an ERP order, MES operation, and SCADA event through that contextualization step. Its role is to relate equipment, products, process steps, and events while retaining their source identifiers. The pilot must still test those relationships against actual records and route unresolved matches to a person rather than treating a plausible match as fact.

Humble sits downstream of that contextualization work. Humble consumes the plant’s contextualized information to support the selected decision. The plant controls the semantic model, while ERP and MES remain the systems of record.

Assigning approval rights and exception ownership

Assign a named approver and a backup for each decision the agent may recommend. For a scheduling exception, the production planner might approve a proposed sequence change, while the shift supervisor confirms that the affected line can execute it. Record which person can approve each action, who covers a shift change, and when an approval expires. Until those assignments are in place, the agent should recommend actions without writing changes to MES or ERP.

Route exceptions to an owner before the pilot starts. Send conflicting records or stale evidence to the person responsible for validating the source data. Send decisions outside the pilot’s limits to the operations owner, and specify who takes over if that person cannot respond within the decision window. Log the handoff and its resolution so an unanswered exception cannot silently become an approved action.

This runbook assumes your plant has already settled who may grant and withdraw the agent’s authority. If those rights remain open, use the decision authority buyer’s guide to resolve them before assigning pilot approvers.

How to advance the agent from shadow mode to bounded autonomy

In shadow mode, the agent proposes an action for the selected decision loop, but people continue making and recording the actual decisions. For each proposal, capture the source records, their timestamps, the constraints applied, and the action the operator took. Compare recommendations with the pilot’s baseline across routine cases and exceptions, not just cases the agent handles cleanly. Review disagreements with the people making the live decisions and check whether the agent used current evidence. The comparison should expose errors in the pilot’s rules before you consider write access.

Move to assisted decisions when the shadow audit shows that recommendations meet the pilot’s predefined quality threshold, cite current evidence, and route unresolved source conflicts to a person. In assisted mode, the agent prepares the proposed action and its evidence, and a named approver accepts, edits, or rejects it before anything changes in MES or ERP. Record the approver’s reason when a proposal changes. Compare decision time and operational outcomes with the baseline, while reviewing rejected proposals for recurring errors. A faster approval cannot justify expansion if the loop produces unacceptable exceptions.

Grant bounded autonomy only when assisted decisions in a defined action class meet the agreed outcome and exception thresholds without routine human correction. Specify the exact actions, constraints, and system writes the agent may perform. Route cases outside those limits to the named approver. Before enabling writes, test whether you can withdraw the loop’s permission and reverse an action using the affected system’s established procedure. Keep reviewing outcomes against the baseline after autonomy begins, and return the loop to assisted or shadow mode if evidence goes stale, exceptions exceed the agreed limit, or outcomes deteriorate. Use the audit record, rather than a fixed number of days, to decide whether the agent has earned the next level of authority.

Audit trails, revocation, rollback, and failure handling

Before an agent can write to MES or ERP, the pilot owner needs an event record that connects each proposed action to its evidence and authority. Record the recommendation, cited source records and timestamps, applicable constraints, decision logic version, intended write, and target record identifier. Add the named approver or the rule that permitted autonomous action. After execution, record what the destination system accepted, rejected, or changed, along with the observed outcome when it becomes available. Keep the decision record even if the write fails.

Revocation must work for one decision loop without disabling unrelated workflows. The digitalization team should define who can withdraw that loop’s authority, how the write permission is removed, and what happens to actions already queued. Test revocation before granting autonomy. Once authority is withdrawn, the agent may continue to prepare recommendations, but a person must approve any further write.

Plan rollback for each permitted action. For a reversible schedule change, record the prior values and verify that restoring them will not overwrite a later human edit. If an action cannot be safely reversed, such as releasing work that has already started, keep it behind human approval and define a corrective action instead. ERP and MES retain the authoritative records throughout.

Failure handling should route uncertain decisions to a person rather than force an answer. Missing evidence, stale timestamps, conflicting records, or a failed write should stop that action and notify its named exception owner. The owner can resolve the evidence conflict, approve a revised action, or leave the original state unchanged. Test these paths in shadow and assisted modes before the loop receives write authority.

When should you expand, pause or stop the agent pilot?

Review the pilot against the baseline metric, its agreed measurement window, and the audit record before changing its scope. Compare like operating conditions where possible. A shorter approval time, for example, does not establish a useful gain if the pilot also increased scheduling errors or pushed more exceptions onto supervisors.

Expand to another bounded decision loop only when the baseline metric improves, the audit record accounts for the decisions behind that change, and exceptions stay within the limits you approved. Keep the new loop separate so you can measure its performance and withdraw its authority without affecting the first.

Pause when missing records, stale data, or unresolved exceptions prevent a fair comparison. Assign an owner to fix the evidence or routing problem, then repeat the review over a defined window. Stop when a comparable record shows no worthwhile improvement, or when recurring failures breach the pilot’s agreed limits. In either case, ERP and MES remain the systems of record. Ending the agent pilot does not require replacing them.

How does this pilot runbook differ from the decision authority guide?

The decision authority buyer’s guide helps you determine who owns the decision logic, authorizes MES or ERP writes and can withdraw that authority. Settle those rights before assigning authority in a pilot.

This runbook starts once those authority boundaries are set. It covers the baseline, evidence, approval paths and outcome gates needed to operate and evaluate the first decision loop.

Test a Manufacturing AI Agent Pilot with Humble

If you have a candidate decision loop, take the Humble fit test to start scoping it against your current MES and ERP setup. Bring the baseline metric, required evidence, and approval owner into that discussion. Humble acts as a customer-controlled decision layer that consumes contextualized data. Your MES and ERP remain the systems of record.

Book a Call with Humble

If you want to review a candidate decision with Humble, you can book a call. Bring the current metric and its source in MES or ERP. You can work through the evidence the agent may use, who approves its recommendations, and which exceptions need a human owner.

Stay Current on Governed Manufacturing AI

Subscribe to Humble’s newsletter for practical guidance on evaluating governed AI decision loops across existing manufacturing systems.

Frequently asked questions

How long should shadow mode run?

Run shadow mode until the pilot has observed enough normal decisions and exceptions to test its rules against real MES and ERP records. Use the decision cycle and exception frequency to set the review point, rather than choosing a fixed number of days.

What evidence supports a move to assisted decisions?

Compare the agent’s recommendations with decisions made by named operators or planners, including cases where they disagree. Move to assisted mode when reviewers can trace each recommendation to current source records, resolve recurring data conflicts, and identify who will handle exceptions. The agent should still wait for a person to approve each action.

How does rollback differ from revocation?

Revocation removes the agent’s permission to make future autonomous decisions in a specific loop. Rollback addresses an action already taken, such as correcting an ERP record through the plant’s approved procedure. Log both events and retain the original decision record.

How should the pilot use existing ISA 95 mappings or an industrial ontology?

Use the plant’s existing mappings or semantic model when the pilot must reconcile equipment, order, or event identifiers across systems. The decision layer consumes that context, while MES and ERP retain their roles as systems of record. If a limited mapping resolves the pilot’s identifiers reliably, the first loop need not wait for broader ontology work.