Acceptance criteria for forward deployed engineers

Short version: acceptance criteria have to be agreed in writing before the project starts. Raise them on delivery day and they stop being criteria and become a dispute. A workable set answers three questions: which data the result is measured on, how a pass is decided, and who has the authority to say “accepted”. Below are three groups of criteria — function and performance, systems and integration, organisation and use — followed by a one-page template and a list of warning signs.

Two complaints come up again and again. The client says “just change this one thing” and acceptance slides another few weeks. Or twelve people sit in a room, one of them objects, and nobody signs — because whoever signs owns the problem after go-live.

Neither is a failure of character. Both are structural.

Why acceptance disputes are the default

The criteria were never fixed up front. The contract says “development completed” or “performance meets expectations”. Both sides agree what those words mean on the day of sign-off, by which point the leverage has shifted: the system exists, the final payment has not moved.

Nothing is measurable. Some contracts do carry a number — “accuracy of at least 90%” — without saying which dataset, who labels it, or who decides a disagreement. On a different dataset, 90% becomes 60%. Or 98%.

Nobody owns the definition of “done”. A sign-off meeting with no single owner is the safest possible meeting to attend: objecting costs nothing, signing carries all the liability. So nobody signs.

The cost of that arrangement is visible at industry level: more than 60% of enterprise AI attempts remain stuck in pilot. Not all of them stop because the technology failed. A fair share stop because nobody could define “finished”, so the work sits in a permanent state of “still being optimised” — impossible to accept, impossible to kill.

Group one: function and performance

1. The evaluation set is agreed by both sides and independent of the build. Cover routine cases, edge cases and explicit counter-examples. Each case must resolve to a yes or a no; where grading is unavoidable, state the consequence of each band rather than “we suggest further tuning”.
2. Write the freeze date into the contract. Frozen before development starts; cases added later belong to the next version. Without that rule the work never finishes, because a failing case can always be found.
3. Say who holds it. The client keeps the frozen version, both sides hold a copy with a checksum, and every change is logged with version, date, reason and who agreed.
4. Measure against your own baseline. Run the same cases through the current manual process and record the result. The comparison is improvement over the baseline, not perfection — demanding that the system beat human performance outright will reject projects that are worth doing.
5. Define which failures are unacceptable. Systematic errors on one class of case are a blocking defect; occasional errors on the long tail are usually a known limitation you can go live with.

Group two: systems and integration

One test covers this group: can your people run it without the original developer?

1. Monitoring and logs. Someone must be able to answer “how many items ran, how many failed, and what kind of thing failed” without reading code. Where the logs live, how long they are kept, who can see them.
2. An evaluation suite you can run. Delivered as something repeatable. One acceptance condition: your staff run it themselves, once, unsupervised — not watch a demonstration.
3. Integration logic. Interface inventory, how access is requested, retry and fallback behaviour, all documented. An implementation only its author understands quietly ties maintenance to one person.
4. A runbook. Who looks at what, and when it escalates, plus the known failure modes and what to do about each.
5. A cost model. Cost per item processed and how the monthly figure moves with volume — a formula your own team can apply, not a single number.
6. Ownership of the running system. Deployment is roughly 20% of the cost; the other 80% is keeping the system working through model upgrades, data drift and edge cases (John Sangyeob Kim of Solidroad, reported in Computerworld; the reference period is 18 months). Most contracts price only the first part. Acceptance criteria should say who owns the other 80%, and in what form.

Group three: organisation and use

1. Name the users. Not “the business” — the role and the person. The acceptance condition for training is that a trained person completes one full run unaided, including what to do when it fails.
2. Name the maintainer. A first responder inside your organisation and an escalation route with the supplier, both in the document.
3. Review at 30, 60 and 90 days. Which metric, who convenes it, and what result triggers what: continue, adjust scope, retire.
4. Allow for retirement. Write down what happens if nobody uses it — who decides, when, and on what basis. That is cheaper than pretending adoption is guaranteed.
5. A route for scope change. Who can raise a new requirement, through what process, and how it affects schedule and acceptance. An acceptance meeting that doubles as a requirements review is the most common way acceptance is lost.

Running the sign-off meeting

Who is in the room. Three roles, all required: someone who can sign today rather than “take it back to the team”; someone who will use the system daily; someone able to judge the implementation. Without the second you get a system nobody uses. Without the third the meeting becomes an argument about whether something counts as a bug.

Do not require unanimity. Appoint one owner, usually the head of the department that will use it. Everyone else’s input falls into two buckets: blocking defects — failures against the agreed criteria with a reproducible path — and non-blocking comments, which are logged for the next phase and do not extend acceptance.

Classify every issue in the room. Each point raised is a defect (the criteria asked for it and it is not there), a new requirement (outside the criteria, goes to change control), or a misunderstanding (clarify and record). The owner decides which, and the decision goes in the minutes. Debating all three together is the usual failure: one new requirement is enough to delay acceptance by a month.

Put a deadline and a default in the contract. For example: no blocking defect submitted in writing within five working days of submission for acceptance means acceptance. That is more effective than any amount of chasing.

A one-page template

Project:                                  Date of acceptance:
Scope of this acceptance (state what is explicitly out of scope):
Single sign-off owner (name / role):

1. Evaluation set
  Location:            Version or checksum:      Freeze date:      Held by:
  Pass rule (how each case resolves to yes/no):

2. Metric and baseline
  | Metric | Baseline (today) | Target | How measured | Data source |

3. Handover items (delivery date and how the client verifies each independently)
  [ ] Monitoring and logs   [ ] Evaluation suite   [ ] Integration logic and interface docs
  [ ] Runbook               [ ] Cost model

4. Decision
  [ ] Accepted   [ ] Accepted with conditions (auto-accepted once blocking defects close)
  [ ] Not accepted (cite the criterion that fails)

5. Open items and review
  Non-blocking comments:
  Day 30:              Day 60:              Day 90:
  Maintainer: client                    supplier

Eight signs this acceptance will turn into an argument

1. On delivery day the criteria are still adjectives — “meets expectations”, “feels smoother”.
2. The evaluation set was assembled by the supplier in the week before the meeting, and the client is seeing it for the first time.
3. More than half the issues raised are appearing for the first time. Nothing was validated along the way.
4. The person who has to sign is not a user of the system, and needs to “take it back to the team”.
5. Someone raises a new requirement in the room and is asked for a delivery date on the spot.
6. Nobody can say who maintains the system after go-live.
7. The discussion keeps returning to “can we just improve this bit” — an instruction that can never be marked complete.
8. The meeting has happened twice already, and both times ended in “let’s look again next week”.

Three or more, and the meeting is not worth holding yet. Rewriting the criteria is cheaper than meeting again.

FAQ

When should the criteria be set? Before the contract is signed, and no later than the kick-off. The technical approach can change during the project; the measurement rules should not. Every project that says “we will define the standard once we see how it performs” ends up arguing about the standard.

Can PoC acceptance and production acceptance be one event? We would not combine them. A PoC answers “is this worth further investment”, judged on validation metrics plus kill criteria. Production acceptance answers “is delivery complete”, judged on the three groups above. Merge them and projects that should stop at PoC keep spending, because “acceptance failed” is not an available outcome.

What if nobody gives a decision after submission? Separate the reasons: a blocking defect (fix it), a new requirement (change control), or nobody willing to carry the signature (an organisational problem — go back to the single owner). Then put a default in the contract: no written response within the agreed period counts as acceptance.

---

Acceptance criteria are not there to catch anyone out. They turn “finished” from an adjective into a clause. Our own habit is to fill this list in before kick-off: the cells we cannot fill are usually the ones that later become the delay.

If you are still deciding whether the project is worth doing at all, read What is a forward deployed engineer? and FDE vs outsourcing. To see how we deliver, look at how we deliver and our FDE outsourcing service. If you have a specific scenario in hand, talk to us for an assessment.