Five months, six profile builders, one silent spec drift

We build tooling for UCP, the Universal Commerce Protocol: validators, a generator, a WordPress plugin, an npm package, a GitHub Action. Staying current with the spec is the whole product.

On August 31 I ran our own AI Agent Test against ucptools.dev and it came back an F, 25 out of 100. Transport 0/6. Schema Loaded 0/12. Endpoint 0/12. Signing Keys 0/7.

None of it was broken. Every one of those zeros was a false negative, produced by our validator disagreeing with our generator while both modelled a version of UCP that had stopped existing in April.

The audit that followed took the rest of the day. If you build anything that reads or writes a versioned spec, the mechanism that hid this is worth recognising.

The drift was older than the release that exposed it

UCP shipped v2026-08-25 in late August, which is where the investigation started. The gap turned out to be five months wide rather than six days.

The profile shape changed in 2026-04-08:

// 2026-01-11: services keyed by namespace, transports nested under their name,
// capabilities an array of objects carrying their own `name`.
"services": {
  "dev.ucp.shopping": {
    "rest": { "schema": "...", "endpoint": "https://shop.example/ucp" }
  }
},
"capabilities": [{ "name": "checkout", "spec": "..." }]

// 2026-04-08 and later: services are arrays, transport is a field,
// endpoint is flat, capabilities are a map keyed by name.
"services": {
  "dev.ucp.shopping": [
    { "transport": "rest", "endpoint": "https://shop.example/ucp", "schema": "..." }
  ]
},
"capabilities": { "checkout": [{ "spec": "..." }] }

Payment handlers moved from root $.payment into $.ucp.payment_handlers in the same release, and signing_keys became keys in 2026-08-25.

We were still emitting the January shape in September.

Six builders, and the change had to be made in all of them

The audit was scoped to one defect in the validator. It found the same stale shape in six independent places:

There was more outside the table: a WordPress plugin live on wp.org declaring 2026-01-15 and failing our own validator with nine errors, an npm package five weeks behind its own repo, and a Shopify app depending on that package at a version from March.

The table is the root cause. A spec change had to be made in six places, so it was made in zero. Each builder was written for a different surface at a different time, each looked reasonable on its own, and no single change ever felt like the migration.

Why the tests were green

Three things kept this invisible, and all three are worth checking for in your own repo.

The first was a fixture that agreed with the code. We had a file called official-sample-profile-2026-04-08.json, with a header comment citing the upstream samples repository. It was not official. It was a 2026-01-11-shaped profile with the version strings swapped, and it passed because it agreed with the validator while both disagreed with the spec. A fixture built from your own output records your behaviour instead of testing it.

The second was tolerance. Our normalize helpers already handled the current shape, under names like "Shopify style" - something met in the wild and accommodated as a vendor variant rather than recognised as the spec. Because the code could read correct profiles, nothing ever failed loudly enough to make anyone ask why two shapes existed.

The third was short-circuiting. Our pipeline skips rules and network validation whenever structural validation reports an error. Two of the validator's three defects were errors, so they did more than add noise: they suppressed every real check behind them. One false error, and no analysis at all. If your validation pipeline has stages, find out which failures silence the later ones.

What actually caught it

A fixture built from the published spec repository at the tag, cross-checked against source/schemas/, rather than from our own samples or from the release notes.

The method turned out to matter more than I expected. Across this work, suspicions taken from the changelog were overturned five times by reading the published schemas or exercising the running code, and four of the six items on my original checklist did not survive contact with the real fixture. Two of them:

• We hard-errored when identity linking omitted config.supported_mechanisms. Grepping source/ and docs/ at all three published tags returns zero hits for that field. We had invented it.
• We warned that EdDSA was an unsupported signing algorithm. The spec lists it as well-known, recommends Ed25519 for Web Bot Auth interop, and says verifiers MUST tolerate algorithms they do not recognise. We replaced the closed-set check with a rule the spec does state as a MUST, that alg must agree with a well-known crv, so a real check took the place of the wrong one instead of leaving a gap.

Grep could not have found the worst cases either. Three surfaces declared version strings that are not UCP releases at all: 2026-01-14, 2026-01-15, and a generator stamping today's date. The real releases are 2026-01-11, 2026-04-08 and 2026-08-25, so a search for known versions matches none of those three. Reading the files and validating what they produced did.

Fix 1: one assembler, and versions as data

All six builders now call one pure, isomorphic module, and it is the only thing that knows how a profile is laid out:

export function assembleProfile(parts: ProfileParts): UcpProfile {
  const ucpVersion = parts.ucpVersion || CURRENT_UCP_VERSION;
  const defaults = ucpProfileDefaultsFor(ucpVersion);
  // ... shape services, capabilities and payment handlers per `defaults.shape`
}

The per-version facts sit in a data table rather than in branches scattered through the builder:

'2026-01-11': {
  shape: 'legacy',
  keysField: 'signing_keys',
  paymentHandlersAt: 'root.payment',
  urls: { /* the unversioned URLs that tag really specified */ },
},
'2026-04-08': {
  shape: 'map',
  keysField: 'signing_keys',
  paymentHandlersAt: 'ucp.payment_handlers',
  urls: { /* version-scoped paths */ },
},

Two decisions there are load-bearing. The generator is version-aware rather than version-bumped: ask for 2026-01-11 and you get the legacy shape, because profile_schema.json at that tag really does define root payment. The old output was correct for its own version. And the next spec release becomes an entry in a table instead of a rewrite.

Re-versioning six builders in place would have produced the same output and set up a seventh divergence later. Collapsing them was the fix.

Fix 2: guards that fail the next time

Self-validation on its own would not have been enough. The WooCommerce guide's stale snippet nearly passed our validator, because the validator and the content shared the same wrong model. So the guards look for disagreement with an external source, and for disagreement between our own surfaces:

• own-surfaces-validate.test.ts requires that what we publish passes what we sell. Our /.well-known/ucp has to declare a real, current version and validate with zero errors.
• tools-shape-alignment.test.ts runs identical input through every builder and requires one answer.
• plugin-zip.test.ts checks that the plugin declares nothing it cannot honour.
• npm-publish-drift.yml fails CI when package.json is ahead of the registry. It caught its own case on day one.

The outcome

Our own AI Agent Test went from F/25 to D/62, entirely by removing false negatives. Every generated profile now points at schema URLs that resolve. We published npm 2.1.0 after five weeks of unpublished fixes, moved the Shopify app off its March build, and shipped version 1.2.0 of the WordPress plugin.

We checked legacy profiles explicitly and they still validate clean. Merchants on the old shape were not asked to do anything.

Three things to take away

1. Count your builders before you plan a migration. If the same artifact is constructed in more than one place, the spec bump is a refactor, and the duplication is what will make you miss it.
2. A fixture from your own output tests nothing. Build it from the published spec, at the tag, and record where it came from. Ours ships with a PROVENANCE.md.
3. Audit artifacts rather than identifiers. A per-surface audit finds only the surfaces you point it at, and a grep for version strings finds only the versions you already know. The two worst offenders here declared versions that were never released.

None of this was hard once it was visible. It stayed invisible for five months because every individual piece looked fine and nothing in the system was ever required to compare them.

UCP is an open standard co-developed by Google and Shopify. UCPtools is an independent community tool and is not affiliated with either company. Every schema detail above was read from the published specification in the Universal-Commerce-Protocol/ucp repository, not from the release notes.

You can check a profile against v2026-08-25 with the free validator at ucptools.dev, or run it in CI with the UCP validate GitHub Action. If you are migrating from an older version, the silent breaking changes in v2026-08-25 is the companion piece to this one.