Build a UCP Watchdog: Catch the Production Breaks Your CI Never Will

You wired UCP validation into CI. Every push runs the checks, every PR gets a score, and a bad profile fails the build before it merges. Good - that is the right baseline.

Here is what it does not catch: the break that happens when nobody touches the code.

The standard here is UCP (Universal Commerce Protocol) - an open standard that gives AI shopping agents a machine-readable entry point to a store at /.well-known/ucp. (Quick disclaimer: UCP is owned and maintained by Google and Shopify. UCPtools, which I work on, is an independent community tool - not affiliated with either.)

A CI gate is triggered by your commits. But a UCP profile is a live production surface, and most of the things that break it are not commits at all:

• A TLS certificate renews and propagates to your origin but not to every CDN edge.
• A capability schema host your profile references goes down - someone else's outage, your broken profile.
• A CDN or DNS change starts serving a cache page or a redirect at /.well-known/ucp.
• Your platform (Shopify, BigCommerce, a WooCommerce plugin update) quietly changes the served manifest or strips the Content-Type: application/json header.
• A signing key rotates in your infra but not in the published profile.

None of these trips a build, because there is no build. Your CI is green. Your store works fine for human browsers. The only thing that regressed is the machine-readable layer that no human ever visits - and the AI agent that hits it does not file a bug. It just leaves for the next merchant whose profile answers.

CI catches what you break on merge. A watchdog catches what breaks itself. You need both.

What a Watchdog Actually Watches

A pre-merge gate asks "is the profile I'm about to ship valid?" A watchdog asks a different question on a schedule: "is the profile that is live right now still valid, from outside, the way an agent sees it?"

Two design rules make the difference:

1. Check from outside your network. A check that runs inside your own infra can hit a warm cache or an internal route and report healthy while external agents get errors. Fetch your public URL over the public internet.
2. Compare against a baseline, not just against pass/fail. A profile can stay technically valid while its score quietly slides from A to C. Alert on regression from a known-good baseline, not only on hard failures.

Let's build it two ways: a dependency-free cron version, and a GitHub Action with Slack alerts.

The 10-Line Version: cron + curl

UCPtools exposes a public remote-validation endpoint that fetches a live domain's profile and runs the checks server-side. You can hit it from anything that runs curl and jq:

#!/usr/bin/env bash
# ucp-watch.sh - alert if the live UCP profile is broken
DOMAIN="mystore.com"

resp=$(curl -sS -X POST https://ucptools.dev/v1/profiles/validate-remote \
  -H "Content-Type: application/json" \
  -d "{\"domain\":\"$DOMAIN\"}")

ok=$(echo "$resp"     | jq -r '.ok')
errors=$(echo "$resp" | jq -r '[.issues[] | select(.severity=="error")] | length')

if [ "$ok" != "true" ] || [ "$errors" -gt 0 ]; then
  codes=$(echo "$resp" | jq -r '[.issues[] | select(.severity=="error") | .code] | join(", ")')
  curl -sS -X POST "$SLACK_WEBHOOK_URL" -H 'Content-type: application/json' \
    -d "{\"text\":\":rotating_light: UCP profile for $DOMAIN is broken: ${codes}\"}"
fi

The endpoint returns the live result, shaped like this:

{
  "ok": false,
  "profile_url": "https://mystore.com/.well-known/ucp",
  "issues": [
    { "severity": "error", "code": "UCP_SCHEMA_FETCH_FAILED",
      "path": "$.ucp.capabilities[0]", "message": "...", "hint": "..." }
  ],
  "validated_at": "2026-06-05T14:33:57Z"
}

Schedule it and you have a watchdog:

*/15 * * * * SLACK_WEBHOOK_URL=https://hooks.slack.com/... /opt/ucp-watch.sh

Now UCP_SCHEMA_FETCH_FAILED or UCP_ENDPOINT_NOT_HTTPS showing up at 3am - hours after a cert renewal, with no deploy in sight - pages you instead of silently costing you agent traffic.

The GitHub Action Version: scheduled, with a baseline

If your store already lives in GitHub, you can run the same idea on a schedule: trigger and reuse the existing ucp-validate-action - the same action people put in CI - but pointed at your live production domain and run on a clock instead of on push. The difference is entirely in the trigger and what you do with the result.

name: UCP Watchdog
on:
  schedule:
    - cron: '*/30 * * * *'   # every 30 minutes
  workflow_dispatch:          # let me run it by hand too

jobs:
  watch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - id: ucp
        uses: Nolpak14/ucp-validate-action@v1
        with:
          domain: 'mystore.com'   # your LIVE domain, not staging

      - name: Alert on regression
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
        run: |
          score="${{ steps.ucp.outputs.score }}"
          grade="${{ steps.ucp.outputs.grade }}"
          found="${{ steps.ucp.outputs.ucp-found }}"
          baseline=$(cat .ucp-baseline 2>/dev/null || echo 0)

          echo "Live: score=$score grade=$grade found=$found | baseline=$baseline"

          if [ "$found" = "false" ] || [ "$score" -lt "$baseline" ]; then
            curl -sS -X POST "$SLACK_WEBHOOK_URL" -H 'Content-type: application/json' \
              -d "{\"text\":\":rotating_light: UCP regression on mystore.com - score ${score} (grade ${grade}), baseline ${baseline}\"}"
            exit 1
          fi

Commit a one-line baseline file the first time you go green:

echo 90 > .ucp-baseline   # your known-good score

The action exposes score, grade, ucp-found, passed, and result-json, so you can build whatever alerting logic you want on top. The point is that the trigger is a clock, the target is production, and the comparison is against your last known-good state.

Alert Hygiene (so you don't train yourself to ignore it)

A watchdog that cries wolf gets muted, and a muted watchdog is worse than none. Three things keep it honest:

• Baseline, don't just pass/fail. A slow slide from grade A to grade C is the regression you most want to know about, and a binary "still valid?" check will miss it entirely. Diff the score.
• Debounce flaps. A single failed fetch can be a transient network blip. Require two consecutive bad checks before paging, or alert on a sustained drop rather than one data point.
• Bump the baseline when you improve. When you legitimately raise your score, update .ucp-baseline in the same PR. The baseline is a ratchet - it should only move up on purpose.

Platform Notes

The watchdog is platform-agnostic - it reads the open /.well-known/ucp standard, not platform internals - but the regression that pages you tends to differ by stack:

• WooCommerce: a caching or security plugin update that starts serving /.well-known/ucp from cache, behind a challenge, or as text/html.
• BigCommerce / headless: a frontend deploy or app change that moves an endpoint the profile still advertises, or a storefront-scope mismatch.
• Shopify: the platform changing what it serves at the well-known path out from under you.

In every case the failure is invisible until something fetches the live profile from outside and compares it to what you expect. That is the whole job of the watchdog.

CI proves the profile you wrote is correct. A watchdog proves the profile your customers' agents actually hit is still correct - at 3am, after a cert renewal, when no one shipped a thing. Both are a few lines of YAML. The merchants who win the agentic-commerce transition will treat the second one like uptime, because that is exactly what it is.

If you would rather not run your own, UCPtools does hosted monitoring with break-alerts across all four validation levels - start here. Either way: watch the live profile, not just the build.

UCP is an open standard by Google and Shopify. UCPtools is an independent community tool. Built by Peter at UCPtools.