Replay
Agent 4 · claim verification
BehaviorSense watches for decline in an older adult living alone. A language model writes the daily summary, but it never sees video, and it never decides whether it told the truth. Four deterministic checks do that, and anything that fails one is withheld rather than flagged.
Resident ·
The checks below each claim are the real arithmetic, run in your browser against the same evidence table the model was given. Faults are injected here so the checks are visible — in the measured run, Qwen2.5‑7B failed 6.5–8.2% of its claims unprompted.
The verifier
Using a model to check a model shares its blind spots. These are arithmetic: they resolve a citation, compare two numbers, recompute a percentage, and read the sign of a change. Counts are from 498 claims generated in a single offline session.
C1
Does the cited evidence row exist for this day?
0 failures
The model is handed resolvable references rather than asked to compose them, so it has never once invented a citation. That is a design choice doing real work, not a property of the model.
C2
Is the quoted number the recorded one, within 2%?
39 failures
Where nearly everything goes wrong. The number is in the prompt, and the model still restates it loosely.
C3
Does the quoted change hold up when recomputed, within 2 points?
0 failures
A percentage against a zero baseline is undefined, not 0%. Quoting one there is itself a failure.
C4
Does the sentence match the sign of the change?
3 failures
The dangerous one. “Walking improved to 700 s” quotes the right figure and inverts the meaning — C1 to C3 all pass it, and a caregiver remembers the sentence, not the caveat.
The pipeline
Agents do not call each other. They hand over versioned schema objects, each one persisted and independently replayable. The seam names below are the actual types.
Agent 1
RTMO pose, tracking, and open-set re-identification, so a measurement belongs to a person rather than to a box on screen. Strangers are rejected at a fitted threshold instead of being enrolled by accident.
emits PersonObservation
Agent 2
A four-stream ST‑GCN++ ensemble over 30-frame skeleton windows, temperature-calibrated so its confidence means something, then smoothed into segments.
emits ActivitySegment
Agent 3
Thirty-day robust baselines per person, drift detection, and ten alert rules. Statistical rather than learned, because a decline has to be explainable to the person acting on it.
emits BehaviourState
Agent 4
Writes the summary as separate, citable claims — then every claim goes through C1–C4 before anyone sees it.
emits CaregiverReport
The model receives a BehaviourState: numbers,
baselines, deviations, alerts. No endpoint accepts video, and none returns
unverified output — so “the model never sees pixels” is a property
of the interface rather than a promise about how carefully it is called.
Measurements
Training and evaluation run offline on Kaggle. Where a number is weak it is printed weak, because a results table that only contains good news cannot be checked.
Fall detection
0.822AUPRC
AUPRC, not AUROC — at a 0.49% positive rate AUROC flatters. An operating point quoted per hour needs real negative hours to be a decision rather than an artefact.
Activity recognition
0.189mean-class, 20 classes
The weak result. Four of the worst classes have no skeleton-visible evidence at all — a pill bottle and a phone are the same pose — which is why object context is part of the architecture rather than an afterthought.
Identity
0.355fitted threshold τ
The control is the load-bearing row. Untrained embeddings scoring 0.534 through the identical protocol is what rules out a protocol bug that would have flattered anything.
Decline detection
5.07false alerts / 100 days
Five separate defects sat between 67.66 and 5.07, each found by evaluation rather than by reading the code.
Calibration
Before the verifier was pointed at a real model it was pointed at two generators whose answers were already known: one that reports the evidence honestly, and one that corrupts a controlled fraction of its claims. The honest one has to score zero, and the corrupted one has to track its own injection rate.
injected measured
Zero on honest output means the verifier has no false positives, so every rate below it is real. Tracking the injection rate means it does not saturate, so it can tell two models apart.
Forcing the model to emit JSON that matches the claim schema is the standard fix. Measured over ~500 claims per arm it made no difference: 8.2% constrained against 6.5% free, z = 1.06, p = 0.29, with confidence intervals overlapping across most of their range — at 7.5× the decoding time.
What the grammar does buy is a guarantee: it cannot emit a claim that fails the schema, where free decoding merely happened not to. In a caregiver-facing system that is worth paying for. It is not a faithfulness result, and it is not written up as one.
An earlier run put the constrained arm at 39.9%. That figure was our own prompt: the model was never shown the shape of the JSON it was being scored against, so it lost claims for guessing an envelope nobody had described to it.
Transfer is uneven and two corpora sit close to chance. Falls in these datasets are acted, and camera geometry differs per site; the honest reading is that a new home needs its own calibration, not that the model generalises.
Status
Final-year engineering project, in active development. The numbers on this page come from completed offline GPU sessions; the work below is what is currently running.
Nothing on this page is a mock-up. The report at the top runs the verifier's arithmetic; every measurement is reproducible from the repository with one command.
Read the code Backend, agents, notebooks and 152 tests