BehaviorSense Caregiver reports with checked evidence

Behaviour monitoring · research prototype

A clearer daily report, with evidence you can check.

Read a caregiver summary alongside findings checked against recorded evidence. Five deterministic checks cover the structured claims. The model's summary and recommendation remain separate, unverified prose.

Daily report

Resident ·

Claims
—
Shown
—
Withheld
—
Evidence and original English report

Original model summary and recommendation. These paragraphs are not verified by C1–C5.

    The checks below each claim are the real arithmetic, run in your browser against the same evidence table the model was given. Faults are injected here so the checks are visible — in the measured run, the language model failed 2.8–3.7% of its claims unprompted.

    Demo faults

    The verifier

    Five checks, and not one of them asks a language model.

    Using a model to check a model shares its blind spots. These are arithmetic: they resolve a citation, compare two numbers, recompute a percentage, read the sign of a change, and ask whether the sentence quotes the figure at all. Counts are from 1,120 claims generated in the latest measured run.

    1. C1

      Citation resolves

      Does the cited evidence row exist for this day?

      1 failure

      The model is handed resolvable references rather than asked to compose them — one fabrication in five runs, against zero in every run before. A design choice doing real work, not a property of the model.

    2. C2

      Value matches

      Is the quoted number the recorded one, within 2%?

      0 failures

      This used to be nearly the whole residual — 39–42 per run, the model restating a number it had been given. The prompt now carries a filled template to copy from, and field misquotes went to zero.

    3. C3

      Percentage survives

      Does the quoted change hold up when recomputed, within 2 points?

      0 failures

      A percentage against a zero baseline is undefined, not 0%. Quoting one there is itself a failure.

    4. C4

      Direction agrees

      Does the sentence match the sign of the change?

      13 failures

      The dangerous one. “Walking improved to 700 s” quotes the right figure and inverts the meaning — C1 to C3 all pass it, and a caregiver remembers the sentence, not the caveat.

    5. C5

      Prose quotes the figure

      Does the sentence contain the number the field records?

      22 failures

      The newest, and now the largest category. The field can be exactly right while the sentence says “fell noticeably” — C2 is blind to that by construction, because it compares field against row and both agree. The sentence is what a caregiver reads.

    On your own footage

    Upload a clip to see the observations and caregiver report.

    Choose a video to see the recorded activities and a caregiver report. The report shows which findings passed the evidence checks.

    Drop a clip here, or

    Up to 60 MB and 12,000 retained frames, about ten minutes at 20 Hz. Longer uploads are refused. Each tracked person's activity is shown separately.

    Optional enrollment saves the subject's name and an identity embedding on this machine. Saved names, roles and embeddings are sent to Kaggle for matching on later uploads. To forget enrollment, stop the bridge, delete its gallery file and restart it.

    Connect a backend to analyse a clip.

    Activity values come from this clip. Comparisons use a simulated reference, not this person's recorded daily history. A short clip cannot establish a personal daily decline.

    The pipeline

    Four agents, and one boundary the whole design rests on.

    Agents do not call each other. They hand over versioned schema objects, each one persisted and independently replayable. The seam names below are the actual types.

    1. Agent 1

      Perception

      RTMO pose, tracking, and open-set re-identification, so a measurement belongs to a person rather than to a box on screen. Strangers are rejected at a fitted threshold instead of being enrolled by accident.

      emits PersonObservation

    2. Agent 2

      Activity

      A ST‑GCN++ ensemble over 30-frame skeleton windows, temperature-calibrated so its confidence means something, then smoothed into segments. The served configuration uses the bone stream alone — selected on validation, where averaging all four streams traded tail recall for head accuracy.

      emits ActivitySegment

    3. Agent 3

      Behaviour

      Fourteen-day robust baselines per person, drift detection, and ten alert rules. Statistical rather than learned, because a decline has to be explainable to the person acting on it.

      emits BehaviourState

    4. Agent 4

      Report

      Writes a summary, recommendation and separate citable claims. C1–C5 check the structured claims; the summary and recommendation remain unverified model prose.

      emits CaregiverReport

    The reporting model receives a BehaviourState: numbers, baselines, deviations and alerts. Uploaded video goes to the perception pipeline on Kaggle. Only structured evidence is sent to the reporting provider, and accepted claims must pass C1–C5. The report's free-text summary and recommendation are outside those checks.

    Measurements

    Every figure here came off a GPU, including the unflattering ones.

    Training and evaluation run offline on Kaggle. Where a number is weak it is printed weak, because a results table that only contains good news cannot be checked.

    Fall detection

    0.822AUPRC

    • Sensitivity0.951
    • False alarms0.99 / hour
    • Source-stratified AUROC, 4 corpora0.928–0.991

    AUPRC, not AUROC — at a 0.49% positive rate AUROC flatters. The 0.928–0.991 range is one binary checkpoint scored on each source’s shards, which include training windows; it is not a leave-one-corpus-out transfer estimate, and the fitted operating point has not been validated on an unseen recording setup. The 20-class activity ensemble’s own fall posterior sits at 0.47–0.60 on the same folds, which is why the two-model design exists.

    Activity recognition

    0.187mean-class, 20 classes

    • Best single streambone
    • Best macro‑F1 (deployed)0.146
    • Ensemble top‑1 / macro‑F10.375 / 0.128
    • Chance0.050
    • Toyota CS, 3D historical (13 cls)0.736
    • Toyota RTMO, served (15 cls)0.600

    The weak result is the hard protocol: Charades, measured actor-disjoint over 41 held-out actors. On the metric the behaviour layer actually consumes, bone alone wins macro‑F1 (0.146) and ties mean-class (0.187); the four-stream ensemble buys top‑1 but loses 1.8 macro‑F1 points, so the served configuration drops the ensemble. Four classes have no skeleton-visible evidence at all — a pill bottle and a phone are the same pose — which is why object context is part of the architecture rather than an afterthought. The Toyota row carries two numbers, not one: 0.736 is the historical 3D head on INRIA poses, test‑selected on the CS test subjects (epoch 48 of 60, picked by their score); 0.600 is the served RTMO checkpoint, 22 classes, 20 Hz, over the 15 classes present in its test partition. They differ in input, checkpoint, selection protocol and class support, so their gap does not isolate the cost of live pose extraction. Both are in results/evaluation.md.

    Identity

    0.355fitted threshold τ

    • Open-set AUROC0.998
    • ImageNet control0.534
    • Closed-set rank‑199.58%

    The control is the load-bearing row. Untrained embeddings scoring 0.534 through the identical protocol is what rules out a protocol bug that would have flattered anything.

    Decline detection

    5.07false alerts / 100 days

    • Recall24 / 24
    • Median latency4 days
    • Started at67.66

    Five separate defects sat between 67.66 and 5.07, each found by evaluation rather than by reading the code.

    Report faithfulness

    3.2%hallucination rate

    • Claims checked1,120
    • Grammar‑constrained2.8%
    • Unconstrained3.7%

    Every claim withheld before a caregiver reads it. Field misquotes went from ~40 per run to zero by giving the model a filled template to copy from; the residual is prose that never quotes the figure, which only C5 can see. The grammar’s p‑value for faithfulness is 0.40 — its value is a parseability guarantee, not a truthfulness one.

    Calibration

    A metric nobody tried to break is a decoration.

    Before the verifier was pointed at a real model it was pointed at two generators whose answers were already known: one that reports the evidence honestly, and one that corrupts a controlled fraction of its claims. The honest one has to score zero, and the corrupted one has to track its own injection rate.

    Measured failure rate against injected corruption 696 claims per condition · 116 simulated resident-days

    injected measured

    Zero on honest output means the verifier has no false positives, so every rate below it is real. Tracking the injection rate means it does not saturate, so it can tell two models apart.

    The result we did not want

    Grammar-constrained decoding did not reduce hallucination.

    Forcing the model to emit JSON that matches the claim schema is the standard fix. Measured over ~550 claims per arm it made no difference: 2.8% constrained against 3.7% free, z = 0.84, p = 0.40, with confidence intervals overlapping across most of their range — at 6.4× the decoding time.

    What the grammar does buy is a guarantee: it cannot emit a claim that fails the schema, where free decoding merely happened not to. In a caregiver-facing system that is worth paying for. It is not a faithfulness result, and it is not written up as one.

    An earlier run put the constrained arm at 39.9%. That figure was our own prompt: the model was never shown the shape of the JSON it was being scored against, so it lost claims for guessing an envelope nobody had described to it.

    Fall detection scored separately on four training sources source-stratified · AUROC · 3,064 windows across four corpora

    One binary checkpoint, trained across all four fall corpora, scored on each source’s shards — which include training windows. Unseen‑corpus transfer and the fitted operating point’s behaviour off its own validation set are not measured here. Falls in these datasets are acted, and camera geometry differs per site; the honest reading is that a new home needs its own calibration, not that the model generalises.

    Status

    Still being built.

    Final-year engineering project, in active development. The numbers on this page come from completed offline GPU sessions; the work below is what is currently running.

    • next Wiring object context into the classes skeletons cannot separate — medication, phone use, watching television.
    • next Staged-to-wild fall transfer on unscripted footage, the one protocol not yet measured.
    • done Training on the activity streams completed at its patience limits; segment-level accuracy after temporal smoothing is measured and in results/evaluation.md.

    The replay report runs the verifier's arithmetic on stored evidence. Live uploads use the connected backend; the development backend returns synthetic fixtures. Training protocols and evaluation commands are documented in the repository.

    Read the code Backend, agents, notebooks and automated checks