Loop Room

User Manual ← Back to the dashboard

Loop Room — User Manual

The AI operations control plane: what it watches, what it fixes on its own, and what it queues for you. Service on port 8090; this page is served at /manual.

1.What is the Loop Room?

The Loop Room is the AI operations brain — a standalone service that watches every automated loop, agent, and wiki in the estate and answers four questions the moment you open it: what ran overnight, what it changed, what broke, and what is waiting on you. It monitors the Hermes crons, the six wikis, the trial ledger, the QMD retrieval index, Wingman's task pipeline, and the knowledge writeback flywheel; it measures everything against baselines; it raises flags when something drifts; and it auto-fixes the mechanical class of problems itself, queuing only the judgement class for a human. Health is always computed, never asserted — a loop that reports "ok" over an empty output directory shows red.

2.Architecture overview

Primary truth stays in the estate's own files — registry.yaml, the flag-queue JSONL, the trial ledger markdown, the wikis themselves. The Loop Room's SQLite database (data/looproom.db) holds only derived state: it is safe to delete, and the collectors rebuild it from sources.

The data plane

registry.yaml ─────────── declared truth: loops, emits→consumed_by contracts,
     │                    wikis, baseline metrics
     ▼
Hermes crons ──▶ collectors (read-only) ──▶ SQLite (derived) ──▶ Dashboard SPA
estate files      cron · artefact · flag        looproom.db        Board · Loops ·
portal.db         log · trial · signal ·           │               Trials · Flags ·
qmd status        wingman · qmd · race ·           └──▶ SSE ──▶    Wikis · Map ·
workspaces        writeback · wiki                 /api/events     Decisions

The flag lifecycle

collector detects drift
     │  append_flag() — one live flag per (stream, target)
     ▼
flag queue ── ~/ai-ops-wiki/operations/flag-queue/*.jsonl
     │        append-only, one file per stream, last line wins
     ▼
dispatcher (nightly 03:00, or manual run) — POLICY table routes every open flag
     │
     ├─▶ auto-fix ──▶ mutate file ─▶ git commit "loop-room:" ─▶ post-condition
     │                                                          ├ pass → auto_fixed
     │                                                          └ fail → needs_adam + card
     ├─▶ queue    ──▶ decision card (Decisions view) or held backlog
     ├─▶ suppress ──▶ logged reason, no action
     └─▶ no policy ─▶ "unroutable stream" decision card

Running it

In production the service runs under systemd via loop-room.service (Restart=always, 10-second backoff): install with systemctl enable --now loop-room, follow logs with journalctl -u loop-room -f. The database lives at data/looproom.db (WAL mode) and is safe to delete — collectors rebuild it from sources.

The steward tick loop

The steward is a 60-second background loop. On each tick it:

3.The Board

The Board is the 7-am view: everything from one GET /api/state call, arranged so the whole estate reads in under 30 seconds.

Tiles

TileValueTurns amber/red when…
Loopsgreen ✓ / amber △ / red ✗ counts red if any loop is red; amber if any is amber.
Trialsactive trial count, next review date red when a trial review is OVERDUE.
Flagsopen flags, median half-life, streams measured amber when more than 50 flags are open. means the stream has never resolved a flag.
Wikisworst link coverage across the wikis amber when the worst wiki's coverage is below 30%.
RetrievalQMD chunks pending embedding, KB injection hit rate amber when the pending backlog exceeds 50,000 chunks.
StackWingman tasks in the last 7 days informational only.
Writebackgenerative pages/day, answer-file rate red when a writeback-stalled decision is open (§11).
Racesbest harness race score, deployment status amber when a race winner has not been deployed back to Wingman.

Colour semantics

Sparklines plot the last 100 samples of each baseline metric. The blue line is the measured value, the blue dot is the latest sample, and the dashed yellow line is the value recorded at the 2026-07-05 review — the reference everything is measured against. Each card states which direction is good (↑ good / ↓ good).

4.Loops view

Every loop declared in registry.yaml, worst health first. The R/P/C dots are the health model: RAN (did the scheduler fire on time, with status ok?), PRODUCED (does the declared artefact exist, fresh within its max_age_hours?), CONSUMED (did emitted streams land in their consumers?). Green means the check passed, red means it failed, grey means the loop declares no contract for that check. Click a row for the loop's detail page: schedule, run history, and its full consumption contract from the registry.

Green vs red: green requires RAN and PRODUCED to pass. Red is one of three deterministic failure states — failed (the run errored), missing (the scheduled slot plus a 30-minute grace period passed with no run evidence), or silent-failed (ran ok but the artefact is absent or older than its contract allows).

LoopScheduleWhat it doesArtefact contract
dream-cycle01:00 daily Nightly wiki-healing pass over the personal wiki: citation audit, consolidation candidates, stale-page detection. Emits citation-flags, consolidation and stale-pages into the flag queue, and backlink-fixes to code. agent-logs/dream-cycle/*.md ≤ 26 h
email-wiki-sync01:30 daily Email → wiki delta sync.none declared
wiki-freshness02:00 daily AI Ops Wiki freshness check. agent-logs/wiki-freshness/*.md ≤ 26 h
nas-health03:00 daily NAS health report. agent-logs/nas-health/*.md ≤ 26 h
fleet-health-check04:00 daily Agent fleet health check. agent-logs/health-checks/*.md ≤ 26 h
nightly-synthesis04:00 daily Nightly synthesis brief — cross-estate knowledge synthesis; emits synthesis-actions to the decision queue (§11). synthesis/*.md ≤ 26 h
morning-briefing06:00 daily Morning workboard briefing, delivered to Matrix. none — no filesystem artefact
ai-lab-daily-digest07:00 daily Research / skills-library digest pages plus email. none declared yet
wiki-staleness-audit06:00 Mondays Weekly wiki staleness audit. agent-logs/wiki-staleness/*.md ≤ 192 h
signal-capture05:00 Sundays Learning loop — weekly signal capture; emits weekly-signals to the trial verdict engine. learning-loop/signals/*.json ≤ 192 h
training-data-extraction02:00 Sundays Weekly training-pair extraction; emits training-pairs to the training gate.none declared
identity-extraction04:00 Sundays Weekly identity extraction.none declared
trial-verdict-engine07:00 Sundays Measures active trials against captured signals and writes verdict files; emits trial-verdicts to the decision queue. learning-loop/trials/*-verdicts.md ≤ 192 h
weekly-ops-review12:00 Sundays Weekly operations review. agent-logs/weekly-review/*.md ≤ 192 h
monthly-analysis06:00 first Sunday of the month Learning loop — monthly analysis. learning-loop/analysis/*.md ≤ 800 h
monthly-compaction02:00 on the 1st Monthly operations compaction. agent-logs/_monthly/*.md ≤ 800 h
monthly-intelligence01:00 on the 1st Monthly intelligence pipeline.none declared
loop-room03:00 daily (dispatch slot) The engine itself, registered under its own contract: its collectors emit backlink-fixes and qmd-context-drift for its dispatcher to consume, and writeback-failures / writeback-stalled for the decision queue.n/a — engine

Onboarding a new loop is one stanza in registry.yaml: an id, a schedule, an optional artefact glob with a freshness bound, and optional emits → consumed_by contracts. The Map view draws itself from those contracts.

5.Collectors

Collectors are strictly read-only against the systems they observe (the one carve-out: the writer adapters that append flags to the Loop Room's own queue, and the qmd collector's daily index refresh, which maintains a resource the Loop Room owns). Each runs on its own cadence inside the steward tick; any of them can be run immediately with POST /api/collect/{name}.

CollectorCadenceWhat it measuresFlags it can raise
cron5 min Reads ~/.hermes/cron/jobs.json and the cron output directory — RAN evidence for every Hermes job, historical backfill on first boot, and a sweep for jobs that exist but are not registered.
artefact5 min Evaluates every registry artefact glob (the PRODUCED half of health) and recomputes health for all loops; worst glob wins. — (drives red/silent-failed states)
flag5 min Mirrors the flag-queue JSONL into the flags table, computes per-stream half-life (flag_half_life_hours), and sweeps for streams no registry entry declares.
log15 min Adapter: parses dream-cycle logs and converts the ranked findings into queue flags. A cursor keeps each log parsed once. citation-flags, consolidation, stale-pages
trial15 min Indexes the trial ledger and verdict files; detects OVERDUE reviews (P9). — (overdue trials surface via decisions)
signal15 min Learning-loop signal captures — per-week totals and per-pattern correction counts, the measurement substrate for trial verdicts.
wingman15 min Wingman read-only: task throughput from portal.db and the kb_context_injected hit rate from workspace event logs.
qmdhourly Runs qmd update (at most once per 24 h — the daily BM25 re-index the Loop Room owns) then parses qmd status: embedding backlog (qmd_pending_chunks) and per-collection freshness. Detects collection context descriptions whose declared page count drifts more than 10% from the live index. qmd-context-drift
racehourly Indexes harness-bench race results (per-candidate scores from comparison.json, single runs from summary.json); tracks winners against the deployed Wingman harness config (race_winners_deployed). race-resultsrace_completed, winner_undeployed, and score_regression between consecutive races of a family
writebackhourly The knowledge-compounding collector: writeback velocity and totals across the wikis, answer-filing and research-logging rates, user save events, generative growth, and stall/push checks against Wingman's writeback_log (§11). writeback-stalled, writeback-failures
wiki6 h Scans all six wikis in a worker thread — pages, [[link]] coverage, orphan detection, top orphans, and the estate-wide count of namespaced cross-wiki [[ns:path]] links. backlink-fixes — for wikis with emit_backlink_flags (ai-ops-wiki), a ranked, budgeted 10 flags per scan
Fail-soft rule: a crashing collector logs its error to the Glass Box event stream and the tick continues. Collectors must never take the steward down.

6.Flags & dispatch

The flag lifecycle

Flags live in append-only JSONL stream files under ~/ai-ops-wiki/operations/flag-queue/ — one file per stream, primary truth on disk, DB rows only a derived index. A flag's current state is the last line carrying its id; transitions are appended, never edited in place. A flag is raised open and terminates in exactly one of four states:

Deduplication keeps the queue honest: nightly loops re-flag the same pages, but the queue holds one live flag per (stream, target), not one per night. Half-life — the median hours from raised to resolved, per stream — is the headline compounding metric. means a stream has never resolved a flag; the Flags view shows the all-time and 7-day medians with an improving/worsening trend.

POLICY routing

The dispatcher routes every open flag somewhere — nothing terminates in a report. The policy table:

StreamActionHandler / destinationNightly budget
stale-pagesauto stale_frontmatter_fix20
backlink-fixesauto backlink_insert30
qmd-context-driftauto qmd_context_refresh6
citation-flagsqueue held in the flag-queue backlog until the citation-repair budget lands
consolidationqueue decision card — consolidation needs judgement
writeback-failuresqueue decision card — a failing writeback path needs a human; auto-retry would mask bugs
writeback-stalledqueue decision card — same reasoning
any undeclared streamescalate an "unroutable stream" decision card, so new streams never pool silently

The nightly 03:00 window

The steward fires the dispatcher once per local day, at the first tick after 03:00 — after the dream cycle (01:00) and freshness check (02:00) have raised the night's flags. A paused steward does not burn the day's slot; dispatch resumes the day you resume the steward. You can also trigger it from the Flags view: Run dispatcher (gated, real mutations) or Dry run, which reports exactly what each flag would get — no file writes, no transitions, no cards.

Guardrails

Everything the dispatcher does lands in the action audit trail at the bottom of the Flags view: kind, target, policy (auto vs gated), state, post-condition result, commit hash, and a one-click rollback (git revert — the revert is itself a commit, nothing is lost; the action is then marked rolled_back).

7.Auto-fixes

The auto class is deliberately narrow: small, reversible, verified mechanical fixes only. Each handler is paired with a post-condition, every change is a git commit, and everything can be rolled back from the audit trail.

HandlerConsumesWhat it doesPost-condition
stale_frontmatter_fixstale-pages Repairs frontmatter date fields (updated, last_reviewed, date, …) that have drifted behind the file's real modification time. Purely mechanical — genuinely stale content is not auto-fixable and reports "no change", leaving the flag queued. A re-scan finds no remaining drifted date fields.
backlink_insertbacklink-fixes Adds missing [[backlinks]] to the target page: appends to an existing ## Backlinks section, else creates one before ## Timeline / ## Sources, else at the end of the page. Every source link now resolves in the target page.
qmd_context_refreshqmd-context-drift Rewrites a QMD collection's declared page count in ~/.config/qmd/index.yml with the live indexed count — a raw-text edit scoped to that collection's block, so formatting and the rest of the description survive untouched. The declared count is within 10% of the live scan.

Outcomes: fixed (change committed, post-condition passed, flag → auto_fixed); no change (nothing mechanical to do, flag → queued with the reason); failed / post-condition failed (flag → needs_adam plus a decision card — the committed change is left in place for you to review, revert, or finish by hand).

An hourly re-verification pass re-checks each passed post-condition exactly once on a later tick. A regression raises an "auto-fix regressed" card and returns the flag to needs_adam — flagged, not retried.

8.Decision cards

Decision cards are the judgement class — everything the system will not do without you. They appear in the Needs You zone on the Board and in the Decisions view, deduplicated by key so nightly re-sweeps never stack duplicates.

Acting on a card

Approving a card executes its writeback — approval is not an acknowledgement, it is the mutation itself. The result (commit hashes, new trial ids, errors) is stored on the card and shown in the resolved ledger, so the audit trail shows exactly what each approval did. Defer keeps the card for later; Reject / Dismiss closes it without acting. A failed or blocked writeback leaves the card open — failures are visible, never swallowed by an "approved" state.

Card kindRaised when…Approval executes
flag_judgement A queue-class flag (consolidation, writeback failures/stalls) reaches the dispatcher. State change only — the card is the routing decision.
loop_failure A loop goes RED (one card per loop per day). Your choice of two writebacks: Re-run now (triggers hermes cron run for cron loops, or the loop's declared re-run command) or Investigate (raises a high-severity loop-investigations flag).
auto_fix The mutation budget ran out before a flag's fix could run (a budget-held fix). The same auto-fix handler, gated, outside the budget.
trial_verdict The verdict engine measures a trial. Writes the verdict into the trial ledger, appends to the evolution log in trial-principles.json, and on PROMOTE moves the rule into the fleet SOUL standard.
soul_promotion A rule is put up for fleet-wide deployment. P6-checks the promotion against every active trial's 14-day attribution window before touching any file; a conflict blocks it outright and raises a p6_blocked card. A clean promotion patches the SOUL files and registers itself as a new trial, so the promotion is itself measured.
action_failed An auto-fix post-condition failed or regressed on re-verification. Acknowledgement — the card points you at the commit to review.
p6_blocked, flag_unroutable, synthesis-actions Attribution conflicts, streams with no policy entry, and actions proposed by the nightly synthesis brief. Acknowledgement / follow-up.

The needs_adam state

needs_adam is the flag state for issues only a human can close: a crashed handler, a failed post-condition, a regression caught by re-verification. These flags sit in their own red-badged group at the top of the Flags view and always arrive with a matching decision card explaining what happened and what to inspect.

Approving a decision whose writeback would mutate the estate is refused while the kill switch is on — resume the steward first.

9.Kill switch

The kill switch is the toggle at the bottom of the navigation rail (steward active / MUTATIONS PAUSED), also available as POST /api/steward/pause and /resume. It is the single lever that stops the Loop Room changing anything, estate-wide, immediately.

When to use it: whenever an auto-fix misbehaves, a mutation looks wrong and you want to investigate before more land, or you are doing surgery on the wikis and don't want the dispatcher writing underneath you.

While paused:

10.Baselines & sparklines

Baselines are the review's ledger, made continuous. Each metric was recorded once at the 2026-07-05 strategic review; the collectors now sample them on every pass, and the Board draws each one as a sparkline against its dashed yellow review line. The point of the panel is a single question: is the system compounding, or was the review value a high-water mark?

MetricUnitReview valueGoodWhat it measures
aiops_link_coverage%10.0 ai-ops-wiki files containing at least one [[link]] — the O7 backlink floor. Rising means the backlink auto-fixes are landing.
personal_wiki_uncitedpages7,314 Personal-wiki pages with no incoming links (orphans). The dream cycle and backlink fixes should grind this down.
qmd_pending_chunkschunks57,270 QMD chunks awaiting embedding. A growing backlog means retrieval is falling behind the estate.
flag_half_life_hourshours Median hours from flag raised to resolved. It was infinite at review — four of six streams had never resolved a flag. Any finite, falling number is the acting system working.
training_pair_outflowpairs/week0 Training pairs flowing to actual training runs. Zero means extraction produces inventory nothing consumes.
race_winners_deployedcount0 Harness race winners deployed back to Wingman. A winner that never deploys is a loop whose output is consumed by hope.
kb_injection_hit_rate%unmeasured Share of Wingman tasks receiving kb_context_injected — is retrieval actually reaching the agents?
writeback_velocity_24hpages0 Wiki pages created by Wingman writeback across all wikis in the last 24 h.
writeback_totalpages0 Total Wingman-authored pages across all wikis — the accumulated knowledge base.
answer_file_rate%0 Share of recent tasks that auto-filed answers to a wiki.
research_log_rate%0 Share of recent tasks that auto-logged research findings.
knowledge_save_count_7dcount0 User-triggered "save this" events in the last 7 days.
wiki_growth_generativepages/day0 Agent-generated wiki pages per day, averaged since the first writeback landed.
synthesis_brief_presencedaysunmeasured Days since the last nightly synthesis brief. Should hover at zero once the loop is healthy.
cross_wiki_backlink_countlinks Namespaced [[ns:path]] links across the whole estate — knowledge in one wiki citing another. Sampled by the wiki collector; the metric row is created on first sample.

What triggers concern: a "down is good" metric trending up past its review line, an "up is good" metric flat at zero weeks after the loop that feeds it went green, or a sparkline that simply stops — the collector feeding it has gone quiet, which is itself a loop-health problem.

11.Knowledge compounding

The writeback flywheel is the estate's answer to "do the agents make the knowledge base better, or just consume it?" Wingman's writeback system files task knowledge into the wikis as git commits prefixed wingman:, landing only under three generative directories — insights/, research-logs/ and knowledge-captures/ — and emits Glass Box events (knowledge_saved, knowledge_filed, research_logged) into each workspace's events.jsonl. The Loop Room's hourly writeback collector observes all of it, read-only, and turns it into the compounding baselines and two flag streams.

What is tracked

Stall and failure flags

Two conditions are checked against Wingman's writeback_log table on every collector pass. Both raise high-severity flags that the policy table routes straight to decision cards — auto-retrying a broken writeback path would only mask bugs.

The Board's Writeback tile goes red while a stalled-writeback decision is open.

The nightly synthesis brief

The nightly-synthesis loop (04:00 daily) reads what the estate learned that day and writes a brief to ~/ai-ops-wiki/synthesis/, emitting synthesis-actions into the decision queue for anything that warrants follow-up. It is monitored like any other loop: an artefact contract of ≤ 26 hours, and the synthesis_brief_presence baseline (days since the last brief) on the Board.

12.API reference

All endpoints are under /api and return {"ok": true, …} on success or {"ok": false, "error": …} on failure.

EndpointDescription
GET/api/state Everything the Board needs in one call: loops, open decisions, trials, wikis, baselines with samples, races, digest, collector stats, flag counts, half-life, steward state.
GET/api/loops All loops, worst health first; filter with ?health=red.
GET/api/loops/{id} One loop: health detail, last 30 runs, and its registry contract.
GET/api/flags Flags with per-stream half-life; filters: ?stream=, ?state=, ?severity=, ?target= (substring), ?limit=.
POST/api/flags/{id}/suppress Suppress a flag; body {"reason": "…"} — a reason is mandatory and logged.
GET/api/actions The action audit trail — every mutation the dispatcher made (?limit=).
POST/api/actions/{id}/rollback git revert the commit(s) an action made and mark it rolled back. Allowed even while paused.
POST/api/dispatcher/run Run the dispatcher now (gated; refused while paused). ?dry_run=true returns the per-flag plan without executing anything.
POST/api/steward/pause The kill switch on — all mutations stop; collectors keep observing.
POST/api/steward/resume The kill switch off — mutations may run again.
GET/api/trials The trial ledger, all verdicts, and weekly per-pattern signal counts.
GET/api/decisions Decision cards; filters ?state=, ?kind=.
POST/api/decisions/{id}/resolve Resolve a card; body {"resolution": "approved"|"rejected"|"deferred", "writeback": …, "note": …}. Approval executes the writeback; a failed or P6-blocked writeback leaves the card open.
GET/api/baselines Every baseline metric with up to 200 samples each.
GET/api/wikis Per-wiki scan results (pages, coverage, orphans, QMD freshness).
GET/api/registry The parsed registry.yaml — the declared truth the Map is drawn from.
POST/api/collect/{name} Run one collector immediately (cron, artefact, flag, log, trial, signal, wingman, qmd, race, writeback, wiki).
GET/api/events/recent The most recent Glass Box events (?limit=, max 500).
GET/api/events Server-sent events stream — one connection, multiplexed by event kind; drives the live dashboard and the Map pulses.
GET/ · /manual The dashboard SPA and this manual.

13.Wikis view

One card per registered wiki, refreshed by the 6-hourly wiki scan and the hourly QMD pass:

WikiPathQMD collectionNotes
personal-wiki~/personal-wiki personal-wiki ~7,500 pages; the dream cycle's healing target; orphan count is the personal_wiki_uncited baseline.
ai-ops-wiki~/ai-ops-wiki ai-ops-wiki Emits budgeted backlink-fixes flags (the O7 backlink floor: 10% coverage at review); also hosts the flag queue and synthesis briefs.
ai-lab~/ai-labnone Design docs and harness-bench; scanned for coverage only.
tcd-wiki~/tcd-knowledge/wiki tcd-wikiTCD knowledge wiki.
hydrastorm-wiki~/hydrastorm-wiki hydrastorm-wikiHydrastorm project wiki.
tcd-reference-library~/tcd-knowledge/reference-library tcd-reference-libraryTCD reference library.

The same wikis appear on the Map as the right-hand column of the estate's dependency graph — blue nodes fed by the crons and the engine, with hover tooltips showing the scan numbers and a click-through back to this view.