What we shipped on 2026-07-08
Today’s most humbling shipment was a fix for a fix. brain/clock_skew_probe.py landed in (PR #2212) to catch something real: poindexter-postgres-local had returned now() roughly four hours in the future during a window, self-resolving without a restart. Root cause was a WSL2 CLOCK_REALTIME excursion – the VM clock jumping on host sleep/resume, Postgres reading that jump straight off gettimeofday() into every DB-written timestamp. Grafana time-windows missed future-stamped rows. A real-time correlation broke the day it happened, silently, because the stack had no detector for a wrong DB wall-clock.
So we built one – an external-absolute reference (an HTTP Date: header) compared against clock_timestamp(), deliberately avoiding the trap of comparing the probe’s own utcnow() to Postgres, since both live in the same VM and would drift together. Good design. Then we found out it never ran.
The brain Dockerfile COPYs its probe modules by an explicit filename list, and clock_skew_probe.py wasn’t on it – not in the flat COPY ... ./ line, not in the /app/brain/ package-mirror cp block. Both import paths failed in-container, and the existing try/except ImportError → _HAS_CLOCK_SKEW_PROBE = False guard turned that into a quiet skip. clock_skew_samples stayed empty. The probe “deployed” but was dead-inert – confirmed by finding clock_skew_probe.py living only at the bind-mounted /host-deploy/brain/... path, never at the baked /app/... paths the imports actually used. (PR #2215) added it to both COPY blocks. The fix was two lines in a Dockerfile; the bug was a whole detector shipping to do nothing.
There was a second flavor of the same disease today: things silently poisoning other things. test_image_gen_self_heal.py installs a bare types.ModuleType("torch") stub – only .float16/.cuda, no .Tensor – into sys.modules at collection time, and never restores it. When that file gets collected before any real-torch importer, the husk survives and later poisons test_ragas_eval.py::TestDispatcherWrappers with AttributeError: module 'torch' has no attribute 'Tensor', via ragas → transformers → safetensors evaluating torch.Tensor in an annotation at def-time. This is the second distinct polluter of this shape – (PR #2209) fixed the first, a patch.dict(sys.modules) restore that dropped a lazily-imported torch. (PR #2214) scopes the stub so it can’t leak past its own test. And because this class of bug has now bitten twice, (PR #2216) adds a trylast pytest_runtest_teardown hook to the unit-tier conftest that turns a leaked module husk into a deterministic, self-fingerprinting failure – the offending test fails itself, by name, instead of quietly poisoning some innocent test under a specific random seed.
Underneath all that, a genuinely big release went out – 0.98.0 (PR #2219) – carrying the console Task Trace across three phases: incremental per-node atom_runs capture so a killed run still leaves rows for every finished node (PR #2208), then the read API and UI on top – services/trace_read.py’s runs_for_request() treating a request as one-to-N runs so retries and composed media DAGs need no redesign, plus the per-task deep-dive board (PR #2213). Citation reconciliation also closed out its two-part arc: the deterministic linker for multi-word brands and unusual attribution frames (PR #2207), followed by the grounded-LLM tail for the residual cases the regex grammar can’t safely frame-match – the “Full Brim Safety” case that started it all, a real corpus source named in prose but never linked, indistinguishable from a hallucination until now (PR #2211). The new atom only applies a link when the URL is verbatim from the research corpus and the text occurs verbatim outside an existing markdown link – LLM proposes, deterministic code verifies and applies.
Three separate silent-failure shapes, three separate fixes, in one day: a probe that deployed but didn’t run, a test stub that leaked but didn’t announce itself, a citation that was true but looked fake. The common thread is worth sitting with – the stack is getting good at looking done. The conftest hook is the one we’re proudest of, because it converts a whole category of future flakiness into an immediate, named failure instead of another 4am mystery.
Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.
