What we shipped on 2026-07-10
The tell was buried in a GlitchTip stack trace: Timeout passed=90.0, time taken=0.001 seconds. A timeout that fires in a millisecond isn’t a timeout – (PR #2251) chased that lie back to litellm’s own aiohttp transport, which pools keep-alive connections to Ollama. In the sparse nightly window (03:00-05:00), Ollama closes idle sockets server-side; aiohttp reuses a pooled-but-dead one, the first read fails instantly, and litellm’s exception_type relabels the failure litellm.Timeout → APIConnectionError. GlitchTip issue 736 paged us 13 times over five days before we found the real fault line. The fix was a transport default – httpx instead of aiohttp – not a longer timeout.
That was the theme of the day: things that looked like one bug and were actually two, or a symptom standing in for the real cause. GlitchTip #863 – WARN finding lost on audit write – fired 92 times and turned out to have two unrelated root causes both making the audit_log INSERT raise. The louder one was a pool-close race: the Prefect content flow builds and tears down its own DB pool per run, and when spend-throttle fired mid-flight, close() beat the fire-and-forget audit write to the punch. Throttle state resets every fresh subprocess, so the same ~2-minute run kept re-losing the same finding – hence the count. audit_log._pending_writes plus a drain_pending_writes() call before DatabaseService.close() (PR #2248) closes that race without touching the loud-drop escalation logic that surfaced it in the first place.
The social drafts atom had a quieter version of the same class of bug: state.get("pool") was reading a key that never got seeded. The canonical_blog graph path only ever sets state["database_service"], and the sibling atoms (media_persist, podcast_persist) already knew that – this one was the odd one out, silently swallowing its own exception and quietly generating zero social drafts across fourteen posts before GlitchTip issue 855 caught it (PR #2245). The fix is the idiom the rest of the codebase already uses: getattr(database_service, "pool", None).
Image generation had its own wedged-but-healthy failure: every post from 07-08 16:09 onward shipped a Pexels stock photo instead of a generated one, both featured and inline, and it took walking posts.metadata->>'featured_image_data' timestamps to prove the last real local generation was hours earlier than the first fallback (PR #2244). Meanwhile the analytics side got its own forensic pass – beacon raw traffic was 1,708 against GA’s 178 over the same window, with one Chrome UA on Linux accounting for 1,532 hits, 90% of the noise. page_views.is_bot plus a page_views_human view (PR #2246) separates “is the pipe flowing” (raw, for liveness alerts) from “how many readers” (human, for everything reader-facing) – a distinction we’d been fudging for a while.
Not everything today was forensic. We retired the legacy :9837 host-slideshow lane entirely – dead weight since the shot-list renderer took over months ago, never selected by any live job, kept alive only by the recurring temptation to “fix” its host.docker.internal URL (PR #2254). And the Hardware & Power panel now meters true wall draw off a Shelly smart plug instead of fighting iCUE for the HX1500i’s USB bus (PR #2249) – get_shelly_psu_metrics() slots into the existing psu_power.py::select_power_source preference with no downstream changes, inert until the operator sets shelly_psu_url.
All of it landed in 0.99.0 (PR #2257) alongside the Trace-tab console rework and the media-QA Layer 2 semantic scoring. The through-line across today’s fixes wasn’t a single system – it was pool lifecycles and stale connections quietly lying about their own failure modes. Worth remembering next time a timeout looks too clean.
Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.
