What we shipped on 2026-07-07
Content generation went dark for roughly two hours before anyone noticed – 02:31 to 04:57, per the audit log – and the worker looked completely healthy the whole time: ONLINE, heartbeating, executing nothing. fix(brain): reclaim CANCELLING zombies wedging the content-pool slot (PR #2175) is the story of why. The content-generation deployment runs on content-pool at concurrency_limit=1, so one wedged flow run holds the only slot for everyone. A host/WSL event left two runs – qualified-corgi, inscrutable-gharial – stuck in CANCELLING: a graceful cancel of a process that’s already dead, with nobody left to confirm the kill. Nothing ever promotes them to CANCELLED. Meanwhile ~73 runs piled up SCHEDULED behind the wedge.
The frustrating part: we already had a probe for exactly this failure mode. prefect_stuck_flow_probe auto-CRASHED a genuine RUNNING zombie (masked-griffin) at 21 minutes, exactly as designed. It just never looked at CANCELLING – WATCHED_STATE_TYPES = ["RUNNING", "PENDING"] had a blind spot for the one state that actually caused the incident. qualified-corgi sat wedged 62 minutes, inscrutable-gharial 28, both eventually freed by an operator force-cancel instead of the automation doing its job.
Once that fix landed, we killed a half-finished idea rather than ship it out of inertia. We’d added PREFECT_FLOWS_HEARTBEAT_FREQUENCY=30 as groundwork for a Prefect-native zombie-crash Automation – but that automation only ever catches RUNNING zombies, already covered by the probe, and does nothing for the CANCELLING wedge that actually bit us. chore(prefect): drop inert PREFECT_FLOWS_HEARTBEAT_FREQUENCY groundwork (PR #2178) rips the dead env var back out and reworks docs/operations/self-healing.md so the decision doesn’t get re-proposed next time someone reads the ticket.
The other thread running through today was the Sonnet cutover, and it surfaced two latent bugs that had been quietly lying about money. First: every paid cloud call was logging cost_usd=0. hasattr(response, "response_cost") is always False – litellm actually stores the computed price on response._hidden_params["response_cost"], not as a top-level attribute. Harmless while everything ran local, catastrophic the moment real paid calls started: the first canary draft (8k in / 3.9k out) logged $0 for a call that actually cost about $0.055. Since cost_guard’s cap sums cost_logs.cost_usd, a table of zeros can never trip the daily or monthly limit – a spend-safety gate that silently couldn’t fire. fix(llm): capture paid-call cost from litellm _hidden_params (PR #2183) rewrites _extract_response_cost to read the hidden-params value with a litellm.completion_cost() fallback, verified in-container against a real 6.8e-05 probe, with a new six-test regression suite and the full 245-test provider suite green.
Second bug, same cutover: the writer was still forwarding Ollama-only params – num_ctx and the think=False toggle from #2163 – into cloud calls unconditionally. litellm’s drop_params only strips OpenAI-spec params the target rejects, and num_ctx/think aren’t OpenAI-spec, so they sailed straight through to Anthropic, which 400’d with num_ctx: Extra inputs are not permitted. fix(llm): drop Ollama-only params (num_ctx/think) from cloud calls (PR #2177) adds _OLLAMA_ONLY_PARAMS and _is_local_prefix() to strip both for any non-local target while leaving local Ollama calls untouched. The verification here was satisfying: once fixed, the failing call turned into a genuine Anthropic error with a real request_id – confirming the earlier api_base scoping fix (#2167) actually reached api.anthropic.com and wasn’t hitting a local shim. That made it the last blocker before Sonnet drafts could complete end to end.
With the cost ledger finally trustworthy, the rest of the cost-attribution design (§5-6) could land on solid ground. fix(cost): route cost_guard onto the cost_ledger seam (PR #2173) deletes cost_guard’s hand-rolled SUM(cost_usd) query – the cap now reads spend exclusively through cost_ledger.get_spend, strict=True, failing closed rather than silently zeroing out on a DB hiccup. And feat(cost): spend throttle defers new work at a soft budget (PR #2184) adds the softer sibling: a fail-open spend_throttle sitting at the Prefect claim seam, deferring new work over a soft ceiling while letting in-flight generations finish untouched. Operator-triggered runs bypass it entirely. Between the hard cap and the soft throttle, “slow down” is now a real, safe operating mode instead of a manual intervention.
One more fix worth naming even though it’s a quieter one: fix(console): narrow contract snapshot to 2xx responses (PR #2174) stopped drift-detection from firing on console-irrelevant backend changes. A backend error-envelope tweak had pulled unrelated FastAPI error models into the snapshot’s component closure and opened a refresh PR nobody needed to triage – the new pruneOpenApiSnapshot keeps only 2xx responses and their transitive component closure, which is all the console actually consumes. Small fix, but it’s the difference between drift alerts meaning something and becoming noise the operator learns to ignore.
All of it shipped in release 0.97.0 (PR #2181). The through-line for today wasn’t one big feature – it was closing gaps that only show up once real money and real cloud calls are in the loop: a probe blind to one Prefect state, a cost ledger that could silently zero out, params that only break on the provider you haven’t tested against yet. None of these were visible until the Sonnet cutover actually ran live traffic through them. That’s the tradeoff of moving off local-only: the bugs get more expensive, but they also finally show themselves.
Auto-compiled by Poindexter from today’s commits and PRs. See the work: github.com/Glad-Labs/poindexter.
