The brain_daemon PSU watchdog was silent excepts. PR #428. It wasn’t enough to fix the logic; we had to make the system report when it breaks. The cost dashboard no longer shows static 150W during a PSU outage because the exporter-fetch failure is now logged. We carried this philosophy through the rest of the #455 batch, hunting down the “silent except” traps across the entire stack.
In tap_runner and retention_runner, malformed JSONB rows in config and metadata stopped silently falling through to raw strings. PR #427. We did the same for social_poster metric increments and validator_config bootstrap imports, so Prometheus errors and DSN resolution failures surface in traces. PR #426.
The content pipeline stages dropped issues when schemas shifted, so the rewriter looped endlessly on the same draft. PR #425. jobs/check_memory_staleness and media reconciliation jobs stopped passing without a breadcrumb, and task_executor stopped hiding model selection failures and timeout errors. PR #423 and PR #418. The rails–deepeval, ragas_eval, self_consistency–now log why they’re disabled when is_enabled() fails, and RAGAS notifies us when it can’t find a judge model. PR #421 and PR #420.
Performance-wise, we stopped creating new httpx.AsyncClient instances for every URL check and revalidation burst. PR #424 and PR #419. The GPU scheduler and URL validator share one client, closing it cleanly on shutdown to avoid leaking TCP pools. PR #417.
We traded code volume for data density. The operator finally sees which jobs are dormant and why the cost dashboard is drifting. We still don’t love the amount of glue code needed to make this loud, but at least we’re not guessing anymore.
Auto-compiled by Poindexter from today’s commits and PRs.
Sources
- https://github.com/Glad-Labs/glad-labs-stack/pull/428
- https://github.com/Glad-Labs/glad-labs-stack/pull/427
- https://github.com/Glad-Labs/glad-labs-stack/pull/426
- https://github.com/Glad-Labs/glad-labs-stack/pull/425
- https://github.com/Glad-Labs/glad-labs-stack/pull/423
- https://github.com/Glad-Labs/glad-labs-stack/pull/418
- https://github.com/Glad-Labs/glad-labs-stack/pull/421
- https://github.com/Glad-Labs/glad-labs-stack/pull/420
- https://github.com/Glad-Labs/glad-labs-stack/pull/424
- https://github.com/Glad-Labs/glad-labs-stack/pull/419
- https://github.com/Glad-Labs/glad-labs-stack/pull/417
