BOLT-1853: investigate Sensi telemetry delivery collapse
blockedAgent: richie-engineer
Priority: 0
Branch:
winston/bolt-1853-sensi-expired-cache-backfillPR: #14357
Linear: BOLT-1853
Richie requested a live production investigation 2026-08-13. Customer report: all 19 Sensi devices fell from ~4.8 fresh samples/device/hour to 0.2–0.4/h since 00:00 UTC despite is_online=true and rows arriving in SensiWebhookIncoming. Trace Sensi ingress → processor → Kafka/device.detected or backfill → Device domain telemetry. Establish root cause, recovery ETA, and prevention; attach evidence and Linear BOLT-1853.
Event Timeline
created
progress
Confirmed root cause and ruled out delivery lag: mapped Sensi webhooks still arrive within seconds of Sensi event_timestamp, and all ingress records process successfully. At the 00:00 UTC daily cache poll, processor snapshots the then-current OAuth access-token expiry. It publishes device.detected only before that expiry; after expiry it marks incoming mapped webhooks processed but skips all Device-domain writes. The exact per-device stop times align with cached expiry. Preparing minimal safe hotfix to backfill state after cached-connection expiry without asserting reachability.
subagent_spawned
spawn claim: Preparing hotfix for Sensi telemetry
progress
PR #14357 opened for expired Sensi cache telemetry hotfix. Local focused tests (11), ESLint, and Prettier passed; CI running.
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] PR #14357 opened; focused tests, ESLint, and Prettier pass; CI running.
failed
lease expired — re-queued for retry
in_progress → queued
status_change
queued → in_progress
status_change
in_progress → blocked
progress
Verified OEM source of truth: all Sensi configs enabled and daily poll ran at 00:00 UTC. Sensi access-token JWTs are exactly 24h; Connect created active successor tokens throughout the day, but OEM cache still holds the pre-refresh token expiries, so 21/27 cached entries became stale/blocked. Investigating refresh scheduler buffer values and correct durable propagation path.
status_change
blocked → in_progress
failed
lease expired — re-queued for retry
in_progress → queued
status_change
queued → blocked
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → blocked
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → blocked