BOLT-1650: investigate stale dashboard device status and offline→unknown regression
completedAgent: richie-engineer
Priority: 0
Branch:
winston/bolt-1650-dashboard-offline-statusPR: #14041
Linear: BOLT-1650
Richie requested investigation 2026-08-05. Diagnose Grace’s remaining BOLT-1650 report: offline devices render unknown rather than offline, likely dashboard/query/indexing/state mapping. Confirm scope works across orgs/workspaces; prioritize WEC/Washington Electric Cooperative as affected tenant. Trace source-of-truth update path (device state, Elastic docs, GraphQL/dashboard config), identify root cause, ship smallest safe cross-tenant fix with regression coverage.
Event Timeline
created
progress
Root cause isolated: the dashboard Devices list uses ES search, while detail reads the canonical device row. The ES hot-update path refreshed only reachable/online events; explicit offline/asleep events updated the DB but omitted onlineStatus from ES, leaving legacy or stale unknown status in list results. Implemented a scoped device-domain patch to write explicit non-online statuses without overwriting lastOnlineAt, plus regression coverage.
progress
Opened PR #14041: https://github.com/TextureHQ/mono/pull/14041. Cross-tenant fix forwards explicit offline/asleep status events to every DeviceOnWorkspace ES document while preserving lastOnlineAt semantics. Focused regression suite passes; CI watch active.
failed
lease expired — re-queued for retry
in_progress → queued
status_change
queued → in_progress
failed
lease expired — re-queued for retry
in_progress → queued
progress
Validated Codex P2 as real: delayed/retried ES jobs could overwrite a newer canonical reachability transition. Pushed 41bae6f0b5, which reads the persisted Device status/lastOnlineAt before updating ES; added regression coverage for stale offline→newer online ordering. Local lint, typecheck, and 4 focused tests pass. CI rerun started.
status_change
queued → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
progress
Validated the second Codex P2: DB snapshot before ES write still had a read→write race. Pushed 62399115e6: reachability now uses a version-guarded Elasticsearch scripted update keyed by canonical Device.updatedAt, so older jobs cannot overwrite a newer indexed transition. Local lint, typecheck, build, and focused tests pass; CI restarted.
status_change
failed → blocked
subagent_spawned
spawn claim: fix-ci-pr-14041
status_change
blocked → in_progress
status_change
in_progress → blocked
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Subagent verified ES dual-write behavior for PR #14041 reachability updates
progress
Pushed missing elasticsearch package files (22ba1f0ae4) to resolve build failure.
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → blocked
subagent_spawned
spawn claim: debug-ci-pr-14041
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Subagent failed, restarting it.
progress
Addressed PR #14041 review and pushed 6925188b47: reachability now stays in the single standard dual-write update; removed the scripted ES API/version marker; restored offline/absent-payload/no-direct-client tests. Local device lint, typecheck, build, and focused test pass; fresh CI/DeepSource runs are pending.
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Pushed PR #14041 review fixes; CI/DeepSource pending
status_change
failed → blocked
subagent_completed
subagent done: TASK_COMPLETE: Sub-agent fix_pr_14041_test_failure timed out, releasing lock
subagent_completed
subagent done: TASK_COMPLETE: ACP background task fix_pr_14041_test_failure done
subagent_completed
subagent done: TASK_COMPLETE: ACP background task fix_pr_14041_test_failure done
subagent_spawned
spawn claim: Address PR #14041 review feedback: Elasticsearch dual-write for reachability updates
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] failed due to lease expiry
subagent_spawned
spawn claim: fix-ci-pr-14041
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] failed due to lease expiry
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] fix-ci-pr-14041
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Subagent blocked on approval
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → blocked
subagent_spawned
spawn claim: fix-ci-pr-14041
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_spawned
spawn claim: fix-deepsource-pr-14041
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] [NO_TASK_COMPLETE_MARKER] failed: tool calls blocked by permissions
subagent_spawned
spawn claim: fix-deepsource-pr-14041-retry-permissions
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] [SUBSTRATE-ABORT] Sub-agent spawn failed due to syntax error in heartbeat script. No work started.
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_spawned
spawn claim: Fix DeepSource JavaScript and Test coverage failures in PR #14041
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Invalid taskName in sessions_spawn
subagent_spawned
spawn claim: Fix DeepSource JavaScript and Test coverage failures in PR #14041
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Invalid taskName in sessions_spawn
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] clearing wedged lock
subagent_spawned
spawn claim: Fix PR 14041 DeepSource
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Completed DeepSource fix attempt for PR #14041
subagent_spawned
spawn claim: Fix DeepSource for PR #14041
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] abandoned — failed to find codebase
progress
Legacy claude ACP sub-agent for BOLT-1650 finished with no meaningful output. I have already spawned a new native sub-agent to continue the investigation.
progress
Native sub-agent exited prematurely with no data. Respawning with explicit execution instructions.
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] [NO_TASK_COMPLETE_MARKER] empty output
subagent_spawned
spawn claim: bolt_1650_v2
progress
Sub-agent traced ES update path to Go device-detected-ingestor bullmq package. Investigating buildJobData mapping for dropped offline status.
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
progress
Root cause found in Go device-detected ingestor. Explicit offline events were updating the onlineStatus but neglecting to set lastOnlineAt, breaking the ES sync expectations. Fix authored and pushed as PR #14070.
status_change
failed → in_progress
progress
Monitoring CI for PR #14070.
status_change
in_progress → blocked
subagent_spawned
spawn claim: fix-pr-14070-coderabbit
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] failed: gateway restart during commit
subagent_spawned
spawn claim: fix-pr-14070-coderabbit-retry
subagent_completed
subagent done: TASK_COMPLETE: Fixed CodeRabbit feedback on PR #14070
progress
Subagent successfully pushed fix for CodeRabbit feedback on PR #14070. Commits were salvaged from the previous interrupted run. CI watch started.
status_change
in_progress → blocked
subagent_spawned
spawn claim: fix-ci-pr-14070
status_change
blocked → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
subagent_spawned
spawn claim: fix-ci-pr-14070
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] subagent failed due to lease expiry
subagent_spawned
spawn claim: fix-ci-pr-14070-deploy-failure
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] Fixed deploy / Deploy edges-site CI check for PR #14070.
failed
lease expired — max retries reached, marking failed (poison pill)
in_progress → failed
status_change
failed → in_progress
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] abandoned — wedged after 25min
progress
PR #14070 deploy / Deploy edges-site completed successfully at 26m12s. All checks are now passing.
status_change
in_progress → completed
status_change
completed → blocked
status_change
blocked → completed
subagent_release
abandoned — wedged after 30min but task now completed
subagent_release
subagent failed due to lease expiry
subagent_release
Fixed deploy / Deploy edges-site CI check for PR #14070.
subagent_release
failed: gateway restart during commit
subagent_release
spawn claim: bolt_1650_v2
subagent_release
empty output
subagent_release
abandoned — failed to find codebase
subagent_release
Subagent blocked by bash tool approval requirements
subagent_release
spawn claim: Fix DeepSource for PR #14041
subagent_release
Fix PR 14041 DeepSource
subagent_release
Invalid taskName in sessions_spawn
subagent_release
Fix DeepSource JavaScript and Test coverage failures in PR #14041
subagent_release
failed: tool calls blocked by permissions
subagent_release
Fixed the failing `deploy / Deploy edges-site` CI check for PR #14070.
subagent_release
TASK_COMPLETE: Fixed CodeRabbit feedback on PR #14070
subagent_completed
subagent done: [NO_TASK_COMPLETE_MARKER] abandoned - incorrect execution