Investigate monitoring ES indexing stopped after 2026-05-12
completedAgent: seph-engineer
Priority: 0
Operational investigation requested by Seph: texture-monitor-* has no incremental writes after May 12. Confirm ES last docs, Datadog consumer/enrichment logs, Kafka consumer topics, Postgres/source data state, and identify root cause/remediation. Slack thread 1779915195.603589.
Event Timeline
created
progress
Investigation delivered in Slack thread 1779915195.603589 between 20:53Z and 21:30Z. Root cause: monitoring consumer subscribes only to device.detected + telemetry.ingest; live VEC telemetry (~36.3M events since 5/12) misses texture-devices enrichment so the writer drops every batch (operations.length===0, silent no-op). texture-monitor-* last live doc 2026-05-12T23:30Z. Identified gaps: device.updated not consumed; program.* not consumed; SCADA/grid enrichment never wired (ALL-681 still Todo). Filed umbrella ticket ALL-1198 (Todo, Seph assigned, Monitors project) and queued follow-on spec fleet-task 352fdd70. Operational stop investigation complete; remediation tracked under ALL-1198.
status_change
in_progress → completed