- ID
- f40715c7-4bf0-4308-b7e6-be441bdc78e8
BE-6842 Create Datadog monitor for spikes in "invalid items removed ..."
- source :: https://hungryroot.atlassian.net/browse/BE-6842
- tags :: Hungryroot API Project
- epic :: EP-411 Eng Parking Lot
- sprint :: HR BE Sprint 101 HR BE Sprint 102
- status :: Status - Complete
- link :: https://app.datadoghq.com/monitors/251590880
Tasks
DONE Look into how to create monitors in DD
- ID
- 395c5b5b-d057-ddac-1e95-48a6736ff450
- Note taken on
Ended up asking ChatGPT to walk me through setting this up, and it ended up going pretty quick.
DONE Move to a simple log alert
- ID
- 6ca2bde2-dbdb-8bb0-22ae-d694b370e57e
Description
https://hungryroot.slack.com/archives/C0A38CJ930E/p1765571451341439
DD logs
We need to add observability around a specific ingestion event that indicates
potential issues in box content processing.
Background
An ingestion event with the message “Invalid items removed from box content
after shuffle” has been observed in production. OR group identified and fixed an
issue related to this, but this type of problem should have been detected
earlier in staging. Currently, there is no automated alerting when these events
spike.
As noted by Nannan Chen:
OR group have identified the issue, fixed it, and are currently working on local
tests, but this should have been easily caught in staging and we did not receive
any ingestion failure signals there.
Request
Create a Datadog monitor that detects spikes in ingestion events with the
message “Invalid items removed from box content after shuffle”.
Requirements / Acceptance Criteria
Monitor tracks the ingestion event message:
“Invalid items removed from box content after shuffle”
Monitor applies to both:
Staging environment
Production environment
Trigger condition:
Alert on a spike or abnormal increase above a reasonable baseline (threshold to
be determined based on historical data).
Alerting behavior:
Send alerts to the #datadog Slack channel.
Alert message should instruct responders to investigate and raise the issue with
#boxfill-dev if needed.
Monitor configuration should clearly document:
What signal is being monitored
Why it is important
Expected next steps when the alert fires
Goal
Ensure ingestion-related regressions affecting box content are caught early in
staging and surfaced quickly in production, reducing time to detection and
escalation.