Org Web Adapter

Editing hungryroot/jira/be_6842_create_datadog_monitor_for_spikes_in_invalid_items_removed.org

:PROPERTIES:
:ID:       f40715c7-4bf0-4308-b7e6-be441bdc78e8
:END:
#+title: BE-6842 Create Datadog monitor for spikes in "invalid items removed ..."

- source :: https://hungryroot.atlassian.net/browse/BE-6842
- tags ::  [[id:3a477c02-e26d-40f7-a719-0aa1cd169dd8][Hungryroot API Project]] 
- epic :: [[id:86327fb4-0f66-4bc6-9068-c414cea77042][EP-411 Eng Parking Lot]]
- sprint :: [[id:2d34c333-e997-4a5b-9e83-2218d1b76940][HR BE Sprint 101]] [[id:26939bb8-164c-402e-8a6a-6ae8c99d3e59][HR BE Sprint 102]]
- status ::  [[id:4d41d978-e56f-45c1-88d1-e2c76510972e][Status - Complete]]
- link :: https://app.datadoghq.com/monitors/251590880

* Tasks
** DONE Look into how to create monitors in DD :work:chore:datadog:be6842:ep411:sprint101:hungryroot:
:PROPERTIES:
:ID:       395c5b5b-d057-ddac-1e95-48a6736ff450
:END:

- Note taken on [2026-01-16 Fri 11:58]

Ended up asking [[id:e5955159-7006-4c05-a38a-52e9d4b886da][ChatGPT]] to walk me through setting this up, and it ended up going pretty quick.
  
** DONE Move to a simple log alert :work:chore:datadog:be6842:ep411:sprint101:hungryroot:
:PROPERTIES:
:ID:       6ca2bde2-dbdb-8bb0-22ae-d694b370e57e
:END:
  
* Description
https://hungryroot.slack.com/archives/C0A38CJ930E/p1765571451341439

DD logs

We need to add observability around a specific ingestion event that indicates
potential issues in box content processing.

Background

An ingestion event with the message “Invalid items removed from box content
after shuffle” has been observed in production. OR group identified and fixed an
issue related to this, but this type of problem should have been detected
earlier in staging. Currently, there is no automated alerting when these events
spike.

As noted by Nannan Chen:

OR group have identified the issue, fixed it, and are currently working on local
tests, but this should have been easily caught in staging and we did not receive
any ingestion failure signals there.

Request

Create a Datadog monitor that detects spikes in ingestion events with the
message “Invalid items removed from box content after shuffle”.

Requirements / Acceptance Criteria

Monitor tracks the ingestion event message:

“Invalid items removed from box content after shuffle”

Monitor applies to both:

Staging environment

Production environment

Trigger condition:

Alert on a spike or abnormal increase above a reasonable baseline (threshold to
be determined based on historical data).

Alerting behavior:

Send alerts to the #datadog Slack channel.

Alert message should instruct responders to investigate and raise the issue with
#boxfill-dev if needed.

Monitor configuration should clearly document:

What signal is being monitored

Why it is important

Expected next steps when the alert fires

Goal

Ensure ingestion-related regressions affecting box content are caught early in
staging and surfaced quickly in production, reducing time to detection and
escalation.