Org Web Adapter

Editing hungryroot/jira/be_6895_datadog_signup_and_checkout_metrics_appear_to_be_under_reporting.org

:PROPERTIES:
:ID:       c7bde569-3df7-4824-89e9-75097bc02fd7
:END:
#+title: BE-6895 Datadog signup and checkout metrics appear to be under reporting

- source :: https://hungryroot.atlassian.net/browse/BE-6895
- tags ::   [[id:3a477c02-e26d-40f7-a719-0aa1cd169dd8][Hungryroot API Project]] 
- epic ::   [[id:86327fb4-0f66-4bc6-9068-c414cea77042][EP-411 Eng Parking Lot]]
- sprint :: [[id:c82bb209-ab05-47b1-8f5d-3eebfb7a808a][HR BE Sprint 103]] 
- status :: [[id:4d41d978-e56f-45c1-88d1-e2c76510972e][Status - Complete]]

* Tasks
*** DONE Look through existing monitors and see about signup and checkout metrics :work:hungryroot:datadog:metrics:spike:sprint103:be411:be6895:
:PROPERTIES:
:ID:       d5d6da0b-dd15-fa96-adc7-57980264b4d9
:END:
*** DONE Dig into codebase to see how better to configure monitor goals :work:hungryroot:datadog:metrics:sprint103:be411:be6895:
:PROPERTIES:
:ID:       aca1c593-23c3-f8b5-eecf-45f6bd5fd967
:END:
*** DONE Write up findings on general accuracy of monitors versus the endpoint hits :work:hungryroot:datadog:metrics:sprint103:be6895:be411:
:PROPERTIES:
:ID:       655740ed-d76d-d7a2-80bd-6714e6b35fd2
:END:
*** DONE Review GPT notes and come up with a deliverable to metric miscounts :work:hungryroot:project:datadog:metrics:sprint103:be6895:be411:
:PROPERTIES:
:ID:       6fb7f7d4-468a-8875-9157-02c248fe6f91
:END:
<2026-02-12 Thu>
*** DONE Create follow up tickets to clean up order of operations for signup and chekcout :work:hungryroot:project:datadog:metrics:sprint103:be6895:ep411:spike:
:PROPERTIES:
:ID:       78c59435-1c4c-7c89-7240-6f3560cdf6c4
:END:
* Description
https://hungryroot.slack.com/archives/C02N30ZNRSQ/p1767036397438759

DD dashboard to demonstrate

We are seeing a material discrepancy between Datadog metrics, application logs,
and database counts for key customer lifecycle events, specifically signup and
checkout.

When comparing Datadog dashboard metrics to actual API traffic and database
records, the Datadog metrics appear to be significantly lower:

Example:

Datadog dashboard shows ~11.8k calls to /api/v2/auth/signup/ in the past 24 hours
(based on request logs / route-level data)

The Datadog metric hungryroot.customer.signup shows ~4.0k when using sum

Database queries indicate 13k+ new users created in the same time window

Similar discrepancies are observed for checkout-related endpoints such as
/api/place_checkout_order/.

There is uncertainty around:

Whether the metrics are intended to represent attempted actions vs successful
actions

Whether early returns (e.g., return JsonResponse({...})) with HTTP 200 responses
but incomplete business logic execution are inflating route-level logs compared
to metrics

Whether metrics are filtered by status code, execution path, feature flags, or
other conditions not reflected in logs or raw SQL counts

Notes

Early-return patterns with HTTP 200 responses may be contributing to inflated
request counts

Current inability to inspect response bodies in Datadog makes correlation harder

SQL counts currently suggest metrics are under-reporting real business activity

* GPT notes
1) Write down the “source of truth” definitions (before debugging)

For each lifecycle event (Signup, Checkout), explicitly define these three
separate concepts so everyone stops comparing mismatched things:

Attempted: request hit the route (route-level / access logs)

Processed: server reached the part of code that should “do the thing” (business
logic path)

Succeeded: durable state change happened (DB insert/commit, order created, etc.)

Then map each data source:

Datadog route/request logs → almost always “Attempted” (and may include bots,
retries, client errors, internal health checks, etc.)

DB counts → “Succeeded” (usually), but can include backfills, admin tools,
retries that created multiple rows, etc.

Custom metric hungryroot.customer.signup → unknown until you locate where it’s
emitted; could be any of the above.

This framing alone usually explains 50% of “metrics vs logs vs DB” tickets.

1. Find where the metric is emitted and document the exact semantics

The fastest path to truth is: locate the code that increments
    hungryroot.customer.signup and checkout metrics and answer:

Is it emitted on request start, after validation, after user/order creation, or
    after response?

Is it behind status-code checks, feature flags, early returns, exceptions, or
    sampling?

Does it increment only once per request, or can it be skipped or double-fired?

Practical checklist:

Search codebase for the metric name(s) and their wrappers
    (statsd/dogstatsd/client libs).

Identify conditional guards: if success, if created, if status_code == 200, if
    not dry_run, etc.

Confirm whether metric uses increment vs count vs distribution and whether tags
    affect queries.

Deliverable for the ticket: a short “Metric contract” blurb:

“hungryroot.customer.signup increments only after user is successfully created
    and committed; early-return 200s do not increment; failures increment
    ...signup_failed with reason tag.”

If that contract doesn’t exist, create it as part of the fix.

2. Reconcile the discrepancy with a 24h “join” on a correlation key (even
   without response bodies)

Since response bodies aren’t visible, use what you can correlate:

request_id / trace_id (Datadog trace + logs correlation)

user_id (if logged after auth/signup)

email hash (privacy-safe) or client_id

order_id (checkout)

status_code + latency + error tags

Make a small, time-bounded sample (e.g., last 1–2 hours) and do:

For signup

Count route hits to /api/v2/auth/signup/ by status_code and user agent and source
    IP / ASN (bots often explode request logs).

Count how many of those hits generate:

a user creation attempt log line (instrument one if missing)

a DB insert (new users)

the metric increment

For checkout

Same pattern:

route hits to /api/place_checkout_order/

“entered checkout business logic” log marker

order created in DB

checkout metric increment

This usually reveals one of:

metric only increments on “success path”

metric missing for some code path / feature flag

metric increments in background job but route logs include retries

log count inflated by bots/retries, DB count includes additional sources

3. Specifically test the “early-return 200” hypothesis

Your note about return JsonResponse(..., status=200) before business logic
    completes is a big red flag: it creates “Attempted=high,
    Processed/Succeeded=lower”.

Do two targeted checks:

In code: list all early returns in signup/checkout handlers, and what conditions
    trigger them.

In logs: bucket route hits by very low latency (e.g., <50ms) vs normal. Early
    returns often cluster in ultra-low latency.

If “attempts” are high and many are ultra-low latency 200s, you’re counting
    “attempts” not “success”.

4. Validate Datadog query semantics (this bites people constantly)

Even if the metric emission is correct, queries can undercount due to:

using sum on a metric type that should be count/as_count

rollup/aggregation mismatch (e.g., avg over hosts, missing as_count())

tags filtering (env, service, version, endpoint tag mismatch)

ingestion/namespace differences (multiple metric names, renamed tags)

dashboard time window vs logs time window (timezone/rollup)

+ Action:

+ Pull the raw metric in Metric Explorer and confirm:
        type (count/rate/distribution)
        tags present
        whether as_count() is needed
        whether you’re inadvertently grouping by something (host/pod) and then averaging

6. Decide what to fix: instrumentation vs dashboard vs expectation

Once you know what’s happening, the fix usually falls into one (or more) of
    these:

A) Metric is “correct” but being compared to the wrong thing

Update dashboards to show:

Attempts (route hits)

Successes (metric)

DB commits (DB)

Add a panel for failures by reason/status

B) Metric under-reports due to missing code paths

Emit metric at a more authoritative point:

after commit for signup

after order persisted for checkout

Add explicit *_failed metrics with reason tags

Ensure feature-flagged paths also emit

C) Logs over-report because they include non-business traffic

Add filters for bots/health checks/internal probes

Separate endpoints: “preflight” vs “submit”

Ensure retries are observable (idempotency keys)

7. Minimum “next steps” you can put in the ticket right now

Locate the emission point for hungryroot.customer.signup + checkout metric(s);
    document success/failure semantics.

In Datadog, validate the metric type and query (as_count() etc.) and confirm
    tags/filters match environment/service.

Create a 1–2 hour correlation sample:

route hits by status + latency

count of “entered business logic” marker

DB created rows

metric increments

If early-return 200s exist, add:

signup_attempted metric on request entry

signup_succeeded metric after commit

signup_failed with reason tag on early return/exception (Same for checkout.)

If you want, paste where the metric is emitted (a small code snippet) and the
    exact Datadog query used on the dashboard, and I’ll tell you the most likely
    mismatch (query semantics vs instrumentation vs traffic composition) and the
    fastest fix.