Org Web Adapter

Editing hungryroot/jira/be_6874_fix_randomization_and_improve_reporting_in_bulk_experiment_enrollmenet_command.org

:PROPERTIES:
:ID:       b143638e-d4db-410a-98c4-35abe0ab9195
:END:
#+title: BE-6874 Fix Randomization and Improve Reporting in Bulk Experiment Enrollment Command

- source :: https://hungryroot.atlassian.net/browse/BE-6874
- tags ::  [[id:3a477c02-e26d-40f7-a719-0aa1cd169dd8][Hungryroot API Project]] 
- epic :: [[id:86327fb4-0f66-4bc6-9068-c414cea77042][EP-411 Eng Parking Lot]]
- sprint :: [[id:2d34c333-e997-4a5b-9e83-2218d1b76940][HR BE Sprint 101]]
- status :: [[id:4d41d978-e56f-45c1-88d1-e2c76510972e][Status - Complete]]
- pull :: https://github.com/hungryroot/hungryroot/pull/12978
  
** DONE Do a quick pass in Cursor to implement this fix :work:hungryroot:bug:be6874:ep411:sprint101:cursor:
:PROPERTIES:
:ID:       5679b989-caee-eff2-3253-e193b5d5a27f
:END:
** DONE Open a PR against master :work:hungryroot:bug:be6874:ep411:sprint101:
:PROPERTIES:
:ID:       5679b989-caee-eff2-3253-e193b5d5a27f
:END:
** DONE Test on Staging :work:hungryroot:project:qa:be6874:ep411:sprint102:
:PROPERTIES:
:ID:       b93dc3c9-dbc1-2a9b-0405-af1c3ced2104
:END:

- Note taken on [2026-01-22 Thu 17:17]

Ran this on the QA box (`just ecs-exec qa`):
  #+begin_src bash
./manage.py db bulk_enroll_experiment_percentage 19600 "" 1 True 0.3
  #+end_src
* Description
Comes From Andrew’s comment on a previous ticket here

https://hungryroot.slack.com/archives/C04GSJHFSTS/p1766102683103289

The bulk_enroll_experiment_percentage and
bulk_enroll_experiment_control_percentage management commands currently have two
related issues that can result in fewer customers being enrolled than expected
and misleading command output.

First, the randomization logic uses random.choice, which samples with
replacement. This means the same customer ID can be selected multiple times when
building random_ids. When these duplicate IDs are later processed via
bulk_create with ignore_conflicts=True, duplicates are silently skipped,
resulting in fewer actual enrollments than the intended target.

Second, the commands currently report theoretical enrollment counts prior to
randomization, rather than the number of customers actually enrolled. (May be
resolved after resolving the first bullet)

This ticket also expands the interface of
bulk_enroll_experiment_control_percentage to support the same qs_filters
parameter already available in bulk_enroll_experiment_percentage, allowing
callers to filter the eligible customer population before enrollment.

** Scope / Expectations

Update randomization logic in both:

bulk_enroll_experiment_percentage

bulk_enroll_experiment_control_percentage

Replace random.choice with random.sample to ensure sampling without replacement
and prevent duplicate customer IDs from being selected.

Ensure the target enrollment size aligns with the number of unique customers
selected for enrollment.

Update both commands to report how many customers were actually enrolled, based
on persisted records, rather than theoretical counts.

Update bulk_enroll_experiment_control_percentage to accept a qs_filters
parameter, matching the interface and behavior of
bulk_enroll_experiment_percentage:

- qs_filters = list of queries to filter the customers being enrolled

** Add or update tests to cover:

Correct sampling behavior without replacement

Accurate reporting of actual enrollments

Consistent behavior between percentage and control enrollment commands

Proper handling of qs_filters in both commands

** Success Criteria

Duplicate customer IDs are no longer selected during randomization.

Actual enrollment counts closely match intended targets when sufficient eligible
customers exist.

Command output reflects the true number of customers enrolled.

Both management commands support qs_filters and behave consistently.

No silent enrollment shortfalls caused by duplicate sampling.