ID

b143638e-d4db-410a-98c4-35abe0ab9195

BE-6874 Fix Randomization and Improve Reporting in Bulk Experiment Enrollment Command

- source :: https://hungryroot.atlassian.net/browse/BE-6874

- tags :: Hungryroot API Project

- epic :: EP-411 Eng Parking Lot

- sprint :: HR BE Sprint 101

- status :: Status - Complete

- pull :: https://github.com/hungryroot/hungryroot/pull/12978

DONE Do a quick pass in Cursor to implement this fix

ID

5679b989-caee-eff2-3253-e193b5d5a27f

DONE Open a PR against master

ID

5679b989-caee-eff2-3253-e193b5d5a27f

DONE Test on Staging

ID

b93dc3c9-dbc1-2a9b-0405-af1c3ced2104

- Note taken on Thu Jan 22, 202617:17

Ran this on the QA box (`just ecs-exec qa`):

#+begin_src bash

./manage.py db bulk_enroll_experiment_percentage 19600 "" 1 True 0.3

#+end_src

Description

Comes From Andrew’s comment on a previous ticket here

https://hungryroot.slack.com/archives/C04GSJHFSTS/p1766102683103289

The bulk_enroll_experiment_percentage and

bulk_enroll_experiment_control_percentage management commands currently have two

related issues that can result in fewer customers being enrolled than expected

and misleading command output.

First, the randomization logic uses random.choice, which samples with

replacement. This means the same customer ID can be selected multiple times when

building random_ids. When these duplicate IDs are later processed via

bulk_create with ignore_conflicts=True, duplicates are silently skipped,

resulting in fewer actual enrollments than the intended target.

Second, the commands currently report theoretical enrollment counts prior to

randomization, rather than the number of customers actually enrolled. (May be

resolved after resolving the first bullet)

This ticket also expands the interface of

bulk_enroll_experiment_control_percentage to support the same qs_filters

parameter already available in bulk_enroll_experiment_percentage, allowing

callers to filter the eligible customer population before enrollment.

Scope / Expectations

Update randomization logic in both:

bulk_enroll_experiment_percentage

bulk_enroll_experiment_control_percentage

Replace random.choice with random.sample to ensure sampling without replacement

and prevent duplicate customer IDs from being selected.

Ensure the target enrollment size aligns with the number of unique customers

selected for enrollment.

Update both commands to report how many customers were actually enrolled, based

on persisted records, rather than theoretical counts.

Update bulk_enroll_experiment_control_percentage to accept a qs_filters

parameter, matching the interface and behavior of

bulk_enroll_experiment_percentage:

- qs_filters = list of queries to filter the customers being enrolled

Add or update tests to cover:

Correct sampling behavior without replacement

Accurate reporting of actual enrollments

Consistent behavior between percentage and control enrollment commands

Proper handling of qs_filters in both commands

Success Criteria

Duplicate customer IDs are no longer selected during randomization.

Actual enrollment counts closely match intended targets when sufficient eligible

customers exist.

Command output reflects the true number of customers enrolled.

Both management commands support qs_filters and behave consistently.

No silent enrollment shortfalls caused by duplicate sampling.