A 50/50 split logs 50.8/49.2 across 2M users. What does this SRM signal, and how do you fix the assignment below?
An experiment must assign users 50/50, but exposure logs show 50.8% control / 49.2% treatment across 2,000,000 users.
Constraints: assignment is deterministic per user_id; ~40% of traffic is ineligible for the feature; each exposed user is logged in exactly one arm.
ELIGIBLE = load_eligible_users() # ~60% of all users
def assign(user_id: str) -> str:
h = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
if h % 1000 < 500:
return "control"
if user_id in ELIGIBLE: # eligibility tested only here
return "treatment"
return "control" # ineligible -> sent to controlExplain what the SRM means for the readout, then fix the assignment so the split is balanced.
SRM means the split deviates from the intended ratio beyond chance — χ² on the arm counts gives a tiny p-value — so assignment or logging is broken and the readout is void. The bug: eligibility is tested only on the treatment branch, so ineligible users fall to control; filter them out before bucketing.
- ✗Dismissing a significant SRM as harmless rounding noise
- ✗Trying to reweight or patch the arms instead of fixing assignment and re-running
- ✗Applying an eligibility or logging filter to only one arm
- →Which χ² statistic and threshold would you use to flag the SRM?
- →Why can't you salvage the biased data by reweighting after the fact?
The eligibility gate is applied asymmetrically — only users routed toward treatment are tested — so every ineligible user in the top half of the hash space is redirected to control. Control therefore collects roughly 50% plus the ineligible share of the top half, which produces the surplus. Randomise within the eligible population only, deciding membership before bucketing:
def assign(user_id: str) -> str | None:
if user_id not in ELIGIBLE:
return None # not in the experiment at all
h = int(hashlib.md5((user_id + SALT).encode()).hexdigest(), 16)
return "control" if h % 1000 < 500 else "treatment"Now both arms are drawn from the same filtered population, the split returns to 50/50, and the χ² SRM check passes. Discard data collected before the fix and re-run.
