Proctor - A/B Testing Framework by Indeed

Best Practices

Gotchas

“Inactive” vs “not allocated”

There is a difference in how Proctor handles “not allocated” (failed to match rule) and being put into an “inactive” group.

Invalid matrix a given test

Single test will be invalid and will be “random” 100% -1 bucket add interface to log and capture the load. (“loadListener”)

Preventing user drift using group changes

When increasing test sizes, ensure that users are not unintentionally switched from the test to control buckets.

Use one of the following approaches to prevent user drift:

  • Assign groups starting at the “ends” of your allocation ranges and increase the allocations:
control inactive test
10% 80% 10%
control inactive test
20% 60% 20%
  • Assign groups starting at the “beginning” of your allocation ranges and increase allocations by using some inactive users:
control test inactive
10% 10% 80%
control test inactive control test
10% 10% 10% 10% 60%

Ensure that users do not drive from test to control during allocation changes.

Use cases

  • Kill switch
  • Dark deploy
  • Feature toggle
  • Reduce deploy dependencies