“Inactive” vs “not allocated”
There is a difference in how Proctor handles “not allocated” (failed to match rule) and being put into an “inactive” group.
Invalid matrix a given test
Single test will be invalid and will be “random” 100% -1 bucket add interface to log and capture the load. (“loadListener”)
Preventing user drift using group changes
When increasing test sizes, ensure that users are not unintentionally switched from the
Use one of the following approaches to prevent user drift:
- Assign groups starting at the “ends” of your allocation ranges and increase the allocations:
- Assign groups starting at the “beginning” of your allocation ranges and increase allocations by using some
Ensure that users do not drive from test to control during allocation changes.
- Kill switch
- Dark deploy
- Feature toggle
- Reduce deploy dependencies