Proctor - A/B Testing Framework by Indeed

Test Definition

The test definition describes a test’s buckets, allocations, eligibility rules, allocations and constants. Each test’s definition.json file should be under revision control.

├── test-definitions
|   ├── your-test-name
|       ├── definition.json

JSON Schema

JSON Property Description
testType The Identifier to use for this test.
version The version for this test.
salt A salt in the hashing function used to map String identifiers to integer values. A good convention is to use the test name. Salts that start with “&” allow you to align bucket assignments by using identical salts.
description Description of the tests and the features impacted by this test.
rule (Optional) eligibility rule for this test.
constants (Optional) Collection of variables available in the rules for this test.
buckets An array of buckets for this test.
allocations An array of allocations for this test.

Bucket Schema

JSON Property Description
name The human-readable name for this bucket. By convention, this is the same as the name in the application’s specification.
value The integer value for this bucket. This bucket must map to a bucket value of an application’s specification.
description The human-readable description for the behavior this bucket defines.
payload.{payloadType} (Optional) Payload data. The payloadType must be consistent with the payload.type defined in the application’s specification.

Allocation Schema

JSON Property Description
rule (Optional) rule for this allocation.
ranges An array of bucketValue, length pairs describing the bucket distribution.
ranges[i].bucketValue The bucket value for this part of the distribution.
ranges[i].length The group size, [0, 1.0], for the given bucket value.

Example: Single Allocation

Property Value
testType USER
rule null
buckets inactive, altcolor1, altcolor2, altcolor3, altcolor4
constants {}
allocations 1 allocation
Rule Allocation
Default (rule = null)
altcolor1 inactive altcolor2
25% 50% 25%

Example: Multiple Allocations with Rules

Property Value
testType USER
buckets inactive, altcolor1, altcolor2, altcolor3, altcolor4
rule ${proctor:contains(COUNTRIES, country)}
constants "COUNTRIES" : ["US", "CA"]
allocations 3 allocations
Rule Allocation
${ && ua.version > 4}
altcolor1 altcolor2
50% 50%
${ua.IPhone && ua.version > 7}
altcolor3 altcolor4
50% 50%
Default (rule = null)
altcolor1 altcolor2 altcolor3 altcolor4
25% 25% 25% 25%