Fork me on GitHub

Proctor - A/B Testing Framework by Indeed

Test Definition

The test definition describes a test's buckets, allocations, eligibility rules, allocations and constants. Each test's definition.json file should be under revision control.

.
├── test-definitions
|   ├── your-test-name
|       ├── definition.json

JSON Schema

JSON Property Description
testType The Identifier to use for this test
version the version for this test
salt A salt used in the hashing function used to map String identifiers to integer values. A good convention is to use the test name. Salts that start with "&" can allow you to align bucket assignments by using identical salts.
description description of the tests and the features impacted by this test
rule (optional) eligibility rule for this test
constants (optional) collection of variables available in the rules for this test
buckets An array of buckets for this test
allocations An array of allocations for this test

Bucket schema

JSON Property Description
name human-readable name for this bucket. By convention, this is the same as the name in the application's specification.
value int value for this bucket. This bucket must map to a bucket value of an application's specification
description human-readable description for the behavior this bucket defines
payload.{payloadType} (optional) payload data. The payloadType must be consistent with the payload.type defined in the application's specification

Allocation schema

JSON Property Description
rule (optional) rule for this allocation
ranges An array of bucketValue, length pairs describing the bucket distribution
ranges[i].bucketValue The bucket value for this part of the distribution
ranges[i].length The group size, [0, 1.0], for the given bucket value

Example: single allocation

Property Value
testType USER
rule null
buckets inactive, altcolor1, altcolor2, altcolor3, altcolor4
constants {}
allocations 1 allocation
rule allocation
Default (rule = null)
altcolor1 inactive altcolor2
25% 50% 25%

Example: multiple allocations with rules

Property Value
testType USER
buckets inactive, altcolor1, altcolor2, altcolor3, altcolor4
rule ${proctor:contains(COUNTRIES, country)}
constants "COUNTRIES" : ["US", "CA"]
allocations 3 allocations
rule allocation
${ua.android && ua.version > 4}
altcolor1 altcolor2
50% 50%
${ua.IPhone && ua.version > 7}
altcolor3 altcolor4
50% 50%
Default (rule = null)
altcolor1 altcolor2 altcolor3 altcolor4
25% 25% 25% 25%