Fork me on GitHub

Proctor - A/B Testing Framework by Indeed

Terminology

specification

a description of the interface between an application and a test definition. Specifically that is what it expects from that test and how the test buckets map to application-level properties. A specification consists of tests, buckets, and context. It is used to generate the interface code.

test-matrix

a list of active test definitions used by all applications compiled into a single file. Generated by the builder and loaded by each application.

context

a collection of application-specific variables available in the test definition rules. In a web application, these variables typically map directly to the properties of a request. e.g. logged-in, country, language and user-agent

test ( experiment )

a set of behaviors that are simultaneously delivered to different users in order to assess differences in how they affect user actions

test definition

a description of how users should be assigned to buckets in the test. That is done via specifying the active buckets, eligibility rules, allocation rules, test group sizes, test constants, and the salt.

group ( bucket )

a single behavior within a test. A group is determined by evaluating a test definition (in the test matrix) using the provided identifier and context. A bucket has a name, integer value and optional payload containing test-specific data.

payload

optional data associated with each test bucket that defines one aspect of the bucket's behavior.

eligibility rule

an expression determining if a test-definition is relevant for a provided context. If this expression evaluates to false, no bucket for this test should be selected.

allocation

a description of the test's group sizes (bucket distribution). A test-definition can have multiple allocations that are conditionally used depending on its allocation rule.

allocation rule

an expression determining if an allocation is relevant for a provided context. If an allocation's expression evaluates to true, its group sizes should be used.

identifier

an id that will be used when determining the active groups. The id typically unique and persistent across requests (session cookie, account id, etc). Each identifier is associated with a specific test-type. The groups are determined by evaluating the test-matrix for a given identifier and context.

test-type

an enumeration of the identifier to use when allocating test buckets

  • USER: a unique value identifying a user; typically a tracking cookie value
  • ACCOUNT: account-based identifiers, fixed across devices; typically a user-id
  • EMAIL: an email address of a user
  • RANDOM: no identifier will be used.

test constants

variables that can be referenced in an eligibility rule or allocation rule.

special-constants (test-definition)

-- NOTE: interaction between special-constants + rule containing "${}" braces. (convertToConsumableTestDefinition)

builder

a tool responsible for generating the test-matrix consumed by your applications. In practice, this tool validates and combines the test-definitions stored across multiple files into a single file.

user

A client consuming your service. Your A/B tests should be designed to influence the behavior of an individual directly (eg. changing a UI element / funnel ) or indirectly (eg. changing the algorithm used to display search results). Your application should be recording and measuring the behavior of your users so that your A/B tests can be evaluated.

For most web applications, a user maps directly to a human using your application. In some circumstances, a user could represent another application consuming your service (eg. another application consuming the application's API).