Terminology
specification
A description of the interface between an application and a test definition. Specifies the expectation of the test and the ways the test buckets map to application-level properties. Consists of tests, buckets, and context. Used to generate the interface code.
test-matrix
A list of active test definitions used by all applications compiled into a single file. Generated by the builder and loaded by each application.
context
A collection of application-specific variables available in the test definition rules. In a web application, these variables typically map directly to the properties of a request, such as logged-in, country, language and user-agent.
test ( experiment )
A set of behaviors simultaneously delivered to different users to assess effects on user actions.
test definition
A description of the ways users are assigned to buckets in the test. Specifies the active buckets, eligibility rules, allocation rules, test group sizes, test constants, and the salt.
group ( bucket )
A single behavior within a test. A group, also called a bucket, is determined by evaluating a test definition in the test matrix using the provided identifier and context. Each bucket has a name, integer value, and optional payload containing test-specific data.
payload
Optional data associated with each test bucket that defines one aspect of the bucket’s behavior.
eligibility rule
An expression determining whether a test-definition is relevant for a provided context. If this expression evaluates to false, no bucket for this test should be selected.
allocation
A description of the test’s group sizes (bucket distribution). A test-definition can have multiple allocations that are conditionally used depending on its allocation rule.
allocation rule
An expression determining whether an allocation is relevant for a provided context. If an allocation’s expression evaluates to true, its group sizes should be used.
identifier
An id that will be used when determining the active groups. The id is typically unique and persistent across requests (session cookie, account id, etc). Each identifier is associated with a specific test-type. The groups are determined by evaluating the test-matrix for a specified identifier and context.
test-type
An enumeration of the identifier to use when allocating test buckets:
USER: A unique value identifying a user; typically a tracking cookie valueACCOUNT: An account-based identifier, fixed across devices; typically a user-idEMAIL: An email address of a userRANDOM: No identifier is used
test constants
Variables you can reference in an eligibility rule or allocation rule.
special-constants (test-definition)
– NOTE: interaction between special-constants + rule containing “${}” braces. (convertToConsumableTestDefinition)
builder
A tool responsible for generating the test-matrix your applications consume. The tool validates test-definitions, which are stored across multiple files, and combines them into a single file.
user
A client consuming your service. Your A/B tests should be designed to influence the behavior of an individual directly (changing a UI element/funnel) or indirectly (changing the algorithm used to display search results). Your application should record and measure user behavior so your A/B tests can be evaluated.
For most web applications, a user represents a human using your application. In some circumstances, a user could represent another application consuming your service, such as another application consuming the application’s API.