Specification

Overview

The Proctor specification is a JSON file that describes the interface between a test-definition and application-specific code values. Each specification enumerates the tests it expects the test-matrix to contain and how each test’s bucket maps to application code. The specification file should be checked in with the application’s source code. The file must be available at runtime so your application can ensure the test-matrix contains the required tests and buckets.

The code generator uses the specification’s filename, path, and contents, so the information should be meaningful for your application.

# Example application
app: ToyStore
specification: ToyStoreGroups.json

# File structure if using proctor-maven-plugin
.
├── src
|   ├── main
|       ├── proctor
|           ├── org/your/organization/app/store/ToyStoreGroups.json

# File structure if using proctor-ant-task
.
├── src
|   ├── resources
|       ├── org/your/organization/app/store/ToyStoreGroups.json

or for split specifications:

# File structure if using proctor-maven-plugin
.
├── src
|   ├── main
|       ├── proctor
|           ├── org/your/organization/app/store/ToyStore
|               ├── firsttest.json
|               ├── secondtest.json
|               ├── thirdtest.json
|               ├── providedcontext.json

# File Structure if using proctor-ant-task
.
├── src
|   ├── resources
|       ├── org/your/organization/app/store
|           ├── firsttest.json
|           ├── secondtest.json
|           ├── thirdtest.json
|           ├── providedcontext.json

In the above code, providedcontext.json is a required file and the other part specifications contain single-test specifications. The names of the test are determined from the form testname.json. ###Ant In ant, the generated class still uses the provided build.xml classnames and package names. However, the input parameter must be the containing directory, and the output for the generated specifications must be specified. ###Maven In maven the generated class uses the standard of DirectoryGroups and DirectoryGroupsManager in naming files where Directory is the containing directory of the JSONs (in this case, ToyStore).

Refer to the Code Generator page for guidelines about using the maven and ant plugins to generate code based on your application’s specification.

JSON Schema

With a split specification, each file defines a test that would be a property of tests, as well as provided context being separately defined.

Defining Tests

The above specification defines a single test bgcolortst with five buckets: inactive, altcolor1, altcolor2, altcolor3, altcolor4.

Context

A specification’s providedContext defines the collection of application-specific variables available in the test definition rules. The providedContext maps variable names to Java types and is used to auto-generate methods with these types. In a web application, these variables typically map directly to the properties of a request, such as logged-in, country, language and user-agent. When using non-primitive types, like the custom UserAgent interface below, use the fully qualified name.

If an application had the following four variables:

The test definition’s eligibility rules and allocation rules can then reference the variables. The excerpt (complete example) below illustrates how a test-definition can reference the country and ua (UserAgent) variables to build targeted tests. Refer to the Test Definition page for a complete guide to the rule syntax.

{
  "testType" : "USER",
  "constants" : {
    "COUNTRIES" : ["US", "CA"]
  },
  "rule" : "${proctor:contains(COUNTRIES, country)}",
  ...
  "allocations" : [ {
    "rule" : "${ua.android && ua.version > 4}",
    "ranges" : [ {
      "length" : 0.5,
      "bucketValue" : 0
    }, {
      "length" : 0.5,
      "bucketValue" : 1
    } ]
  }
  ...
}

Payloads

Arbitrary data can be associated with each test bucket and delivered to your applications via the test-matrix. An application’s specification can indicate if it expects a given test to have payloads by specifying the payload.type:

Proctor supports 8 types of payloads:

The values for each bucket’s payload are specified in the test-definition (view complete test definition)

{
  ...
  "buckets" : [ {
    "name" : "inactive",
    "value" : -1,
    "description" : "Inactive",
    "payload": {
      "stringValue": "#000000"
    }
  }, {
    "name" : "altcolor1",
    "value" : 0,
    "description" : "Background color 1",
    "payload": {
      "stringValue": "#000000"
    }
  ...
  }
  ...
}

To specify a map type use

"payload": {
  "map": {
    "firstVariable" : [2.2,3.3],
    "2ndVar" : "#000000"
  }
}

"payload": {
  "map": {
    "firstVariable" : "doubleArray",
    "2ndVar" : "stringValue"
  }
}

Payload Validator (Optional)

An application can optionally define a payload.validator string in its specification. Similar to eligibility rules, this string is a boolean expression that should return true if a payload value is valid. During the test-matrix load-and-validate phase, each bucket’s payload will be checked for compatibility using this expression. If no validator is provided, all payload values (of the correct type) will be considered valid.

"payload": {
  "type": "stringValue",
  "validator": "${fn:startsWith(value, '#') && fn:length(value) == 7}"
}

The validator from the above example enforces that each payload, referenced by value in expression, starts with “#” and is 7 characters long. such as ‘#000000’. Unlike eligibility rules and allocation rules, the only available variable is value, the payload value. The specification context variables and test-constants are NOT available in the validator expression.

However, in the case of a map payload, use the variable names in the validator instead such as:

"payload": {
  "type": "map",
  "schema" : {
       "var1" : "doubleValue",
       "vartwo" : "longArray"
  },
  "validator": "var1 + vartwo[0] < 10"
}

Currently, json payloads do not support payload validators.

An application should always provide a default payload value in code and be resilient situations in which the test-matrix cannot be loaded.

Multiple Tests

Multiple tests can be enumerated in an application’s test specification by adding another entry to the tests map.

{
    "tests" : {
        // Using a proctor test as a feature flag
        "featureA": { "buckets" : {"inactive": -1, "disabled":0, "enabled":1}, "fallbackValue" : -1 },
        // horizontal/vertical/reverse layout test
        "layouttst": { "buckets" : {"inactive": -1, "horizontal":0, "vertical":1, "reverse":2}, "fallbackValue" : -1 }
    },
    "providedContext": {
        "lang": "String",
        "country" : "String"
    }
}

Split Specifications

A split specification (as opposed to a single large specification) can be used as documented above and on the Code Generation page. The format of these split specifications with a test similar to the Multiple Tests example would look like this:

featureA.json

{
    // Using a proctor test as a feature flag
     "buckets" : {"inactive": -1, "disabled":0, "enabled":1},
     "fallbackValue" : -1
}

layouttst.json

{
    // horizontal/vertical/reverse layout test
    "buckets" : {"inactive": -1, "horizontal":0, "vertical":1, "reverse":2},
    "fallbackValue" : -1
}

providedcontext.json

{
    "lang": "String",
    "country" : "String"
}

Note on Fallback Values

A test’s fallbackValue corresponds to the bucket value that should be used if a test matrix is not valid with the application’s specification. The Proctor-loader page describes how a test-definition can be invalid. In this scenario, the new definition will be used in place of the invalid matrix. This definition will have type=RANDOM and will allocate 100% of users to the test bucket signified by fallbackValue. If Identifiers.randomEnabled is set to false, the invalid test will not be allocated. Refer to the Using groups page for details about handling this correctly in your application’s code.