Fork me on GitHub

Proctor - A/B Testing Framework by Indeed

Using Groups

At this point you should be familiar with the test specification, code generator and loader concepts. This document expands on the quick start guide and provides guidance on how to use each proctor component.

setup

final JsonProctorLoaderFactory factory = new JsonProctorLoaderFactory();
// Loads the specification from the classpath resource
factory.setSpecificationResource("classpath:/org/example/proctor/ExampleGroups.json");
// Loads the test matrix from a file
factory.setFilePath("/var/local/proctor/test-matrix.json");
final AbstractJsonProctorLoader loader = factory.getLoader();
// schedule the loader to refresh every 30 seconds
final ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
scheduledExecutorService.scheduleWithFixedDelay(loader, 0, 30, TimeUnit.SECONDS);
// Create groups manager for application-code usage
final ExampleGroupsManager exampleGroupsManager = new ExampleGroupsManager(loader);

how to determine groups

When initializing an application's AbtractGroups class, ExampleGroups in our case, a best-practice is to use the generated AbstractGroupsManager api, ExampleGroupsManager in our case, instead of the using Proctor directly.

public ProctorResult determineBuckets(Identifiers identifiers 
                                      [,context-variable_1 ... ,context-variable_N]) { ... }
public ProctorResult determineBuckets(HttpServletRequest request, 
                                      HttpServletResponse response, 
                                      boolean allowForceGroups,
                                      Identifiers identifiers
                                      [,context-variable_1 ... ,context-variable_N]) { ... }

The generated AbstractGroupsManager handles several things for you:

  • Uses the most-recently loaded test-matrix provided by the AbstractProctorLoader implementation -- Will return a default ProctorResult if AbstractProctorLoader returns a null Proctor instance
  • Maps application context variables to their variable names as defined in the specification context.
  • Overrides groups specified via a url-parameter or cookie-value for internal testing (if allowForceGroups is true)

where to determine groups

Most applications should initialize their applications in a single location in code so to ensure consistently across usages of the groups. For web applications, this usually means placing the group initialization early in the request-response lifecycle so that groups are across requests. The group determination must be after all context variables and test identifiers can be resolved for a given request. Your application's values for this information will impact where the group determination can happen.

At Indeed, we use the ACCOUNT identifier for loggedin-account-based tests and language and country context parameters for locale-specific tests. Because of this, the groups determination must come after the code that identifies a user based on her cookies and determines the language for this request.

In practice, this occurs in a javax.servlet.Filter or Spring org.springframework.web.servlet.HandlerInterceptor. Other web frameworks have their own injection points to pre-process requests.

Example Spring interceptor for determining groups:

NOTE: lines 39-41 catch all exceptions including those thrown when evaluating the javax.el rule expressions. At Indeed, we've chose to log these exceptions and use empty-groups instead of erroring during the request

using the generated code

Java

The groups generated code contains convenience methods for accessing the each test's group and checking if a given group is active.

The ExampleGroups.json specification will generate the following ExampleGroups.java class (the method implementations have been removed for brevity)

 1 public class ExampleGroups extends AbstractGroups {
 2     public ExampleGroups(final ProctorResult proctorResult) {
 3         super(proctorResult);
 4     }
 5     public enum Test {
 6         BGCOLORTST("bgcolortst");
 7     }
 8     public enum Bgcolortst implements Bucket<Test> {
 9         INACTIVE(-1, "inactive"),
10         ALTCOLOR1(0, "altcolor1"),
11         ALTCOLOR2(1, "altcolor2"),
12         ALTCOLOR3(2, "altcolor3"),
13         ALTCOLOR4(3, "altcolor4");
14     }
15 
16     // test bucket accessor
17     public Bgcolortst getBgcolortst() { }
18     public int getBgcolortstValue(final int defaultValue) { }
19 
20     public boolean isBgcolortstInactive() { }
21     public boolean isBgcolortstAltcolor1() { }
22     public boolean isBgcolortstAltcolor2() { }
23     public boolean isBgcolortstAltcolor3() { }
24     public boolean isBgcolortstAltcolor4() { }
25 
26     // Payload accessors
27     public @Nullable String getBgcolortstPayload() { }
28     public @Nullable String getBgcolortstPayloadForBucket(final Bgcolortst targetBucket) { }
29 }

unassigned -> fallback

Previous releases of the proctor library allowed users to be placed in an unassigned bucket that differed from the fallback bucket. This is no longer the case; the generated code will now consider unassigned users to be a part of the fallback bucket declared in the project specification.

sometimes inactive != control

For many tests the fallback bucket displays the same behavior as the control group. This is typical if you want to place your users into equal-sized buckets but do not want to set the test-behavior to 50%.

For example, you're showing an experimental design to 10% of your users; the remaining 90% will continue to see the status-quo (control). It's common to allocate three buckets: inactive (80%), control (10%), test (10%), and mark the inactive bucket as the fallback. Users in the inactive bucket will experience the same behavior as users in the control. Although 90% of users will experience the status-quo design, only 10% will be labeled as control. This makes absolute comparisons across metrics easier because the control and test groups sizes are of equal size.

When inactive and control users experience the same behavior, the following code correctly displays the new feature to the test group.

if(groups.isFeatureTest()) {
  // do test
} else {
  // fall through for "control" and "inactive"
}

However, there are situations where control != inactive.

For example, you've chosen to integrate with google-analytics help track your tests by setting a custom variable. Users in the control and test buckets should include a unique GA variable value so their traffic can be segmented in google-analytics. The inactive users will not get a custom variable. Your application code should explicitly handle the test and control buckets separate from the inactive users.

if(groups.isFeatureTest()) {
  // do test
} else if(groups.isFeatureControl()) {
  // do control
} else {
  // fall through for "inactive"
}

In general:

  • program towards the test behavior, not the control behavior
  • create an inactive bucket rather than assigning control as the fallback bucket

Javascript

Once you have used your generated Java code to determine buckets for each of your test groups, you can pass the allocations to your generated javascript.

There is a method in AbstractGroups that will create an array of group allocations and payload values. Pass it an array of Test enums from Test.values().

public <E extends Test> List<List<Object>> getJavaScriptConfig(final E[] tests);

Now you can serialize this result to JSON or just write it out to the page in Javascript. Then init() your groups and use them as needed.

    var proctor = require('com.indeed.example.groups');
    proctor.init(values);

    ...

    if (proctor.getGroups().isFeatureTest()) {
        //do test
    else {
        //fall through for "control" or "inactive"
    }

using payloads interface

invalid test-definitions

how to record group assignment

The AbstractGroups interface provides a toString() method that contains a comma-delimited list of active groups. Only tests with non-negative bucket value will be logged; this follows the convention that the inactive fallback bucket has a value of -1 and doesn't contain test behavior. The string representation for each group is [test_name][bucket_value].

Consider the following specification:

{
    "tests" : {
        // Test different single-page implementations (none, html5 history, hash-bang )
        "historytst": { "buckets" : {
          "inactive": -1, "nohistory":0, "html5":1, "hashbang":2
        }, "fallbackValue" : -1 },
        // test different signup flows
        "signuptst": { "buckets" : {
          "inactive": -1, "singlepage":0, "accordian":1, "multipage":2, "tabs":3 
        }, "fallbackValue" : -1 }
    }
}
historytst signuptst Groups.toString()
inactive inactive ""
nohistory inactive "historytst0"
html5 tabs "historytst1,signuptst3"

It's up to your application to decide how to log these groups and analyze the outcomes. At Indeed, we log all the groups on every requests and make each test available in all of our analyses. In at least one situation, this has helped us identify and explain unexpected user-behavior on pages not directly impacted by a test-group's behavior.

extending groups

Feel free to extend the generated Groups classes to provide application-specific functionality. At Indeed, we log additional information about the request as part of the groups string. For example, if a user is logged in, we'd append loggedin to the group string. This simplifies down-stream analysis by making these synthetic groups (labels) available to all of our tools that supported group-based analysis.

Adding information to the groups string can be done by overriding appendTestGroups method and isEmpty methods.

/**
 * Return a value indicating if the groups are empty 
 * and should be represented by an empty-string
 */
public boolean isEmpty() { ... }
/**
  * Appends each group to the StringBuilder using the separator to delimit
  * group names. the separator should be appended for each group added
  * to the string builder
  */
public void appendTestGroups(final StringBuilder sb, char separator) { ... }

forcing groups

Groups can be forced by using a prforceGroups url parameter. It's value should be a comma-delimited list of group strings identical to that used in the Groups.toString() method documented above.

eg. forcing the altcolor2 bucket (value 1) for the bgcolortst in the example groups

http://www.example.com/path?prforceGroups=bgcolortst1
  • When forcing groups, a test's eligibility rules and allocation rules are ignored.
  • Groups that were not forced will continue to be allocated.
  • An X-PRFORCEGROUPS request header can be used instead of the url parameter.
  • a cookie set on the fully-qualified domain, path="/" will be set containing your forced groups. manually delete this cookie or reset it with another prforceGroups parameter.

viewing the proctor state

The proctor loader provides some state via VarExport variables.

Proctor comes with several other servlets that can be used to display the proctor state. Typically these pages are restricted to internal-developers either at the application or apache level.

AbstractShowTestGroupsController: Spring controller that provides three routes:

url page
/showGroups displays the groups for the current request
/showRandomGroups displays a condensed version of the current test matrix
/showTestMatrix displays a JSON test matrix containing only those tests in the application's specification
// Routes the controller to /proctor
// Valid URLS: /proctor/showGroups, /proctor/showRandomGroups, /proctor/showTestMatrix
@RequestMapping(value = "/proctor", method = RequestMethod.GET)
public ExampleShowTestGroupsController extends AbstractShowTestGroupsController {

  @Autowired
  public ExampleShowTestGroupsController(final AbstractProctorLoader loader) {
    super(loader);
  }

  boolean isAccessAllowed(final HttpServletRequest request) {
    // Application restriction on who can see these internal pages
    return true;
  }
}

ViewProctorSpecificationServlet: servlet used to display the application's specification

web.xml:

<servlet>
    <servlet-name>ViewProctorSpecificationServlet</servlet-name>
    <servlet-class>com.indeed.proctor.consumer.ViewProctorSpecificationServlet</servlet-class>
    <init-param>
        <param-name>proctorSpecPath</param-name>
        <param-value>classpath:/org/example/proctor/ExampleGroups.json</param-value>
    </init-param>
</servlet>

<servlet-mapping>
    <servlet-name>ViewProctorSpecificationServlet</servlet-name>
    <url-pattern>/proctor/specification</url-pattern>
</servlet-mapping>