Using Groups

This page expands on the Proctor quick start guide and provides guidance about using each Proctor component. You should be familiar with the test specification, code generator and loader concepts.

Setting Up Groups

• Create a test specification: org/example/proctor/ExampleGroups.json
• ExampleGroups and ExampleGroupsManager generated by the Java codegenerator
• ExampleGroups.js generated by the JavaScript codegenerator
• test-matrix compiled by the Proctor builder

• Create an instance of the AbstractJsonProctorLoader, schedule it to refresh every 30 seconds, and create an instance of ExampleGroupsManager using the following loader:

     final JsonProctorLoaderFactory factory = new JsonProctorLoaderFactory();
   // Loads the specification from the classpath resource
   factory.setSpecificationResource("classpath:/org/example/proctor/ExampleGroups.json");
   // Loads the test matrix from a file
   factory.setFilePath("/var/local/proctor/test-matrix.json");
   final AbstractJsonProctorLoader loader = factory.getLoader();
   // schedule the loader to refresh every 30 seconds
   final ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
   scheduledExecutorService.scheduleWithFixedDelay(loader, 0, 30, TimeUnit.SECONDS);
   // Create groups manager for application-code usage
   final ExampleGroupsManager exampleGroupsManager = new ExampleGroupsManager(loader);  

How to Determine Groups

When initializing an application’s AbstractGroups class, such as ExampleGroups, a best practice is to use the generated AbstractGroupsManager API, such as ExampleGroupsManager, instead of using Proctor directly.

   public ProctorResult determineBuckets(Identifiers identifiers
                                      [,context-variable_1 ... ,context-variable_N]) { ... }
   public ProctorResult determineBuckets(HttpServletRequest request,
                                      HttpServletResponse response,
                                      boolean allowForceGroups,
                                      Identifiers identifiers
                                      [,context-variable_1 ... ,context-variable_N]) { ... }

The generated AbstractGroupsManager handles several things:

• Uses the most recently loaded test-matrix provided by the AbstractProctorLoader implementation.
• Returns a default ProctorResult if AbstractProctorLoader returns a null Proctor instance.
• Maps application context variables to their variable names as defined in the specification context.
• Overrides groups specified via a url-parameter or cookie-value for internal testing (if allowForceGroups is true).

Where to Determine Groups

Most applications should initialize their applications in a single location in code to ensure usage consistency across groups. For web applications, this usually means placing the group initialization early in the request-response lifecycle so groups are across requests. The group determination must be after all context variables and test identifiers can be resolved for a given request. Your application’s values for this information will impact where the group determination can happen.

Indeed uses the ACCOUNT identifier for loggedin-account-based tests and language and country context parameters for locale-specific tests. Because of this, the groups determination must come after the code that identifies a user based on their cookies and determines the language for this request.

In practice, this occurs in a javax.servlet.Filter or Spring org.springframework.web.servlet.HandlerInterceptor. Other web frameworks have their own injection points to pre-process requests.

Example Spring interceptor for determining groups:

NOTE: Lines 39-41 catch all exceptions including those thrown when evaluating the javax.el rule expressions. Indeed logs these exceptions and uses empty-groups instead of erroring during the request.

Using the Generated Code

Java

The groups generated code contains convenience methods for accessing the each test’s group and checking if a given group is active.

The ExampleGroups.json specification will generate the following ExampleGroups.java class (the method implementations have been removed for brevity):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
public class ExampleGroups extends AbstractGroups {
    public ExampleGroups(final ProctorResult proctorResult) {
        super(proctorResult);
    }
    public enum Test {
        BGCOLORTST("bgcolortst");
    }
    public enum Bgcolortst implements Bucket<Test> {
        INACTIVE(-1, "inactive"),
        ALTCOLOR1(0, "altcolor1"),
        ALTCOLOR2(1, "altcolor2"),
        ALTCOLOR3(2, "altcolor3"),
        ALTCOLOR4(3, "altcolor4");
    }

    // test bucket accessor
    public Bgcolortst getBgcolortst() { }
    public int getBgcolortstValue(final int defaultValue) { }

    public boolean isBgcolortstInactive() { }
    public boolean isBgcolortstAltcolor1() { }
    public boolean isBgcolortstAltcolor2() { }
    public boolean isBgcolortstAltcolor3() { }
    public boolean isBgcolortstAltcolor4() { }

    // Payload accessors
    public @Nullable String getBgcolortstPayload() { }
    public @Nullable String getBgcolortstPayloadForBucket(final Bgcolortst targetBucket) { }
}

Unassigned -> Fallback

Previous releases of the proctor library allowed users to be placed in an unassigned bucket that differed from the fallback bucket. This is no longer the case. The generated code will now consider unassigned users to be a part of the fallback bucket declared in the project specification.

Sometimes inactive != control

For many tests the fallback bucket displays the same behavior as the control group. This is typical if you want to place your users into equal-sized buckets but do not want to set the test-behavior to 50%.

For example, you’re showing an experimental design to 10% of your users. The remaining 90% will continue to see the status-quo (control). It’s common to allocate three buckets: inactive (80%), control (10%), test (10%), and mark the inactive bucket as the fallback. Users in the inactive bucket will experience the same behavior as users in the control. Although 90% of users will experience the status-quo design, only 10% will be labeled as control. This makes absolute comparisons across metrics easier because the control and test groups sizes are of equal size.

When inactive and control users experience the same behavior, the following code correctly displays the new feature to the test group.

if(groups.isFeatureTest()) {
  // do test
} else {
  // fall through for "control" and "inactive"
}

However, there are situations where control != inactive.

For example, you might decide to integrate with google-analytics help to track your tests by setting a custom variable. Users in the control and test buckets should include a unique GA variable value so their traffic can be segmented in google-analytics. The inactive users will not get a custom variable. Your application code should explicitly handle the test and control buckets separate from the inactive users.

if(groups.isFeatureTest()) {
  // do test
} else if(groups.isFeatureControl()) {
  // do control
} else {
  // fall through for "inactive"
}

In general:

• Program towards the test behavior, not the control behavior.
• Create an inactive bucket rather than assigning control as the fallback bucket.

map payload returns

Using a map payload would instead return

   public ExampleGroupsPayload.Bgcolortst getBgcolortstPayload() { }

Where ExampleGroupsPayload is another generated class, and to retrieve variable values make a call similar to

   final String groupVarOne = exampleGroupsInstance.getBgcolortstPayload().getVariableOne();
   final Double[] groupVarTwo = exampleGroupsInstance.getBgcolortstPayload().getVariableTwo(); 
   // returns expected variable type
   // VariableOne and VariableTwo are the names of your variables inside the map payload, capitalized first character 

json payload returns

json payloads provide an additional method to attempt retrieving the payload as the provided class type inputted as the parameter, if the method fails to cast the json to the provided class it returns null:

public @Nullable <T> T getExampleGroupsPayload(final Class<T> payloadType);

JavaScript

After you use your generated Java code to determine buckets for each of your test groups, you can pass the allocations to your generated JavaScript.

A method in AbstractGroups will create an array of group allocations and payload values. Pass it an array of Test enums from Test.values().

public <E extends Test> List<List<Object>> getJavaScriptConfig(final E[] tests);

You can serialize this result to JSON or write it out to the page in JavaScript. Then init() your groups and use them as needed.

    var proctor = require('com.indeed.example.groups');
    proctor.init(values);

    ...

    if (proctor.getGroups().isFeatureTest()) {
        //do test
    }else {
        //fall through for "control" or "inactive"
    }

How to Record Group Assignment

The AbstractGroups interface provides a toString() method that contains a comma-delimited list of active groups. Only tests with non-negative bucket value will be logged. This follows the convention that the inactive fallback bucket has a value of -1 and doesn’t contain test behavior. The string representation for each group is [test_name][bucket_value].

Consider the following specification:

{
    "tests" : {
        // Test different single-page implementations (none, html5 history, hash-bang )
        "historytst": { "buckets" : {
          "inactive": -1, "nohistory":0, "html5":1, "hashbang":2
        }, "fallbackValue" : -1 },
        // test different signup flows
        "signuptst": { "buckets" : {
          "inactive": -1, "singlepage":0, "accordian":1, "multipage":2, "tabs":3
        }, "fallbackValue" : -1 }
    }
}

Your application must decide how to log these groups and analyze the outcomes. Indeed logs all the groups on every requests and makes each test available in all analyses. In at least one situation, this has helped us identify and explain unexpected user behavior on pages not directly impacted by a test group’s behavior.

Extending Groups

Feel free to extend the generated Groups classes to provide application-specific functionality. Indeed logs additional information about the request as part of the groups string. For example, if a user is logged in, we’d append loggedin to the group string. This simplifies downstream analysis by making these synthetic groups (labels) available to all of our tools that support group-based analysis.

You can add information to the groups string by overriding appendTestGroups method and isEmpty methods.

/**
 * Return a value indicating if the groups are empty
 * and should be represented by an empty string
 */
public boolean isEmpty() { ... }
/**
  * Appends each group to the StringBuilder using the separator to delimit
  * group names. the separator should be appended for each group added
  * to the string builder
  */
public void appendTestGroups(final StringBuilder sb, char separator) { ... }

Forcing Groups

Groups can be forced by using a prforceGroups url parameter. Its value should be a comma-delimited list of group strings identical to the list used in the Groups.toString() method documented above. The following example forces the altcolor2 bucket (value 1) for the bgcolortst in the example groups:

http://www.example.com/path?prforceGroups=bgcolortst1

• When forcing groups, a test’s eligibility rules and allocation rules are ignored.
• Groups that are not forced will continue to be allocated.
• You can use the X-PRFORCEGROUPS request header instead of the url parameter.
• A cookie set on the fully-qualified domain, path=”/” will be set containing your forced groups. Manually delete this cookie or reset it with another prforceGroups parameter.

Viewing the Proctor State

The Proctor loader provides some state via VarExport variables.

Proctor comes with several other servlets that can be used to display the Proctor state. Typically these pages are restricted to internal developers either at the application or Apache level.

AbstractShowTestGroupsController: Spring controller that provides three routes:

// Routes the controller to /proctor
// Valid URLS: /proctor/showGroups, /proctor/showRandomGroups, /proctor/showTestMatrix
@RequestMapping(value = "/proctor", method = RequestMethod.GET)
public ExampleShowTestGroupsController extends AbstractShowTestGroupsController {

  @Autowired
  public ExampleShowTestGroupsController(final AbstractProctorLoader loader) {
    super(loader);
  }

  boolean isAccessAllowed(final HttpServletRequest request) {
    // Application restriction on who can see these internal pages
    return true;
  }
}

ViewProctorSpecificationServlet: Servlet used to display the application’s specification

web.xml:

<servlet>
    <servlet-name>ViewProctorSpecificationServlet</servlet-name>
    <servlet-class>com.indeed.proctor.consumer.ViewProctorSpecificationServlet</servlet-class>
    <init-param>
        <param-name>proctorSpecPath</param-name>
        <param-value>classpath:/org/example/proctor/ExampleGroups.json</param-value>
    </init-param>
</servlet>

<servlet-mapping>
    <servlet-name>ViewProctorSpecificationServlet</servlet-name>
    <url-pattern>/proctor/specification</url-pattern>
</servlet-mapping>