test matrix has two on-disk representations:
- A compiled single-file
- Uncompiled file per test-definition
The single-file variant is compiled using the Proctor builder and loaded by applications via the Proctor loader implementations. The single-file schema contains information about when the file was generated and which version it represents.
Single-file JSON Schema
||Collection of test-definitions. A map of
||Metadata about the test matrix.|
||Version number for the test matrix. At Indeed, the test-definitions are stored in SVN and the
||String indicating who was responsible for compiling the matrix. At Indeed, the builder is initiated by Jenkins and the
||Timestamp of the generated file.|
File per test-definition
In its uncompiled format, the test-matrix is stored as a single file per test-definition. In the test-definitions directory, each test must have a corresponding directory and definition.json file. The directory name dictates the test name and the definition.json file describes the buckets, allocations, and other attributes of the test definition.
. ├── test-definitions | ├── appfeatureatst | ├── definition.json | ├── appfeaturebtst | ├── definition.json | ├── otherappfeaturetst | ├── definition.json
The above example would describe a test matrix with three tests:
appfeaturetst, appserptst, otherappfeaturetst
Test Naming Conventions
If you’re using Proctor across multiple applications, it’s advisable to use standard test naming conventions to help differentiate the tests you use for each application.
Indeed has used a convention similar to this example:
application-name: A short name you use for your application, such as a product codename or JIRA project.
feature-name: Description of the feature impacted by the test.