Testplane Events

Overview

How event descriptions are organized

Below are described all the Testplane events you can subscribe to in your plugin.

Each event description begins with tags, which come in the following variants:

sync or async denote, respectively, synchronous and asynchronous event handler call modes;
master indicates that this event is available from the master process of the testplane;
worker indicates that this event is available from the workers (subprocesses) of the testplane;
can be intercepted indicates that this event can be intercepted and accordingly, modified.

Next are:

a description of the circumstances under which the event is triggered;
a code snippet showing how to subscribe to the event;
the event handler parameters;
and optionally, examples of using this event in a plugin or plugins.

Event generation scheme

Testplane event generation

Event sequence description

Testplane can be launched through both the CLI (command line interface) and its API: from a script using the run command.

After launching, Testplane loads all plugins and proceeds to parsing the CLI, if it was launched through the CLI, or directly to the initialization stage if it was launched through its API.

CLI Parsing

If Testplane was launched through the CLI, it triggers the CLI event. Any plugin can subscribe to this event to add its options and commands to the Testplane before Testplane parses the CLI.

If Testplane was launched via the API, the CLI parsing stage will be skipped and it will move straight to the initialization stage.

Initialization

During initialization, Testplane triggers the INIT event. This event occurs only once for the entire launch of testplane. Plugins can use this event to perform all necessary preparatory work: open and read files, set up a dev server, initialize data structures, etc.

Testplane then launches subprocesses (known as workers) in which all tests will be executed. Tests are not executed in the master process of testplane; the master process handles the overall orchestration of the test run process, including generating events upon the completion of individual tests.

The number of workers that Testplane launches is regulated by the workers parameter in the system section of the Testplane config. When a new worker is launched, Testplane triggers the NEW_WORKER_PROCESS event.

info

Testplane runs all tests in workers to avoid memory and CPU limitations on the master process. Once the number of tests completed in a worker reaches the value of testsPerWorker, the worker stops, and a new worker is launched. Accordingly, the NEW_WORKER_PROCESS event is triggered again.

Test reading

Afterwards, Testplane reads all tests from the file system in the master process. Before reading each file, it sends the BEFORE_FILE_READ event, and after reading it — the AFTER_FILE_READ event.

After all tests are read, the AFTER_TESTS_READ event is triggered.

Test execution

Then Testplane sends the RUNNER_START and BEGIN events, and starts a new session (a browser session) in which the tests will be run. At the start of the session, Testplane triggers the SESSION_START event.

info

If the number of tests executed within one session reaches the value of the testsPerSession parameter, Testplane will end the session, triggering the SESSION_END event, and start a new one, sending the SESSION_START event.

If a test fails with a critical error, Testplane will:

prematurely terminate the session and the browser associated with it;
create a new session;
request a new browser and link it to the new session.

This is to ensure that a failure in a session during one test run does not affect the subsequent test runs.

After creating a new session, Testplane proceeds to run the tests. All tests are executed in workers, but the actual starting and collecting of the test results is done in the master process. The master process triggers the SUITE_BEGIN event for describe-blocks in the test file and TEST_BEGIN for it-blocks. If a test is disabled using helpers like skip.in and others, the TEST_PENDING event is triggered.

Next, the workers receive from the master process the information about specific tests they need to run. Since tests are stored in files, workers read specifically those files where the required tests are located. Before reading each of those files in each worker, the BEFORE_FILE_READ event is triggered, and after reading — the AFTER_FILE_READ event.

Once the relevant test files are read by the worker, the AFTER_TESTS_READ event is triggered.

The listed 3 events — BEFORE_FILE_READ, AFTER_FILE_READ, and AFTER_TESTS_READ will be triggered in the workers during the test runs every time workers receive new tests to run from the master process. Except when the corresponding test file has already been read by the worker before. After the first reading of any file, the worker caches it to avoid re-reading the test file next time.

Why may a file be requested multiple times?

Because one file may contain many tests. The test run is done per individual test, not per file. Therefore, at some point in time, a test from a file that already had another test run may be executed. In such cases, caching protects from unnecessary re-reading of the same files.

Before a test is run, the NEW_BROWSER event is triggered. However, this event will not be triggered for all tests, as the same browser can be used many times to launch tests (see the sessionsPerBrowser parameter). Also, in case of a critical test failure, the browser is recreated to prevent other tests in that browser from failing due to a system crash. In this case, the NEW_BROWSER event will be sent again.

Test completion

After a test completes, the SESSION_END event can be sent. But this is only if the total number of tests run in the session exceeds the value of the testsPerSession parameter.

Then everything will depend on the result of the test run. If the test passed successfully, Testplane triggers the TEST_PASS event. If the test failed — TEST_FAIL. If the test failed but needs to be re-run (see the retry and shouldRetry settings in the Testplane config), instead of the TEST_FAIL event, the RETRY event will be sent.

If the test does not need to be re-run, and the result is final, Testplane triggers the TEST_END and SUITE_END events if it refers to the completion of a describe-block.

After all tests have been executed and sessions completed, Testplane triggers the END and RUNNER_END events.

Updating reference screenshots

During the test run, reference screenshots may be updated for the following reasons:

The developer launched Testplane in a special GUI mode and commanded "accept screenshots";
The developer specified the --update-ref option when launching testplane;
The tests did not have reference screenshots.

In all these cases, the UPDATE_REFERENCE event is triggered.

Errors and premature termination

If during the test run, a critical error occurs in one of the event interceptors, Testplane triggers the ERROR event for this test. However, the other tests will proceed normally.

If during the test run, Testplane receives a SIGTERM signal (for example, as a result of pressing Ctrl + C), Testplane triggers the EXIT event and prematurely terminates the test run.

About event interception

Testplane allows the developer to intercept certain events and modify them to other events, ignore them, or delay their processing.

Events that can be intercepted are tagged with can be intercepted in their description. There are a total of 7 such events:

Changing one event to another

For example, the code below shows how to intercept the TEST_FAIL event and change it to the TEST_PENDING event — thus automatically disabling failing tests, preventing them from bringing down the overall test run:

module.exports = testplane => {
    testplane.intercept(testplane.events.TEST_FAIL, ({ event, data }) => {
        const test = Object.create(data);
        test.pending = true;
        test.skipReason = "intercepted failure";

        return { event: testplane.events.TEST_PENDING, data: test };
    });

    testplane.on(testplane.events.TEST_FAIL, test => {
        // this handler will never be called
    });

    testplane.on(testplane.events.TEST_PENDING, test => {
        // this handler will always be called instead of the TEST_FAIL handler
    });
};

Leaving the event as is

If for some reason the intercepted event needs to be left as is, its handler should return exactly the same object or any falsey value: undefined, null or false.

module.exports = testplane => {
    testplane.intercept(testplane.events.TEST_FAIL, ({ event, data }) => {
        return { event, data };
        // or return;
        // or return null;
        // or return false;
    });

    testplane.on(testplane.events.TEST_FAIL, test => {
        // this handler will be called as usual,
        // because the interception of the TEST_FAIL event does not change it
    });
};

Ignoring an event

To ignore an event and prevent it from propagating further, return an empty object from the handler (where the event is intercepted):

module.exports = testplane => {
    testplane.intercept(testplane.events.TEST_FAIL, ({ event, data }) => {
        return {};
    });

    testplane.on(testplane.events.TEST_FAIL, test => {
        // this handler will never be called
    });
};

Delaying event processing

The approach above with ignoring an event can be used to delay the occurrence of certain events, for example:

module.exports = testplane => {
    const intercepted = [];

    testplane.intercept(testplane.events.TEST_FAIL, ({ event, data }) => {
        // collect all TEST_FAIL events
        intercepted.push({ event, data });
        // and prevent them from propagating further
        return {};
    });

    testplane.on(testplane.events.END, () => {
        // after the test run is complete, trigger all accumulated TEST_FAIL events
        intercepted.forEach(({ event, data }) => testplane.emit(event, data));
    });
};

Information sharing between event handlers

Events triggered in the master process and in the Testplane workers cannot exchange information through global variables.

For example, this approach will not work:

module.exports = testplane => {
    let flag = false;

    testplane.on(testplane.events.RUNNER_START, () => {
        flag = true;
    });

    testplane.on(testplane.events.NEW_BROWSER, () => {
        // the value false will be output, because the NEW_BROWSER event
        // is triggered in the Testplane worker, and RUNNER_START – in the master process
        console.info(flag);
    });

    testplane.on(testplane.events.RUNNER_END, () => {
        // the value true will be output
        console.info(flag);
    });
};

But the problem can be solved as follows:

module.exports = (testplane, opts) => {
    testplane.on(testplane.events.RUNNER_START, () => {
        opts.flag = true;
    });

    testplane.on(testplane.events.NEW_BROWSER, () => {
        // the value true will be output because properties in the config
        // that are primitives (and the "opts" variable is part of the config)
        // are automatically passed to workers during the RUNNER_START event
        console.info(opts.flag);
    });
};

Or as follows: see the example from the description of the NEW_WORKER_PROCESS event.

Parallel execution of plugin code

The test runner has a registerWorkers method which registers the plugin code for parallel execution in the Testplane workers. The method takes the following parameters:

Parameter	Type	Description
workerFilepath	String	Absolute path to the worker.
exportedMethods	String[]	List of exported methods.

It returns an object which contains asynchronous functions with names from exported methods.

The file at the workerFilepath path should export an object containing asynchronous functions with names from exportedMethods.

Example of parallel plugin code execution

Plugin code: plugin.js

let workers;

module.exports = testplane => {
    testplane.on(testplane.events.RUNNER_START, async runner => {
        const workerFilepath = require.resolve("./worker");
        const exportedMethods = ["foo"];

        workers = runner.registerWorkers(workerFilepath, exportedMethods);

        // prints FOO_RUNNER_START
        console.info(await workers.foo("RUNNER_START"));
    });

    testplane.on(testplane.events.RUNNER_END, async () => {
        // prints FOO_RUNNER_END
        console.info(await workers.foo("RUNNER_END"));
    });
};

Worker code: worker.js

module.exports = {
    foo: async function (event) {
        return "FOO_" + event;
    },
};

CLI

sync | master

The CLI event is triggered immediately upon startup, before Testplane parses the CLI. The event handler executes synchronously. With it, you can add new commands or extend testplane's help.

Subscribing to the event

testplane.on(testplane.events.CLI, cli => {
    console.info("Processing CLI event...");

    cli.option(
        "--some-option <some-value>",
        "the full description of the option",
        // see more at https://github.com/tj/commander.js#options
    );
});

Handler parameters

An object of the Commander type is passed to the event handler.

Example of usage

Let's consider an example of the implementation of the @testplane/test-repeater plugin.

Using the CLI event, the plugin adds a new --repeat option to testplane. With it, you can specify how many times to run tests, regardless of the result of each run.

Click to see the code

const parseConfig = require("./config");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled || testplane.isWorker()) {
        // the plugin is either disabled, or we are in the context of a worker – leave
        return;
    }

    testplane.on(testplane.events.CLI, cli => {
        // add the --repeat option
        cli.option(
            "--repeat <number>",
            "how many times tests should be repeated regardless of the result",
            value => parseNonNegativeInteger(value, "repeat"),
        );
    });

    // ...
};

INIT

async | master

The INIT event is triggered before the run or readTests tasks are performed. The event handler can be asynchronous: in which case, the tasks will only start after the Promise returned by the event handler is resolved. The event triggers only once, regardless of how many times the tasks are performed.

Subscribing to the event

testplane.on(testplane.events.INIT, async () => {
    console.info("Processing INIT event...");
});

Handler parameters

No data is passed to the event handler.

Example of usage

In the INIT event handler, you can organize, for example, the launch of a dev server for your project.

What is a dev server?

A dev server is an express-like application that allows you to develop the frontend of the project.

Below is the shortest implementation. A more detailed example can be found in the section "Automatic dev server startup".

Click to see the code

const http = require('http');
const parseConfig = require('./config');

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled</td></tr> testplane.isWorker()) {
        // either the plugin is disabled, or we are in the worker context – exit
        return;
    }

    // ...

    testplane.on(testplane.events.INIT, () => {
        // content served by the dev-server
        const content = '<h1>Hello, World!</h1>';

        // create a server and start listening on port 3000
        http
            .createServer((req, res) => res.end(content))
            .listen(3000);

        // at http://localhost:3000/index.html it will serve: <h1>Hello, World!</h1>
    });
};

BEFORE_FILE_READ

sync | master | worker

The BEFORE_FILE_READ event is triggered before the test file is read to parse it. The event handler is executed synchronously. The event is also available in Testplane workers.

Subscribing to the event

testplane.on(testplane.events.BEFORE_FILE_READ, ({ file, testplane, testParser }) => {
    testParser.setController("<some-command-name>", {
        "<some-helper>": function (matcher) {
            // ...
        },
    });
});

Handler parameters

The event handler receives an object with the following format:

{
    file, // String: path to the test file
        testplane, // Object: same as global.testplane
        testParser; // Object: of type TestParserAPI
}

testParser: TestParserAPI

The testParser object of type TestParserAPI is passed to the BEFORE_FILE_READ event handler. It allows you to manage the process of parsing test files. The object supports the setController method, which allows you to create your own helpers for tests.

setController(name, methods)

The method adds a controller to the global testplane object, available inside the tests.

name — the name of the helper (or controller);
methods — an object-dictionary where the keys define the names of the corresponding helper methods, and the values define their implementation. Each method will be called on the corresponding test or test suite.

info

The controller will be removed as soon as the current file parsing is finished.

Usage example

As an example, let's create a special helper testplane.logger.log() that allows us to log information about the parsing of the test we are interested in.

Click to view the usage example

Plugin code

testplane.on(testplane.events.BEFORE_FILE_READ, ({ file, testParser }) => {
    testParser.setController("logger", {
        log: function (prefix) {
            console.log(
                `${prefix}: just parsed ${this.fullTitle()} from file ${file} for browser ${this.browserId}`,
            );
        },
    });
});

Test code

describe("foo", () => {
    testplane.logger.log("some-prefix");
    it("bar", function () {
        // ...
    });
});

Another example of using the BEFORE_FILE_READ event can be found in the section “Running tests with helpers”.

AFTER_FILE_READ

sync | master | worker

The AFTER_FILE_READ event is triggered after the test file is read. The event handler is executed synchronously. The event is also available in Testplane workers.

Subscribing to the event

testplane.on(testplane.events.AFTER_FILE_READ, ({ file, testplane }) => {
    console.info("Processing AFTER_FILE_READ event...");
});

Handler parameters

The event handler receives an object with the following format:

{
    file, // String: path to the test file
        testplane; // Object: same as global.testplane
}

AFTER_TESTS_READ

sync | master | worker

The AFTER_TESTS_READ event is triggered after the methods readTests or run of the TestCollection object are called. The event handler is executed synchronously. The event is also available in Testplane workers.

By subscribing to this event, you can perform various manipulations on the test collection before they are run. For example, you can exclude certain tests from the runs.

Subscribing to the event

testplane.on(testplane.events.AFTER_TESTS_READ, testCollection => {
    console.info("Processing AFTER_TESTS_READ event...");
});

Handler parameters

The event handler receives a testCollection object of type TestCollection.

Usage example

Consider the implementation of the testplane-global-hook plugin, which allows you to move actions that repeat before and after each test into separate beforeEach and afterEach handlers.

Using the AFTER_TESTS_READ event, the plugin adds beforeEach and afterEach hooks to each root suite. These hooks are defined by the user in the testplane-global-hook plugin configuration.

Click to view the code

const parseConfig = require("./config");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    // ...

    const { beforeEach, afterEach } = pluginConfig;

    testplane.on(testplane.events.AFTER_TESTS_READ, testCollection => {
        testCollection.eachRootSuite(root => {
            beforeEach && root.beforeEach(beforeEach);
            afterEach && root.afterEach(afterEach);
        });
    });
};

More examples of using the AFTER_TESTS_READ event can be found in the sections “Running tests from a specific list” and “Running tests with helpers”.

RUNNER_START

async | master

The RUNNER_START event is triggered after all Testplane workers are initialized and before the tests are run. The event handler can be asynchronous: in this case, the tests will start running only after the Promise returned by the event handler is resolved.

Subscribing to the event

testplane.on(testplane.events.RUNNER_START, async runner => {
    console.info("Processing RUNNER_START event...");
});

Handler parameters

The event handler receives a reference to the runner instance. Using this instance, you can trigger various events or subscribe to them.

Usage example

Suppose we want to automatically set up an SSH tunnel when running tests and redirect all URLs in the tests to the established tunnel. To do this, we can use the RUNNER_START and RUNNER_END events to open the tunnel when the runner starts and close it after the runner finishes.

Click to view the code

const parseConfig = require("./config");
const Tunnel = require("./lib/tunnel");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    // plugin config defines tunnel parameters:
    // host, ports, localport, retries, etc.
    const tunnel = Tunnel.create(testplane.config, pluginConfig);

    testplane.on(testplane.events.RUNNER_START, () => tunnel.open());
    testplane.on(testplane.events.RUNNER_END, () => tunnel.close());
};

A similar implementation can be found in the ssh-tunneler plugin.

RUNNER_END

async | master

The RUNNER_END event is triggered after the test is run and before all workers are terminated. The event handler can be asynchronous: in this case, all workers will be terminated only after the Promise returned by the event handler is resolved.

Subscribing to the event

testplane.on(testplane.events.RUNNER_END, async result => {
    console.info("Processing RUNNER_END event...");
});

Handler parameters

The event handler receives an object with the test run statistics in the following format:

{
    passed: 0,  // number of successfully completed tests
    failed: 0,  // number of failed tests
    retries: 0, // number of test retries
    skipped: 0, // number of skipped tests
    total: 0    // total number of tests
};

Usage example

See the example above about opening and closing the tunnel when the runner starts and stops.

NEW_WORKER_PROCESS

sync | master

The NEW_WORKER_PROCESS event is triggered after a new Testplane subprocess (worker) is spawned. The event handler is executed synchronously.

Subscribing to the event

testplane.on(testplane.events.NEW_WORKER_PROCESS, workerProcess => {
    console.info("Processing NEW_WORKER_PROCESS event...");
});

Handler parameters

The event handler receives a wrapper object over the spawned subprocess, with a single send method for message exchange.

Usage example

The example below shows how to use the NEW_WORKER_PROCESS event to organize interaction between the master process and all Testplane workers. For example, to update the value of a parameter in all Testplane workers from the master process before the test run starts.

The example also uses the BEGIN event.

Click to view the code

const masterPlugin = (testplane, opts) => {
    const workers = [];

    testplane.on(testplane.events.NEW_WORKER_PROCESS, (workerProcess) => {
        // store references to all created Testplane workers
        workers.push(workerProcess);
    });

    testplane.on(testplane.events.BEGIN, () => {
        // send the parameter value to all workers
        workers.forEach((worker) => {
            worker.send({
                type: PARAM_VALUE_UPDATED_EVENT,
                param: 'some-value'
            });
        });
    });
};

const workerPlugin = (testplane) => {
    process.on('message', ({ type, ...data }) => {
        if (type === PARAM_VALUE_UPDATED_EVENT) {
            const { param } = data;
            console.info(`Received value "${param}" for "param" from the master process`);
        }
    });

    ...
};

const plugin = (testplane, opts) => {
    if (testplane.isWorker()) {
        workerPlugin(testplane, opts);
    } else {
        masterPlugin(testplane, opts);
    }
};

module.exports = plugin;

SESSION_START

async | master

The SESSION_START event is triggered after the browser session is initialized. The event handler can be asynchronous: in this case, the tests will start running only after the Promise returned by the event handler is resolved.

Subscribing to the event

testplane.on(testplane.events.SESSION_START, async (browser, { browserId, sessionId }) => {
    console.info("Processing SESSION_START event...");
});

Handler parameters

The event handler receives 2 arguments:

the first argument — an instance of WebdriverIO;
the second argument — an object of the form { browserId, sessionId }, where browserId is the name of the browser, and sessionId is the browser session identifier.

Usage example

Consider an example where the plugin subscribes to the SESSION_START event to disable scrollbars in browsers using the Chrome DevTools Protocol.

Click to view the code

const parseConfig = require("./config");
const DevTools = require("./dev-tools");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    testplane.on(testplane.events.SESSION_START, async (browser, { browserId, sessionId }) => {
        if (!pluginConfig.browsers.includes(browserId)) {
            // the browser is not in the list of browsers for which scrollbars can be disabled
            // using the Chrome DevTools Protocol (CDP) – exit
            return;
        }

        const gridUrl = testplane.config.forBrowser(browserId).gridUrl;

        // pluginConfig.browserWSEndpoint defines a function that should return the URL
        // for working with the browser via CDP. To allow the function to compute the URL,
        // the function receives the session identifier and the grid URL
        const browserWSEndpoint = pluginConfig.browserWSEndpoint({ sessionId, gridUrl });

        const devtools = await DevTools.create({ browserWSEndpoint });

        devtools.setScrollbarsHiddenOnNewPage();

        await devtools.hideScrollbarsOnActivePages();
    });
};

A more detailed implementation can be found in the hermione-hide-scrollbars plugin.

SESSION_END

async | master

The SESSION_END event is triggered after the browser session ends. The event handler can be asynchronous: in this case, the tests will continue running only after the Promise returned by the event handler is resolved.

Subscribing to the event

testplane.on(testplane.events.SESSION_END, async (browser, { browserId, sessionId }) => {
    console.info("Processing SESSION_END event...");
});

Handler parameters

The event handler receives 2 arguments:

the first argument — an instance of WebdriverIO;
the second argument — an object of the form { browserId, sessionId }, where browserId is the name of the browser, and sessionId is the browser session identifier.

BEGIN

sync | master

The BEGIN event is triggered before the test is run, but after all runners are initialized. The event handler is executed synchronously.

Subscribing to the event

testplane.on(testplane.events.BEGIN, () => {
    console.info("Processing BEGIN event...");
});

Handler parameters

No data is passed to the event handler.

Usage example

See the example above about organizing interaction between the Testplane master process and all workers.

END

sync | master

The END event is triggered right before the RUNNER_END event. The event handler is executed synchronously.

Subscribing to the event

testplane.on(testplane.events.END, () => {
    console.info("Processing END event...");
});

Handler parameters

No data is passed to the event handler.

Usage example

For an example of using the END event, see the section “Delaying event processing”.

SUITE_BEGIN

sync | master | can be intercepted

The SUITE_BEGIN event is triggered before a test suite is run. The event handler is executed synchronously.

Subscribing to the event

testplane.on(testplane.events.SUITE_BEGIN, suite => {
    console.info(`Processing SUITE_BEGIN event for "${suite.fullTitle()}"...`);
});

Handler Parameters

An instance of suite is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.SUITE_BEGIN, ({ event, data: suite }) => {
    console.info(`Intercepting SUITE_BEGIN event for "${suite.fullTitle()}"...`);
});

SUITE_END

sync | master | can be intercepted

The SUITE_END event is triggered after the test suite (suite) has finished executing. The event handler is executed synchronously.

Event Subscription

testplane.on(testplane.events.SUITE_END, suite => {
    console.info(`Handling SUITE_END event for "${suite.fullTitle()}"...`);
});

Handler Parameters

An instance of suite is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.SUITE_END, ({ event, data: suite }) => {
    console.info(`Intercepting SUITE_END event for "${suite.fullTitle()}"...`);
});

TEST_BEGIN

sync | master | can be intercepted

The TEST_BEGIN event is triggered before the test is executed. The event handler is executed synchronously.

Event Subscription

testplane.on(testplane.events.TEST_BEGIN, test => {
    if (test.pending) {
        // test is disabled, no action needed
        return;
    }

    console.info(
        `Handling TEST_BEGIN event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.TEST_BEGIN, ({ event, data: test }) => {
    console.info(
        `Intercepting TEST_BEGIN event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Usage Example

See the example "Profiling Test Runs" here.

TEST_END

sync | master | can be intercepted

The TEST_END event is triggered after the test has finished executing. The event handler is executed synchronously. The event can also be intercepted and modified in a special handler.

Event Subscription

testplane.on(testplane.events.TEST_END, test => {
    if (test.pending) {
        // test is disabled, no action needed
        return;
    }

    console.info(
        `Handling TEST_END event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.TEST_END, ({ event, data: test }) => {
    console.info(
        `Intercepting TEST_END event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Usage Example

See the example "Profiling Test Runs" here.

TEST_PASS

sync | master | can be intercepted

The TEST_PASS event is triggered if the test passes successfully. The event handler is executed synchronously. The event can also be intercepted and modified in a special handler.

Event Subscription

testplane.on(testplane.events.TEST_PASS, test => {
    console.info(
        `Handling TEST_PASS event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.TEST_PASS, ({ event, data: test }) => {
    console.info(
        `Intercepting TEST_PASS event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Usage Example

See the example "Collecting Test Run Statistics" here.

TEST_FAIL

sync | master | can be intercepted

The TEST_FAIL event is triggered if the test fails. The event handler is executed synchronously. The event can also be intercepted and modified in a special handler.

Event Subscription

testplane.on(testplane.events.TEST_FAIL, test => {
    console.info(
        `Handling TEST_FAIL event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.TEST_FAIL, ({ event, data }) => {
    console.info(
        `Intercepting TEST_FAIL event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Usage Example

See the example "Collecting Test Run Statistics" here.

TEST_PENDING

sync | master

The TEST_PENDING event is triggered if the test is disabled. The event handler is executed synchronously.

Event Subscription

testplane.on(testplane.events.TEST_PENDING, test => {
    console.info(
        `Handling TEST_PENDING event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Usage Example

See the example "Collecting Test Run Statistics" here.

RETRY

sync | master | can be intercepted

The RETRY event is triggered if the test fails but is retried. The retry capabilities are determined by the retry and shouldRetry settings in the Testplane config. Testplane plugins can also influence this by modifying these settings on the fly. See plugins retry-limiter and testplane-retry-progressive for examples.

The event handler is executed synchronously. The event can also be intercepted and modified in a special handler.

Event Subscription

testplane.on(testplane.events.RETRY, test => {
    console.info(
        `Handling RETRY event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Handler Parameters

An instance of the test is passed to the event handler.

Event Interception

testplane.intercept(testplane.events.RETRY, ({ event, data: test }) => {
    console.info(
        `Intercepting RETRY event ` +
            `for test "${test.fullTitle()}" in browser "${test.browserId}"...`,
    );
});

Usage Example

See the example "Collecting Test Run Statistics" here.

ERROR

sync | master

The ERROR event is triggered only from event interceptors in case of a critical error. The event handler is executed synchronously.

Event Subscription

testplane.on(testplane.events.ERROR, error => {
    console.info("Handling ERROR event...");
});

Handler Parameters

An error object is passed to the event handler.

Usage Example

See the example "Profiling Test Runs" here.

INFO

reserved

Reserved.

WARNING

reserved

Reserved.

EXIT

async | master

The EXIT event is triggered upon receiving the SIGTERM signal (e.g., after pressing Ctrl + C). The event handler can be asynchronous.

Event Subscription

testplane.on(testplane.events.EXIT, async () => {
    console.info("Handling EXIT event...");
});

Handler Parameters

No data is passed to the event handler.

NEW_BROWSER

sync | worker

The NEW_BROWSER event is triggered after a new browser instance is created. The event handler is executed synchronously. The event is only available in Testplane workers.

Event Subscription

testplane.on(testplane.events.NEW_BROWSER, (browser, { browserId, browserVersion }) => {
    console.info("Handling NEW_BROWSER event...");
});

Handler Parameters

Two arguments are passed to the event handler:

The first argument is an instance of WebdriverIO;
The second argument is an object of the form { browserId, versionId }, where browserId is the name of the browser, and browserVersion is the browser version.

Implementation Example

The NEW_BROWSER event is often used to add new commands to the browser or to extend existing commands. For example, some plugins add custom commands to the browser in the NEW_BROWSER event handler.

Click to view the code

module.exports = (testplane, opts) => {
    // ...

    if (testplane.isWorker()) {
        testplane.on(testplane.events.NEW_BROWSER, (browser, { browserId }) => {
            // ...

            browser.addCommand("vncUrl", vncUrlHandler);
            browser.addCommand("openVnc", createOpenVnc(browser, ipcOptions));
            browser.addCommand("waitForEnter", waitForEnter);
        });
    }
};

UPDATE_REFERENCE

sync | worker

The UPDATE_REFERENCE event is triggered after the reference screenshots are updated. The event handler is executed synchronously. The event is only available in testplane workers.

Event Subscription

testplane.on(testplane.events.UPDATE_REFERENCE, ({ state, refImg }) => {
    console.info("Handling UPDATE_REFERENCE event...");
});

Handler Parameters

An object of the following format is passed to the event handler:

{
    state, // String: the state that the screenshot reflects, e.g., plain, map-view, scroll-left, etc.
        refImg; // Object: of type { path, size: { width, height } }, describing the reference screenshot
}

The refImg.path parameter stores the path to the reference screenshot on the file system, while refImg.size.width and refImg.size.height store the width and height of the reference screenshot, respectively.

Usage Example

Consider the example of the implementation of the testplane-image-minifier plugin, which automatically compresses reference screenshots with a specified compression level when they are saved.

Click to view the code

const parseConfig = require("./config");
const Minifier = require("./minifier");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    const minifier = Minifier.create(pluginConfig);

    testplane.on(testplane.events.UPDATE_REFERENCE, ({ refImg }) => {
        minifier.minify(refImg.path);
    });
};

Event Usage Examples

Running Tests from a Specified List

Consider the example of the implementation of the testplane-test-filter plugin, which allows running only the tests specified in a JSON file.

In this example, the following testplane events and API methods are used:

Click to view the code

const _ = require("lodash");
const parseConfig = require("./config");
const utils = require("./utils");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    if (testplane.isWorker()) {
        // nothing to do in testplane workers – exit
        return;
    }

    let input;

    testplane.on(testplane.events.INIT, async () => {
        // read the file with the list of tests to run;
        // readFile returns a JSON containing an array like:
        // [
        //     { "fullTitle": "test-1", "browserId": "bro-1" },
        //     { "fullTitle": "test-2", "browserId": "bro-2" }
        // ]
        input = await utils.readFile(pluginConfig.inputFile);
    });

    testplane.on(testplane.events.AFTER_TESTS_READ, testCollection => {
        if (_.isEmpty(input)) {
            // test list is empty – run all tests,
            // i.e., do not modify the original test collection (testCollection)
            return;
        }

        // disable all tests
        testCollection.disableAll();

        // enable only those specified in the JSON file
        input.forEach(({ fullTitle, browserId }) => {
            testCollection.enableTest(fullTitle, browserId);
        });
    });
};

Collecting Test Run Statistics

Consider the example of the implementation of the json-reporter plugin.

In this example, the following testplane events are used:

Click to view the code

const Collector = require("./lib/collector");
const testplaneToolCollector = require("./lib/collector/tool/testplane");
const parseConfig = require("./lib/config");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    // collector will accumulate statistics
    const collector = Collector.create(testplaneToolCollector, pluginConfig);

    // subscribe to the relevant events
    // to ultimately obtain the necessary statistics:

    // - how many tests passed successfully
    testplane.on(testplane.events.TEST_PASS, data => collector.addSuccess(data));

    // - how many tests failed
    testplane.on(testplane.events.TEST_FAIL, data => collector.addFail(data));

    // - how many were skipped
    testplane.on(testplane.events.TEST_PENDING, data => collector.addSkipped(data));

    // - number of retries
    testplane.on(testplane.events.RETRY, data => collector.addRetry(data));

    // after the test run is complete, save the statistics to a JSON file
    testplane.on(testplane.events.RUNNER_END, () => collector.saveFile());
};

Automatic Launch of the Dev Server

We will schematically implement the testplane-dev-server plugin for testplane so that the dev server is automatically started each time testplane is launched.

Starting the dev server is optional: the plugin adds a special --dev-server option to testplane, allowing the developer to specify whether to start the dev server when launching testplane.

Additionally, the plugin allows setting the devServer parameter in its configuration.

This example uses the following testplane events:

CLI
INIT

Click to view the code

Plugin Code

const http = require("http");
const parseConfig = require("./config");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled || testplane.isWorker()) {
        // either the plugin is disabled, or we are in the worker context – exit
        return;
    }

    let program;

    testplane.on(testplane.events.CLI, cli => {
        // need to save a reference to the commander instance (https://github.com/tj/commander.js),
        // to later check for the option
        program = cli;
        // add the --dev-server option to testplane,
        // so the user can explicitly specify when to start the dev server
        cli.option("--dev-server", "run dev-server");
    });

    testplane.on(testplane.events.INIT, () => {
        // the dev server can be started either by specifying the --dev-server option
        // when launching testplane, or in the plugin settings
        const devServer = (program && program.devServer) || pluginConfig.devServer;

        if (!devServer) {
            // if the dev server doesn't need to be started – exit
            return;
        }

        // content served by the dev server
        const content = "<h1>Hello, World!</h1>";

        // create the server and start listening on port 3000
        http.createServer((req, res) => res.end(content)).listen(3000);

        // at http://localhost:3000/index.html, the content will be: <h1>Hello, World!</h1>
    });
};

Testplane Configuration

module.exports = {
    // tests will be run in a local browser,
    // see selenium-standalone in the "Quick Start" section
    gridUrl: "http://localhost:4444/wd/hub",
    // specify the path to the dev server
    baseUrl: "http://localhost:3000",

    browsers: {
        chrome: {
            desiredCapabilities: {
                browserName: "chrome",
            },
        },
    },

    plugins: {
        // add our plugin to the list of plugins
        "testplane-dev-server": {
            enabled: true,
            // by default, the dev server will not be started
            devServer: false,
        },
    },
};

Test Code

const { assert } = require("chai");

describe("example", async () => {
    it("should find hello world", async ({ browser }) => {
        // baseUrl, relative to which index.html is specified,
        // is set in the testplane configuration above
        await browser.url("index.html");

        const title = await browser.$("h1").getText();
        assert.equal(title, "Hello, World!");
    });
});

Running Tests with Helpers

Let's consider the implementation of the testplane-passive-browsers plugin.

Using the BEFORE_FILE_READ and AFTER_TESTS_READ events, the plugin allows adding a special helper that can run specified tests or test suites in given browsers. This logic can be useful if you don't need to run most tests in certain browsers. However, you might still want to run some tests in these (passive) browsers to check browser-specific things.

In the example below, we simplified the plugin code a bit by setting the helper name also directly in the code, rather than taking it from the plugin configuration.

This example uses the following testplane events:

It also uses testParser and testCollection.

Click to view the code

Plugin Code

const _ = require("lodash");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    if (testplane.isWorker()) {
        testplane.on(testplane.events.BEFORE_FILE_READ, ({ testParser }) => {
            // in workers, the helper will do nothing,
            // set it to "no operation"
            testParser.setController("also", { in: _.noop });
        });

        return;
    }

    const suitesToRun = {};
    const testsToRun = {};

    testplane.on(testplane.events.BEFORE_FILE_READ, ({ testParser }) => {
        testParser.setController("also", {
            // matcher – parameter passed to the also.in() helper;
            // can be a string, regular expression, or array of strings/regexps;
            // in our case, the matcher defines the passive browsers
            // in which the test(s) need to be run
            in: function (matcher) {
                const storage = this.suites ? suitesToRun : testsToRun;

                if (!shouldRunInBro(this.browserId, matcher)) {
                    // if the current browser is not in the list
                    // specified in the helper, do nothing
                    return;
                }

                if (!storage[this.browserId]) {
                    storage[this.browserId] = [];
                }

                // otherwise, collect the IDs of the tests
                // that should be run for the current browser
                storage[this.browserId].push({ id: this.id() });
            },
        });
    });

    // use prependListener to initially enable tests only
    // in the specified passive browsers, then all other tests
    // that should be enabled will be enabled
    testplane.prependListener(testplane.events.AFTER_TESTS_READ, testCollection => {
        // form the list of passive browsers as the intersection of browsers for tests
        // that were read, and browsers from the plugin configuration
        const passiveBrowserIds = getPassiveBrowserIds(testCollection, pluginConfig);

        passiveBrowserIds.forEach(passiveBrowserId => {
            const shouldRunTest = (runnable, storage = testsToRun) => {
                const foundRunnable =
                    runnable.id && _.find(storage[passiveBrowserId], { id: runnable.id() });

                return (
                    foundRunnable ||
                    (runnable.parent && shouldRunTest(runnable.parent, suitesToRun))
                );
            };

            // disable all tests except those that should be run
            // in the specified passive browsers
            testCollection.eachTest(browserId, test => {
                test.disabled = !shouldRunTest(test);
            });
        });
    });
};

Test Code

testplane.also.in("ie6");
describe("suite", () => {
    it("test1", function () {
        // ...
    });

    testplane.also.in(["ie7", /ie[89]/]);
    it("test2", function () {
        // ...
    });
});

Profiling Test Runs

Let's consider a schematic implementation of profiling test runs. Each time a test starts, we will record its start time, and upon completion, the end time. All information will be saved to a stream, which will be closed upon the runner's completion.

Click to view the code

const parseConfig = require("./lib/config");
const StreamWriter = require("./lib/stream-writer");

module.exports = (testplane, opts) => {
    const pluginConfig = parseConfig(opts);

    if (!pluginConfig.enabled) {
        // plugin is disabled – exit
        return;
    }

    let writeStream;

    testplane.on(testplane.events.RUNNER_START, () => {
        // create a stream for writing profiling data
        writeStream = StreamWriter.create(pluginConfig.path);
    });

    testplane.on(testplane.events.TEST_BEGIN, test => {
        if (test.pending) {
            // test is disabled – do nothing
            return;
        }

        // record the test start time
        test.timeStart = Date.now();
    });

    testplane.on(testplane.events.TEST_END, test => {
        if (test.pending) {
            // test is disabled – do nothing
            return;
        }

        // record the test end time
        test.timeEnd = Date.now();
        // and save the test information to the stream
        writeStream.write(test);
    });

    // in case of an error, close the stream
    testplane.on(testplane.events.ERROR, () => writeStream.end());

    // after the runner completes, close the stream
    testplane.on(testplane.events.RUNNER_END, () => writeStream.end());
};

A more detailed implementation can be found in the testplane-profiler plugin.

Testplane Events

Overview​

How event descriptions are organized​

Event generation scheme​

Event sequence description​

CLI Parsing​

Initialization​

Test reading​

Test execution​

Test completion​

Updating reference screenshots​

Errors and premature termination​

About event interception​

Changing one event to another​

Leaving the event as is​

Ignoring an event​

Delaying event processing​

Information sharing between event handlers​

Parallel execution of plugin code​

Example of parallel plugin code execution​

CLI​

Subscribing to the event​

Handler parameters​

Example of usage​

INIT​

Subscribing to the event​

Handler parameters​

Example of usage​

BEFORE_FILE_READ​

Subscribing to the event​

Handler parameters​

testParser: TestParserAPI​

Usage example​

AFTER_FILE_READ​

Subscribing to the event​

Handler parameters​

AFTER_TESTS_READ​

Subscribing to the event​

Handler parameters​

Usage example​

RUNNER_START​

Subscribing to the event​

Handler parameters​

Usage example​

RUNNER_END​

Subscribing to the event​

Handler parameters​

Usage example​

NEW_WORKER_PROCESS​

Subscribing to the event​

Handler parameters​

Usage example​

SESSION_START​

Subscribing to the event​

Handler parameters​

Usage example​

SESSION_END​

Subscribing to the event​

Handler parameters​

BEGIN​

Subscribing to the event​

Handler parameters​

Usage example​

END​

Subscribing to the event​

Handler parameters​

Usage example​

SUITE_BEGIN​

Subscribing to the event​

Handler Parameters​

Event Interception​

SUITE_END​

Event Subscription​

Handler Parameters​

Event Interception​

TEST_BEGIN​

Event Subscription​

Handler Parameters​

Event Interception​

Usage Example​

Overview

How event descriptions are organized

Event generation scheme

Event sequence description

CLI Parsing

Initialization

Test reading

Test execution

Test completion

Updating reference screenshots

Errors and premature termination

About event interception

Changing one event to another

Leaving the event as is

Ignoring an event

Delaying event processing

Information sharing between event handlers

Parallel execution of plugin code

Example of parallel plugin code execution

CLI

Subscribing to the event

Handler parameters

Example of usage

INIT

Subscribing to the event

Handler parameters

Example of usage

BEFORE_FILE_READ

Subscribing to the event

Handler parameters

testParser: TestParserAPI

Usage example

AFTER_FILE_READ

Subscribing to the event

Handler parameters

AFTER_TESTS_READ

Subscribing to the event

Handler parameters

Usage example

RUNNER_START

Subscribing to the event

Handler parameters

Usage example

RUNNER_END

Subscribing to the event

Handler parameters

Usage example

NEW_WORKER_PROCESS

Subscribing to the event

Handler parameters

Usage example

SESSION_START

Subscribing to the event

Handler parameters

Usage example

SESSION_END

Subscribing to the event

Handler parameters

BEGIN

Subscribing to the event

Handler parameters

Usage example

END

Subscribing to the event

Handler parameters

Usage example

SUITE_BEGIN

Subscribing to the event

Handler Parameters

Event Interception

SUITE_END

Event Subscription

Handler Parameters

Event Interception

TEST_BEGIN

Event Subscription

Handler Parameters

Event Interception

Usage Example

TEST_END