Skip to main content

How to Check Test Stability

Problem​

There are many reasons why tests can fail:

  • infrastructure issues – browser unavailability, network malfunctions, expired access, etc.
  • problems with external services the test interacts with;
  • races within the test itself, where an element does not render in time and the wait period expires;
  • unexpected popups during the test execution that obscure the necessary element, cause differences in screenshots, or prevent clicking the needed element;
  • and so on.

Often these failures do not reproduce on the first attempt. Therefore, to ensure a newly written test is stable, it needs to be run multiple times. But there is one problem: if you run your test in Testplane as is, after the first successful run, Testplane will stop running your test. The test passed – everything is OK. But for stability verification, this is not enough. The test might have passed accidentally, and if you rerun it, it might fail. Ideally, you want to run it not just once or twice, but for example, 20 times and see how many times it passes out of 20. Or 30 times. Or... and so on.

Solution: @testplane/test-repeater​

To solve this problem, the @testplane/test-repeater plugin was developed.

The plugin allows you to run the same test (or group of tests) the required number of times.

The plugin ensures that the tests will be run as many times as you specify, regardless of the results of each run. Moreover, the plugin allows you to start the tests each time in a new browser session. This eliminates browser degradation or other side effects that might occur with repeated runs in the same browser session.

Read more about how to add this plugin to a project, configure, and use it in the plugin documentation.

Usage Examples​

Below are examples of test runs, in which they were run 21 times (1 primary + 20 retries) to check their stability.

Example of Broken Tests​

As seen in the screenshot, the tests were run 21 times and never completed successfully:

Broken Test

Example of Stable Test​

Here, on the contrary, all runs were successful:

Stable Test

Example of Unstable Tests​

In the next screenshot, the first test is almost non-functional – out of 21 attempts, the test passed only once. The second test is quite stable, although 2 failures out of 21 is still not 100% stability. The developer may try to understand why the test fails occasionally.

Unstable Test

Keywords​

  • @testplane/test-repeater