How to run reliable E2E tests & what you should avoid

Buggy software sucks. That's why we use End-to-End testing at Talon.One to validate our systems and check our integrations.

In this post, we are going to share how to avoid one very common pitfall of E2E testing: unstable tests.

When writing E2E tests, there is an unavoidable tradeoff between how thorough the tests are and how reliably tests will run in your CI environment.

We often hit technological roadblocks on the challenging quest to simulate real user behaviour. Selenium inexplicably starts throwing Net::ReadTimeOut errors. Your webapp goes unresponsive for no apparent reason, or all of a sudden some element becomes unclickable… even though none of these issues ever come up on your machine! As a consequence, the test gets labelled as “unstable”. This one test failing becomes “normal” and it can be “ignored”.

Eventually, a few more of these unreliable tests pop up and before you know it, nobody cares about tests failing at all.

Your dream of simulating real user behaviour suddenly disintegrates into nothing more than a checkmark your startup can claim they do (E2E tests? Yeah we've got those!).

We want to help you, so below you’ll see exactly how we avoid this trap.

Companies often choose to run E2E tests against a pseudo-stable dev instance of their app with a dirty database. You know, the type of server that gets re-deployed 5 times a day with untested builds.

This is a recipe for disaster and can lead to constant failures that have nothing to do with your tests at all. Maybe your tests run just as the app is restarting, a page gets redesigned without your knowledge, or somebody pushes a change that corrupts the database. All these events will lead to red lines in your reports, and none of it is within your control.

Each test should run completely independent and its result should not depend on the outcome of a previous test. This sounds trivial, but when writing E2E tests, it often happens unintentionally. Imagine this scenario:

Test 1: user visits page A
Test 2: user is on page A, clicks link to visit page B
Test 3: user is on page B, clicks link to visit page C

It might be tempting to run all of these tests right after each other, in the same browser session. Should save some time, right? However, if test 2 fails and you are still on page A at the end of the test, now test 3 will fail as well because you are not even on page B.

In this case, test 3 can only pass if test 2 passes. You've basically created a waterfall of failing tests.

How to avoid this? Test 3 should start with an empty browser, and explicitly visiting page B.

Unstable tests are often long tests that spread multiple UI interactions over many different pages.

This means they will have a lower chance of failing (when Selenium gets lost again).

For example, if you want to run the test Place new order as an existing customer, don’t run your entire sign-up process over the UI, only place the new order through the browser.

Before(test){
 user = MyApi.create_user;
 MyApi.login_with(user);
}

Describe(test){
 user.visit_page;
 assert(page_is_loaded)
 user.place_order
 assert(order_is_placed)
}

Any data required to run the core of the test can be created via API or by inserting data in the database. This will make the test run a lot faster and reduce the risk of unreliable UI interactions messing up your results.

Almost all testing frameworks offer support for flaky tests, either natively or through a plugin. Some examples:

If a test fails unpredictably from time to time and you have run out of ideas for improving reliability, try running it again. And again, and again. We suggest 3 times, just to be safe.

‍

At the very least, remove the most unstable part of it. Sometimes the technical barriers for testing a scenario are higher than the value of the test itself.

It might sound blasphemous to make E2E tests only focus on one specific feature. But the most important thing is that the test are reliable.

Whenever a test fails, everyone involved should be concerned and assume that something is actually wrong. Testing should be taken seriously and we always recommend running E2E tests to validate your systems.

Testing is a good investment for your company and has a lot of benefits for your application. E2E testing helps reliably test all aspects of your application before a release. Although testing can sometimes be painful, you've hopefully learnt a few ways to make your tests more efficient, reliable and save you a big headache.

‍

Interested in joining the Talon.One team? We're hiring! Check out our open positions today.

If you have questions about running your own E2E test or want to share your experience please get in touch! We'd love to hear from you.