Sure, dietitians have the food pyramid and Egypt has the great pyramids. But have you heard of the testing pyramid? More importantly, are you using it? It’s more than just a PowerPoint-ready visual. It’s the formula for a successful testing strategy, a layered approach that utilized different testing methodologies.
There’s a good reason that developers rely on its simple but relevant teaching: It works. So, let’s take a close look at the testing pyramid: What it is, what it prescribes, and what it has to offer developers. Finally, we’ll talk a look at what can happen when it’s taken for granted.
What is the testing pyramid?
Sometimes referred to as the test automation pyramid, the testing pyramid is a framework that helps developers employ adequate testing strategies to build better features and successfully introduce changes. The testing pyramid groups the types of automated testing based on their relevance, order of magnitude, and order of execution.
The concept of the testing pyramid was introduced by computer scientist Mike Cohn in his book Succeeding with Agile: Software Development Using Scrum (2009, Addison-Wesley Professional) as a metaphor for thinking about software development.
In most representations, the testing pyramid contains three types of tests: Unit, integration, and end-to-end. Sometimes they’re called different things. For example, integration tests are referred to as service tests and e2e tests can be called UI tests.
In other representations, there may be more layers to the pyramid, such as a distinction between automated UI tests, and manual UI tests, or splitting integration tests into either system or business integration.
Sometimes, the pyramid changes based on platform. It might look a little different for mobile app automation than a back-end solution. Whatever the permutation of the pyramid, there are similar principles at play. All communicate that your testing suite should include tests of different granularity and complexity.
Unit
A unit test is an automated test that checks that the smallest units of code are working correctly. The goal of a unit test is to verify the accuracy of logic under multiple conditions. If there is a function that adds two numbers together we’d want to verify it outputs 10 when 5 and 5 are provided. We would also want to write tests for edge cases.
What happens when two large numbers are provided or if only one number is provided? These test each bit of functionality that makes up your app. Unit tests test one method or class at a time and do not rely on dependencies. Every precondition a unit test needs to meet should be included in the test itself to ensure accurate results with each test.
Unit tests work by calling a method, passing any necessary parameters, making an assertion about the result, and evaluating the results. Since unit tests represent the lion’s share of testing, they make up the base of the pyramid.
Integration
Also known as service tests, integration tests validate how different components of a system interact with one another. This is where we make sure everything plays nice together.
Where a unit test may test a function in a calculator application, the integration test would test the entire calculator’s behavior. Integration testing can also involve interacting with third-party and external dependencies, like an API. For this reason, integration tests will involve mocking and stubbing dependencies.
Integration testing occurs after unit testing and should only occur once the individual unit tests are verified. With both integration and unit tests, code is called directly with no UI interaction needed.
End-to-End
Finally, let’s talk end-to-end testing. End-to-end testing (often abbreviated as E2E testing) occurs after all other testing is successfully executed. End-to-End testing involves verifying the complete functionality and performance of the entire system, doing its best to replicate a production-like environment where the application will have its full set of features tested.
Since the goal is to simulate how an actual end-user will interact with the system, end-to-end testing is sometimes referred to as UI testing.
E2E testing is complex. It involves multiple systems, sub-systems, external dependencies, different UIs, and permissions. For these reasons, it requires more complex tooling to create end-to-end tests that are capable of emulating user experiences and recording interaction.
Why a Pyramid?
If we consider the width of the pyramid, the amount and frequency of tests, and its height complexity and relative cost, the pyramid serves us well. The unit tests are the foundation of the testing suite. Integration and E2E testing would not be possible without them. Integration tests are fewer in number but more complex than unit tests, while E2E takes the longest to develop.
Now that you understand all three parts of the testing pyramid, how do we structure our test suite? At a high level, we know that unit tests make the majority of test cases, followed by integration and end-to-end. The pyramid informs developers how often they should feature different sections of the test suite. A well-designed test suite based on the pyramid helps developers find and debug issues quicker.
Criticism
Of course, no philosophy is perfect. Here are a few criticisms surrounding the testing pyramid that you’d be wise to keep in mind:
- It doesn’t consider value: A complex test that takes 10x longer to develop might catch 100% of all possible errors. If that’s the case, why would you need a bunch of unit tests?
- Assumes developer resources: Developing according to the test pyramid would require a lot of work. An organization would need the right skill sets to be able to pump out that many tests.
- Outdated: A lot has changed in a decade. Now testing tends to fall heavier on UI testing and modern test automation can simplify the test suite.
- Where are the manual tests?: Automation isn’t appropriate for every scenario. The testing pyramid doesn’t mention manual tests, which many developers see as always necessary to some degree.
Alternatives
Over the years, critics have offered alternatives to the testing pyramid that align closer to modern testing needs.
The testing trophy, introduced by educator Kent Dodds, includes static tests as a base followed by unit tests, integration tests, and end-to-end testing. In this framework, unit tests take up the smallest amount of space with a bigger emphasis on integration tests.
That said, the reverse pyramid (pictured above) may be a simple fix to address some of the criticisms. Here, you’ll notice the the diagram flipped and reordered with unit tests at the bottom, but representing the smallest amount of tests followed by integration and End-to-End. In this framework, more time and focus is spent on the UI testing, than on perfecting unit tests.
Conclusion
The testing pyramid is a time-tested framework. It helps developers make sense of their testing suite and apply testing resources where they’d be most effective. There are alternatives and variations to the testing pyramid and plenty of criticism but the principles still apply – there needs to be a framework for managing your test suite. The testing pyramid can help you get there.
Disclaimer: The views and opinions expressed above are those of the contributor and do not necessarily represent or reflect the official beliefs or positions of Sofy.
