How leogistics views the software development testing pyramid

In recent years, projects have become increasingly agile. This means that longer implementation phases lasting between six and 18 months are broken down into smaller releases that last no longer than two months. Planning is often carried out on a weekly basis as well. For example, in the development of our logistics platform, myleo / dsc, releases are scheduled every two weeks.

In project-based work, however, the scope is constantly evolving, even within the meticulously planned waterfall model. One can speculate about the reasons – a lack of buy-in from specialist departments at the start of the project, insufficient time spent on requirements analysis, or the fluctuating availability of key knowledge holders – but the symptom is evident and very well known.

These are the benefits of rapid iterations

Faster release cycles bring many advantages. There is no need to rely on vague claims such as ‘increased customer satisfaction’, as there are plenty of precisely formulated arguments:

• For example, faster iterations and adjustments to the project plan are possible, and the risk of scope creep is minimised.
• Furthermore, features can be rolled out individually, and users do not have to wait long for a major release. This pays off particularly when the application is already live in production.
• Another aspect, relating to the development organisation, concerns the speed at which developers can work and the assurance that features will continue to function flawlessly even after a release. If a release is rolled out frequently (in our view, at least every four weeks), any bugs that occur in production are fixed more quickly. This is because real or suspected bugs can be traced back more quickly to a change implemented in the past. Once the faulty feature has been quickly identified, the faulty section of the source code can also be found more quickly. It’s just like in real life: when you’re looking for your keys – if you know where and when you last had them in your hand, you can quickly narrow down the places where they might now be.

A brave new, fast-paced world?

So, frequent releases bring a number of advantages, but they also have their downsides. After all, they mean one thing above all else – many small changes, carried out frequently. This means that testing must be carried out after every release at the very latest. From a commercial perspective, the question of who has to test what and to what extent is by no means insignificant.

With a SaaS (Software as a Service) solution, the matter is clear – the manufacturer must take on the lion’s share, as they dictate the release cycle. Testing also plays a vital role in product development and project-based work – after all, it is the only way to verify whether the software does exactly what is expected of it.

In the following, however, I will be referring primarily to SaaS solutions, although these points are certainly also relevant when an implementation project is underway.

The test pyramid from a leogistics perspective

Firstly, we need to set aside the terms ‘developer testing’ and ‘user testing’, as these are too vaguely defined. In product development, the following types of testing exist. Here, it is worth mentioning the test pyramid, which is likely to be familiar and is the industry standard. For the sake of clarity, I have grouped some test types together.

The term ‘pyramid’ seems to refer more to the number of tests (on the left) than to test coverage (on the right). Purists may object that test coverage, too, should form a pyramid, but in this line of thinking they all too often fail to take the budgetary situation into account. In reality, test coverage (measured in terms of both code coverage and branch coverage) is more likely to resemble a stake, with a tapered base and a tapered tip. But even this can take you quite a long way.

However, the underlying principle of both pure theory and practice is the same: automate as much as possible, because automated tests are cheaper in the long run than manual tests.

• Unit tests test individual methods and nothing else. ‘External’ methods must be isolated and either provide reproducible outputs (stub/fake) or validate inputs in a reproducible manner (mock, spy). Common tools used in our practice include ABAP Unit Tests or Jest in the Node.js environment. The effort involved very often correlates with the number of test doubles to be created. Experience shows that tests with more than three test doubles are harder to read, which means the added value can quickly diminish due to increased maintenance effort. In such cases, it is better to move up one level.
• Component/service tests verify the ‘path’ of an API from the top down to the database, but do not test third-party modules or services. Test doubles are therefore also necessary here. Whilst with unit tests we were still working with a scalpel, here we are already working with a pair of scissors – coverage increases, but errors are harder to pinpoint precisely. The initial state of each test is not achieved here via test doubles, but via a defined setup in the database – either in the physical database, isolated in a separate client (for SAP systems), or via an in-memory database started specifically for the test (e.g. https://www.npmjs.com/package/mongodb-memory-server). It is important to note here that the solution using a local in-memory database can usually be run in parallel – i.e. it is also pipeline-capable – whilst a dedicated test client in SAP cannot be run in parallel without significant effort, as there is no separate master data setup per test run, let alone a separate client. It is therefore advisable to run tests serially overnight, rather than in parallel, where race conditions may arise.

Common tools we use in practice include ABAP Unit Tests combined with leogistics’ own framework, as well as Supertest / Jest in the Node.js environment.

• GUI, integration and API tests test the software solution, but not necessarily other systems that are connected to it. The tests are automated and rely directly on the UI or the API. Test doubles are manageable and are limited to simulating incoming messages or verifying outgoing actions (e.g. https://www.inbucket.org/). Provided they can be run on an isolated master data setup, these tests can be parallelised. Otherwise, they are candidates for a serial, periodic test run. Common tools we use in practice include eCATT and Cypress. Incidentally, we are no longer dealing with a scalpel or a pair of scissors here, but rather with a lawnmower – in other words, high coverage, but potentially more time-consuming fault-finding should an error actually be detected.
• Manual testing – this is where the human factor comes into play. Manual testing is essential, but always expensive, and should therefore be used sparingly and with careful consideration. Once repetitive test cases have been identified, one should look further down the pyramid.

Practical example

In the development of myleo / dsc, unit and service tests are firmly integrated into the GitLab pipelines – whenever a commit is made to a remote branch, the tests run. These can be run in parallel and do not interfere with one another. Only once the green light has been given may a merge take place onto the Develop branch, where the tests are run again.

GUI, integration and API tests are run nightly on the first integration environment. Although these can be run in parallel in some cases, they often take too long for the pipelines (typically one to two hours per module), as the UI is also tested. If errors occur overnight, these are visible in the GitLab schedule report in the morning.

A deployment to the pre-production environment takes place every two weeks. The tests are carried out again here. Only once all bugs have been fixed may the release be deployed to the production environment. Manual testing plays a significant role in this process – but more from a UX and business perspective, rather than to safeguard against regressions.

Conclusion

When the railway was invented in the 19th century, followed shortly afterwards by the car, the increase in speed was accompanied by technical safety measures. Compared with technical developments such as car bodies, seat belts, airbags, ABS and ESP, the human factor received less attention. This is also comparable to software development: a faster development pace must not shift the burden of quality assurance exclusively onto the developer or software tester. Here, too, reproducible, technical safeguards are necessary, which must be systematically integrated into the quality assurance process.