When we deliver software, we need to ensure it meets the needs of end-users. The slowest feedback-loop is a “big-bang” release – this is essentially shipping all the features at once after developing for weeks, months or even years. We know this to lead to failure far more often than not… Yet, many teams release software this way, and no we don’t recommend it.
The reason we don’t recommend you practice “big-bang” releases is not that we don’t want to release “perfect software” or because “we are lazy”… It is because we want to lock ourselves into a feedback cycle between the users, any new problems and the code. If you do not course correct often enough, you will build the wrong thing.
The best way to achieve fast feedback loops is with tests, but not just any tests, Extremely Fast Tests.
Unit Tests can be fast since they test isolated aspects of the code. You can isolate away anything that is slow – which is usually anything that involves Input-Output operations. They are also really helpful because they are able to pinpoint the exact location of breakage in functionality. However, this isolation comes with a cost. That cost is that while you know a Unit works, you do not know the Units work together correctly. They provide low-confidence in the correct behaviour of the whole system.
Typically, most teams solve this by writing tests that simulate the act of using a keyboard and mouse to click buttons and fill out fields. We shall call this “Testing through the UI”. This has the benefit of providing high-confidence that the whole system works. These tests come with a cost – that cost is speed. Testing this way is slow. Really, really slow. So slow that you probably want to only have a handful (between 1 and 10) of this type of test.
Many teams realising that these tests are slow, work up from Unit Tests to create so dubbed “Integration Tests”. More often what programmers writing these types of tests do is find load bearing integrations between individual Units, and write tests around those Units to provide increased confidence that they work together.
The downside of Integration Tests is subtle, and we must explore “Acceptance Testing” first to be able to explain why.
Acceptance Tests are what you can talk to your customer about, sometimes your customer can write them. You can have a conversation about them, you can collaborate. This makes them extremely valuable as a tool for answering questions about the Functional requirements of a system. Your customer can pose questions like “What happens to rounding of VAT when there are multiple line items with many different quantities?”, you can quickly answer this question and validate the correctness of the system.
Acceptance Tests are also interesting because you can code them to test through the UI, to test through the HTTP API, or even to test through some other interface. As we know, the slow aspect of testing is I/O operations – so we should avoid that.
This takes us into the world of “Subcutaneous Acceptance Testing” – here defined as testing below the network e.g. HTTP API (which we avoid because it will slow our tests down) and below the framework (we will come back to this later).
This is different from “Integration Testing” in one important detail: acceptance tests are always specified from the perspective of the customer.
This makes them valuable in two ways – you get all the benefits of “Integration Testing” plus you can have a conversation with the customer about them. The subtle downside of integration testing is that it’s not inherent in their nature to encode business rules from the perspective of the customer. That is not to say that you will never need Integration Tests, but you will likely minimize the number of them in favour of Acceptance Tests.
By having a suite of Subcutaneous Acceptance Tests along with Unit Tests, that run in under 30 seconds, borne out of a team that practices disciplined ATDD; you will have Extremely Fast Feedback-Loops. You can ask questions about the functional correctness of the system by writing a single Acceptance Test and get the answer in seconds. The value of this is not to be trifled with, nor should you pass it up without carefully considering the downsides.
So this is the ideal way to write tests. We will explore in a second article at what cost these come, and explore some of the downsides.
About the Author
Principal Software Engineer at Made Tech
Craig is a Principal Technologist and has been at Made Tech for over 7 years. In that time he’s seen Made Tech grow from less than 10 people. He was involved in running some of the earliest iterations of our engineering academies and was a prominent member of the committee (trailblazer) which co-authored the DevOps Engineer Apprenticeship specification.
