Jack: Good afternoon everybody, and welcome to our Made Tech Talks webinar. Today’s topic is “Driving modernisation with the DVLA: a case study in collaboration and automation. “ Our speaker today is Scott Edwards, one of our lead engineers here at Made Tech. Now, before I hand over to Scott I’m going to very quickly run through how today is going to work. So after housekeeping here, it will be a 45 minute presentation from Scott, followed by a 15 minute Q&A at the end of the presentation. So if you do have any questions for Scott throughout the presentation, please make use of the Q&A function found in your toolbar. Once the presentation is concluded, we will then endeavour to answer as many of your questions as possible.
Once the webinar has concluded we will be sending out feedback forms to all of our attendees. They are about four questions long, take about a minute to fill out, and they really help us to improve our webinars for the future. We’re also going to be sharing a little bit of information about what we’ve got coming out soon, including more information on our podcast that we launched earlier this week – “Making Tech Better” – so stick around at the end to find out more about that.
I should also mention, this presentation does come with live captions. If you want to access those, press the ‘cc’ button found in your toolbar. We’ll also be sharing information on how to access these subtitles, throughout the presentation in the chat function, multiple times just in case you missed it on the way in. Last note: this presentation will be being recorded. That’s enough from me. I’m going to hand over to Scott. Scott do you want to take it away?
Scott: Yeah, thanks very much Jack. I am going to steal your screen share from you… Alright, hello everybody! Thanks for joining us today. I’m excited to talk to you about today’s topic, which is “Driving modernisation with the DVLA: a case study in collaboration and automation”. So today we’re going to talk about a quality challenge that a lot of technical teams face, and then we’re going to have a look at how to face this challenge using a case study at the DVLA. We’re going to break that into two parts: we’re going to look at the Agile processes that underpin this case study; and then we’re going to talk about managing increasing testing scope through automation. As we talk through the subject, I’ll mention some points about how you can get started doing this in your own business organisation.
So let’s get going. First up – what is the challenge that I talk about? So, what we’re talking about here is, as you build and scale the service, add new features, bring new users on board and so on, you need to ensure quality, as your service grows. So we’re going to look now at an example of a DVLA service, and talk about how the need for quality could scale in that particular service. So what we’re going to look at here is the logbook service. This is a service you can use online, as you can with many gov.uk services. If you need to replace your vehicle logbook – if for example no names mentioned – looking at myself here, you have recently moved house and managed to misplace your vehicle logbook and need to order a new one, you can pop onto the DVLA website, work your way through the service and get a log book delivered to you at some point, if you fill in all your details correctly.
So we’re looking at the entry point for the service. I’m using myself as a case study, or someone who would have to go through this. I will first have a look at – as a developer, or as a tester as well – when I look at this page, there’s a lot of stuff I see here that would probably need to be tested in order to ensure a quality service. So let’s have a look at that. So this was my first glance-look at this page. Let me turn on my laser pointer. First thing I noticed was there is a feedback link at the top. There’s some sort of behaviour that this system, this process, needs to exhibit when I click that link, whatever that might be. There’s a ‘back’ button, which obviously does something like going back in the service. It might have special behaviour. There are a couple of options I can select on this page, and maybe, depending which one I selec,t I get taken to a different place or I get shown some more options. There’s this big green ‘continue’ button that does something, right, and there’s this section at the bottom with a whole bunch of other things we can link to from this page. So we can have a look at the cookie policy, we can use this in Cymraeg if we want to, we can look at the terms and conditions, and so on and so forth. Quite a lot going on on a single page, I think you might agree. And quite a lot of stuff we need to ensure works correctly to provide a good service to our citizens.
So now if I take this page and I put it in the context of a customer journey of ordering a logbook, it looks something like this. So it might be a bit smaller on your screen. The point is more multiple pages than the specifics of the journey, but just a quick overview: I say I need a new logbook and I specify the reason why, I enter my vehicle registration number, I enter my vehicle identification number, I enter my personal details, and I get some result. In this case, I got ‘unable to continue’ because when I was taking these screenshots I just entered nonsense details for myself just to see what would happen. The point here is it’s not just a single screen anymore, right. In these five screens, there’s five times the complexity of the quality things that we need to ensure on this customer journey. Alright, five times doesn’t sound like too many, but there are multiple customer journeys. So if we take this dotted border on the screen as a customer journey, and we move on to multiple customer journeys, now suddenly this scope and scale of testing need is kind of growing exponentially.
There’s a bunch of other stuff we need to think about too, that isn’t just what you see on the screen. So when you’re building a service like this, there are a bunch of connected services you usually have to talk to, both internal and external. We call these ‘integration points’. There can be logic that’s not directly visible or obvious on the page, that’s happening. Of course we’ve got the multi-language capability that I mentioned, we need to think about accessibility, we need to make sure that people with who have impaired vision can use these pages, people use screen readers, all those sorts of things. We need to think about edge cases, strange customer behaviour. For example, when I went onto that page and entered a bunch of junk data, the servers should react in a particular way. Maybe even bug reports and defects coming in from our customers saying “hey I tried to do a thing, didn’t work as I expected, can you fix this?” Of course we’ve got what I mentioned earlier, which is that ever-growing feature set. So as we add more features onto the service, as we build and deploy these things, we add to the complexity of testing and the complexity of ensuring quality.
And the thing with this is that this list will grow as you build your service, and it’s really really hard to predict the scope and scale of this testing. It does tend to grow exponentially, as I mentioned. You almost need a helicopter to get a full high-level view of what’s actually going on here. It can actually be a bit daunting. So the example I’m using is: imagine that you arrive at this intersection and you are faced with all of these signs, and you have to process all of these while you are being charged by a herd of angry rhinoceros. This is what it can feel like sometimes when you’re faced with this challenge. It’s very easy to get tied up in all the lists, like we need to do multi-language, we need to do all these other things that I listed on that previous slide, and get a bit wound up. Now, I think a good thing to do in that scenario is let’s take a step back and have a look at – rather than the outputs that you’re putting out, which are testing integrated services and all those things – what are the outcomes we’re actually looking for, and what can we do to effectively achieve these outcomes? So obviously our primary outcome here that we’re looking for is to build a high quality service for our customers. To do that, we want to catch defects early, we want to catch them as often as possible, preferably as soon as they happen, and we want to deal with regression risk as our service grows and matures. So when I say regression, what I’m talking about is this idea that while something might be working now, might be in a positive state. You may change something later, not notice that you broke the old thing, so it regresses to a previous state where it wasn’t working as expected, and you need to capture that risk and do something about it.
So naturally, there’s those two parts of building a high quality service. We’d also like to, if possible, lower ongoing operational costs – naturally every business would like to do that – and also lower our organisational risk levels. So if you are doing things like processing payments or perhaps managing personal information, there are a couple of risks associated there. Obviously if things go wrong there, you risk reputational damage. You also risk lost business, which comes via failing to serve your customers, amongst other things. So I guess the question here is – well, this is a great idea, this outcome of building a high quality service, so the question would be – “how do we actually ensure those quality levels?”
So the simple, maybe over-simplistic, answer is – you need to test your service. You need to push that thing through, you know, work that service hard and make sure that it’s doing what you expect. And the typical way to do this – and there’s absolutely nothing wrong with this – is to hire a team to constantly push the service to its limits, and to try to break it on a day-to-day basis. We typically refer to people in this role as ‘testers’.
So, what does that look like? Well, in the traditional approach, let’s call it what I like to call the ‘test last’ approach… You have quite a lot going on in this picture, so I will rely on my trusty laser pointer – So, we’ve got a feature we need to build. We’ll creatively call it ‘Feature A’. Some developers build this feature and then they throw it over this wall, and some testers pick this feature up from a backlog somewhere, and they test it, and they do everything they can to break it. They also add it to a regression pack which says “Hey, every time we go live (‘push to production’ is another way of saying go live) we need to make sure that we test Feature A is working as expected.” Okay great, let’s build Feature B and follow the same process. Developers build it, they throw it over this wall represented by this lovely brick emoji, the testers test it and they add it to their regression pack. We follow this process on and on and on.
Now when I look at this process, I spot two red flags straight away. The first is this wall. So, teams throwing things over walls to each other has a bunch of risks associated with it: miscommunication, passing out of responsibility, all sorts of lovely things like that. That’s the first risk. And the second risk that I spot is this ever-growing regression scope, right. So it doesn’t look too bad right now, because we’ve only got three features, A, B and C. But if I move on to a view of what the future could look like, it might look like this. We might be all the way up to Feature Z, which is the 26th letter of our alphabet, and at this point we now have 26 things that we need to run past our testers every time we go live to production with a new feature. The risk here, the multiple red flag risk as you can see in this case, is that testing becomes a blocker for effective deployment to production. So the risk here is that every time you want to go live to production, you need to run through this massive test pack manually, and the testers need to finish that and then say “yes we’re good to go” or say “no, something went wrong, you need to fix it” – and you have this huge time-sink every time you need to go live. We’ve still got that risk, that red flag, sitting there, of those boundaries, those walls between the teams.
So I look at this and I say “well surely there’s a better approach” – and you’re probably asking the same question, right? Well luckily the answer is yes. I don’t think we’d be hosting this webinar if the answer was no! So, what we’re going to do now is we’re going to look at a case study of a team that’s handling this particularly well. You probably guessed, from the giant logo on the screen and from the title of this webinar, that we’re talking about the Driver and Vehicle Licensing Agency – otherwise known as the DVLA. So this brings us nicely to our DVLA case study.
Let’s give a bit of context here about this case study. So, the DVLA is building a modernisation, a modernised version of their DFP service. DFP stands for “driver’s first provisional” – so that’s the service that allows you to get your first provisional driving license. This enables people to become mobile and do wonderful things like drive to work. I think this is a really important service that we are providing to society. I think you would probably agree with me on that. We at Made Tech are integrated with the DVLA team, so we work as a blended team, what some people refer to as a ‘rainbow team’. There’s no separate Made Tech over here and DVLA over here, we all sit together on the same squads. On these teams, on these squads, we have service designers, business analysts, developers, testers, delivery managers, list goes on. So quite a long list of roles. And one of our key measures of success to ourselves, and to the service that we provide to our customers – which is the citizens who use the service, is service quality.
I’m pleased to say that when Made Tech did arrive on this service to help out, we found that the DVLA team was working really hard to build a robust well-engineered service, and actually doing a really good job of it, so that was great. It’s nice to join a team that’s really doing well already. That being said, there are always some challenges and some improvements we can make as we join deliveries, and as we bring several new pairs of eyes have a look at things. So let’s have a look at two of those challenges. Those two challenges are those groups of people that I mentioned. So we’ve got service designers, business analysts, developers, testers, delivery managers and so on. Those are very defined roles that do very defined things, and the danger there – luckily this wasn’t really happening – but the danger there is that people can move into their little boxes and not collaborate, and rather just go “I am a service designer” just for a random example “and I just do what service designers do, and I pass it over that wall” – so that’s the first challenge we looked at.
The second is that constant increase in testing scope as we build the service out, what I mentioned earlier, right. So we’re constantly building new end-to-end customer journeys. Like I mentioned, that list grows over time. We’ve got those edge-case customer journeys coming through, we’ve got integration testing to do, to make sure that we are connecting to connected services integration points as expected, we’ve got unit testing, which I will say is kind of like testing the ‘plus’ function on a calculator, a small part of a whole. There’s quite a lot to think about here. So let’s look at these one after the other, because they do lead into each other.
First step – we’ll look at challenge 1, which is always driving for more and stronger collaboration. Another way of saying that is that these lovely walls we identified earlier need to go. Just to be clear, this was going pretty well at the DVLA when we arrived, but we can always make things better, that’s part of our job as a blended squad. So if you have this scenario in your organisation, something you can look at as an alternative is – rather than building an entire feature and throwing it over that wall – is moving to a ‘whole team plus vertical slice’ way of working. So rather than having separate groups, you’ve got – as shown on the left here – the service designers, the developers, the business owners, testers, delivery managers, whatever the roles you may have in your organisation, working together to build slices of functionality together. So what we’re talking about here is – you may remember in the previous slides these boxes of different colours were staggered. So in terms of time, that needed to happen one after the other. Well now we’ve got everyone working on the bits and pieces that I need to contribute to deliver Feature A, or vertical slice A, at the same time.
This goes a long way to breaking down those walls, those barriers that we identified earlier. We’ve still got, unfortunately, our risk down here. You may notice this is the red flag of ever-growing regression scope. But we are now… once we start looking at working like this, actually collaborating instead of passing things over walls. So… “this is great, Scott, wonderful. Thanks for telling us that we should collaborate, very helpful. How do we actually do that?!” – So this is the structure we use in our teams at the DVLA. We have everyone across those roles collaborating in the same virtual room. So in a perfect world we wouldn’t be in the global scenario we’re in right now, and we’d all be in a building together doing this, but unfortunately that’s not possible right now, and so we have to do what we can using the various tools available to us like Google Meet and MS Teams and all these wonderful things that are available to us.
So we try to get everyone in the same room collaborating and talking together. We use this structure to allow us to do that called Scrum, which is an Agile framework. This gives us our baseline. Probably I would guess a lot of people in this webinar have heard of Scrum before, but for those of you who haven’t, we’re going to do a quick-run through of how this works, at a very high level. So, the idea is that you gather all of your business needs together, right, so you work together to understand the business needs, rather than necessarily just handing them down from on high. You collaborate to understand them, you refine those into something called ‘user stories’ which are independent pieces of useful functionality that can be delivered to your users. You split your work into sensible iterations, and you assign those user stories to those iterations. An iteration can be a week, it could be two weeks, it doesn’t really matter. Then you execute on building that solution as a single team, using those split-up user stories and those that specified time periods – let’s call it two weeks, for argument’s sake. So that’s great, it gives you some sort of structure to work on this. What you also want to do when you’re doing this, when you’re working with Scrum, once you’ve gotten used to it, is also iterate on the process. So we’re calling this an Agile methodology. I like to think of this as being Agile, so we’ll use this framework. We’ll do what we can using it, but there might be bits of it that we could feel we could do better, or things don’t work for us, so we’ll eventually treat it as a set of guidelines and iterate on that structure in a way that works for us. And this is a constant point of discussion in our DVLA squads. We’re always reflecting on our own processes and seeing what we could be doing better. In an ideal world, the answer would be ‘nothing’ – we’re doing everything great! But there’s always something that could be a bit better. We’re very open about this with each other.
And that leads me into the point about radical transparency. We’re lucky in our teams that we have this to a very high degree. You might have to work on this with your teams and your squads, because it’s very hard if you’re not used to it, but like I mentioned we’re very self-reflective and we understand that we’re all in this together, trying to achieve some strategic objective, whatever that may be. In this case it’s to deliver this high quality service, and we want to make sure that we are doing the best we can as a squad to do that.
Okay, sounds great, so how do I actually make this happen? So I would say, if this is all new to you, a good first choice would be to adopt an Agile framework slowly. Don’t try to throw the book at it on day one. Maybe choose Scrum. There’s other ones out there like Kanban; Scrum is the one you’ll probably hear about the most, I would guess. You need to understand that if you read about all this wonderful Scrum stuff and want to apply it in your teams, it’s not just a case of a process change, it’s actually a way of thinking. It’s a cultural change that you need to apply in your teams. Part of that is that people’s identities are intertwined in their roles. I still refer to myself as a developer all the time. I see myself as a developer. Testers want to test; developers want to develop. It’s just part of being a human, right? And so your way of working is part of who you are, and so you can’t just impose that on people to suddenly work a brand new way. They will resist you.
So something you can start doing here if you want to get people talking collaborating and understanding and empathising with what other roles are doing: you spend a bit of time running domain knowledge sharing sessions. So we’ve got some of these, we’ve got these running on a weekly basis on Friday mornings before lunch, where we talk about various things with business domain stuff, technical stuff, whatever it may be, and the entire team attends, and we share knowledge and information like that.,
After you’ve maybe done that, you could look at actually grabbing the Scrum guide. This is free, this is a full guide to running Scrum in your teams, that you can grab. There’s no sales bit so don’t worry, it’s all absolutely free. You can just go to scrum.org and search on that website and you’ll find it. I would suggest perhaps, if you’re looking to follow this approach, grabbing one small feature or set of features, small features, work on those as a blended squad, so try follow this collaborative approach. And at the end, run what we call a retrospective to figure out – Hey, is this actually working for us? What went well? What didn’t well go so well? What could we improve on next time? There are lots of different ways to run retrospectives. That stuff’s all in the Scrum guide. I definitely recommend giving that a read if you can.
Then once you’ve done that, and you’ve said, okay we’re going to improve it this way, and that way, and tweak it, build another feature. Use the new process that you’ve tweaked, and run another retrospective at the end. The only hard and fast rule I’m giving you today is you’re not allowed to skip retrospectives. It’s tempting sometimes to say oh, we’ll just get on with our work. No, you have to run your retrospective, you have to improve your process. And then over time, what will happen is that your retrospectives will start creating a process that’s tailored for you and your teams and your squads, allowing you to effectively deliver your strategic vision of say a high quality service for drivers’ first provisional applications. Brilliant. But stick to the Scrum guide at first, refine over time, but initially just try it out by the book, always remember that shared goal.
This is great, so what were the results that we’ve been seeing? Not only when we landed but also as we refine this process in our DVLA squads. So there have been a couple of really good outcomes here. One is that we’ve got deeper knowledge-sharing and skill-sharing between teams. So we have those Friday learning sessions, amongst other things going on, where we’re really starting to understand individually the scope of the whole thing we’re doing, which is great. And I’ve noticed a lot of question-asking happening. So for instance, the developers will approach the testers and say “hey, I’m trying to test this thing, can you help me to figure out this bit, A, B and C?” The tester might come to the developer and say “I see you’ve built this feature this way. Can you explain why?” – and they’ll go together to the business analyst and so on. And everyone is asking lots and lots of questions all the time, to make sure that everybody understands what’s going on. What you find here all the time is people asking questions that everybody in the room wants to know the answers to, but is maybe too embarrassed to ask. And as one person starts opening up to answering questions, you find that it becomes a cultural thing. That’s just what you do. And you should… you don’t need to be shy to ask.
Another outcome that we’ve got going on here, which is great is, we split our fairly large team into three squads, all blended squads, with all those roles in them – business analysts, testers, delivery managers, all those ones I mentioned – and what’s starting to happen is that certain squads are becoming specialists in certain areas of our system, let’s call it. So while they may work on all the individual parts, they might work on the bit that facilitates the feedback form or the bit that facilitates capturing custom information or whatever the example may be in your service, and so you find that squads want to grab particular pieces of work and feel ownership of them, because it’s in their sub-domain, global domain, which is pretty great.
I’ve also noticed a drop in re-work, and so with this more open communication and collaboration, when something is maybe going wrong in the development environment and somebody needs to fix it, the immediate instinct is not dive in and fix it, where you run the risk of three people doing that at the same time, possibly standing on each other’s toes or actually just wasting a bit of time, because it’s maybe a one person or two person job. The instinct is now to have a chat about it, understand what’s going on, see if anyone else is actively looking at it, maybe you’ll discover why it’s happening just by having that conversation, and then to decide who’s fixing it or doing whatever the thing is that needs to be done, and diving into it. Pretty great.
The last two are more personal for me, the last two points on this slide. I like to work in a high trust environment, where I can be extremely open and transparent with my colleagues. And when I say that, just to be very clear, I mean all of my colleagues – DVLA, Made Tech, any role, any level of seniority. That is something that’s definitely happening on our delivery, and it’s an absolute pleasure to work like that because it feels like you’re working with your trusted mates rather than with your colleagues, and you have to put on a front and worry about what they think and all this sort of stuff that makes work unpleasant.
And then the second point is that this collaborative environment seems to me to make work much more rewarding for the teams overall. We really feel like a sports team trying to achieve a common goal, you know, we want to put the football into the net – preferably the correct net! So we’re all working towards that goal together, rather than as individuals struggling on individual tasks. In my personal experience, I generally do because I’m an overly enthusiastic happy person, but I really enjoy coming to work for this team every day. I don’t wake up on Monday – “Oh no, not work” – coming to work, building this great service with my friends, pretty awesome.
So, that all sounds great right, but let’s talk about how this ties in to our second challenge that we mentioned earlier. So the main thing that we spoke about as part of our quality challenge is this constantly increasing testing scope. I mentioned these earlier, we’ve got new end-to-end user journeys constantly being built, we’ve got edge-case user journeys that we need to consider, we’ve got to worry about integration testing, unit testing, I mean this list goes on. I always warn people about this, if you get me started on unit testing you will never hear the end of it, so I’m going to move on to the next slide before that happens, and visualise what this could look like. So let me get my trusty laser pointer out.
What I’ve tried to draw in this graph here, I hope this is clear, is the difference between what we have available to tackle this ever-growing testing scope versus what the scope looks like. So let’s say that the components that we have available are people, budget, skills and time. Those are represented by the blue line over time. Now, I’m not naïve, I know that that fluctuates over time, it’s not a straight line, it wobbles a little bit here and there. People go on holiday maybe, you know, things happen, so it’s not always constant, but it’s kind of predictable if you know the shape of your squads. The problem here is that we have this testing scope, and all the way up to this intersection point we have enough people to tackle this, but as the service grows the testing scope rides up and becomes much bigger than we can handle in our squads. It’s just a natural evolution of large services. This is a problem, right? So the question is – well what options do we have here?
So I’ve drawn out two terrible options, in my opinion. The first one is, okay, so we just stretch out the time axis. So yeah, we do the same thing, but we accept that and we just work hard to finish all this testing scope. As you can see, with the stretch type picture, just takes us a really long time to do that. And that ties back to that picture I drew earlier of massively increasing regression scope with lots of red flags on it. That’s not good, right? What about this: What if we say – okay, so we’ve got blue – just the people, time, skills etc we have available to us. We’ve got orange, which is the scope of testing we need to do to ensure that we deliver a quality service. At some point, we’re going to hit an area where these two intersect, and now we need to do more testing that we’re able to do. So you know what, we’re actually just not going to do any testing at that point, we’re just going to hope. Hope to me sounds like a terrible strategy for ensuring quality, and I see 6 000 red flags over here, because you’ve now got this massive area that where this curve should have been, that’s just not tested. Possibly, definitely in my opinion, worse idea than just stretching the time axis out.
So what if we could do this instead: I really hope my diagramming skills are making sense here. So we’ve got our same two lines, alright – we’ve got available people, time, skills etc in the blue; and we’ve got this ever-growing testing scope which we know we want to keep below the blue line. We want to be able to let our people work nine-to-five and enjoy their weekends, but wouldn’t it be good if we could find a way to ensure that the work we do now isn’t once-off work, keeps providing value. So pretty much every business owner’s dream – work done once that keeps adding value going forward.
Another way you can look at this or maybe visualise this, is filling up a swimming pool. You fill it up once and then the water is there. You don’t have to refill it every single time you want to have a swim, because you just use the last water. You top it up occasionally. I appreciate feedback on my analogy skills! So what we’re actually looking at here is moving our desired outcome, of moving from this old bad world, to this new great world, where we have cumulative testing capability being built up over time, without having to hire 600 000 people to do all of our tests, which is not feasible because I don’t think there’s that many developers and testers in the world probably.
So how do we achieve this? What you can do here is you can leverage the collaborative agile framework you adopted, hopefully immediately after I finished the last section. You can take that skill-sharing and knowledge sharing that you’ve now adopted in your teams, and has kind of become a natural part of how you work together. And you can start sharing some of the responsibilities of roles, so you can get your developers to start testing and you can get your testers to actually start writing automated test code, pretty awesome. When we arrived at the DVLA project, most of the testing was actually automated already, which is pretty great to see. Well, to get to my last point on the slide – what you want to end up with here is automated testing, which is exactly what it sounds like. These are tests that run at all layers of the system, front-end, back-end integration, all those massive things from that list that I listed earlier. They’re coded once by the team as a whole, and they’re run over and over again continuously by machines in an automated manner. So we’re leveraging that build-up of test capability over time.
So we’ll look at an example scenario of what this looks like. So if you remember, we have this wonderful collaborative world now. The team’s all collaborating, the testers and the developers and the VA’s and everybody’s working together. Excellent. We’ve still got this massive red flag of this massive regression scope growing over time, right, because we’re still doing manual testing and adding things to the regression pack. So this is what the world looks like before you apply this sort of thing. This is what the world will look like after you apply this way of thinking, this automated testing way of thinking. So what have we got going on in this page? So we have developers and testers working together on the same feature, as we mentioned earlier, constantly communicating to build this feature called A, Feature A.
So what’s happening? Developer writes a unit test, every time someone adds an automated test hub, represented with this little robot. They write some automated unit tests, and then they write the code to execute on those tests. In parallel, the testers are writing end-to-end tests – and this to be clear is the real way we do things with the DVLA. This isn’t just some conceptual idea. Testers are writing end-to-end tests to test that piece of logic, that front-end flow, whatever it may be, and they’re writing them in an automated way, using code. There’s another little robot automated test, right, and they write a bunch of those, and they use their tester’s instinct to go like “oh what happens in this edge-case if I were to do this weird thing?” or “what happens if I change the language halfway through the customer journey?” all sorts of things like that. What typically tends to happen is that we find the testers and the developers working on end-to-end tests together a little bit towards the end of the process, right, so they’re adding more and more of these little robotic automated tests together. And then we finish the feature and everybody’s happy, we can all go have coffee and have a laugh together, rather than throwing things over a wall and back over the wall when things are broken. So we’ve got this this feeling of camaraderie in the team as a result.
So that’s great, so we build Feature A like this, well what happens when we add Feature B, Feature C, Feature D etc? You start building up something that looks a little bit like this. So for each of these features we’ve got, whatever it is, it doesn’t matter what the feature is, and this is just a high level example. So we’ve got Feature A doing a thing, we’ve got a whole bunch of automated tests that make sure that Feature A does what it says on the box. Similarly for Feature B, similarly for Feature C and so on and so forth, all the way up to Feature Z. So perhaps another way of visualising this is we wanted to fill our swimming pool up before we swim, and use the same water to swim in the whole time. We wanted to fill this area under this graph with automated tests, and through the collaboration on building of these tests, that’s what we end up with. The developers and the testers are building these robotic little tests here, adding them to a test pack that’s automatically run by machines, constantly, on every release, on every build, on every feature change, these tests run automatically.
So, sounds great. What are some of the results we’ve seen? So we now have automated – great word – scaled, fast regression testing. So I can discover very quickly as a developer if I make a change, whether I’ve broken something or not, and whether my new change is working as expected, because there are regression tests to make sure I’ve broken things, and there are new tests for my feature to make sure what I’m supposed to be building is working as expected. It gives me the ability to catch errors quickly, early, often, you know. People make mistakes all the time, and this helps you to catch them. It also de-risks key resource dependencies. What I mean here – I hate this word resource in this context – unfortunately I can’t think of a better one: So if I am the expert at part A of the system, and I go on holiday and someone else wants to change it, you’ve got a problem there, because well who’s going to build this thing? No-one knows how it works. Well, we’ve now got the suite of fully automated tests that actually explain at a code level how this thing works, so any developer or any tester, works on the codebase, can go look at those tests and understand very clearly – “oh yes, I get why this works this way, I get why it looks that way, I understand what the feature looks like.”
Oh, very clever VAS as well. I always appreciate this… we always manage to link our code and our test back to tickets in our ticketing system, so I can then go read the business logic of why this thing works this way, which is really important I believe as well. Another great result or outcome we’ve got here, is that we capture common integration issues early. So, what I’m saying here is, if we need to integrate with other systems that do useful things, so for us, we don’t have to do them ourselves, we want to learn very quickly if we are not integrating those systems correctly. And so if a bunch of tests, automated tests, keep failing, and I keep going and looking at them, and I keep seeing every time I call a certain service and ask it for somebody’s address for example, whatever the case may be… I learn very quickly that I’ve probably done something wrong in the way I talk to that service, and I can discover that before I push to production and push that problem onto a customer, because these tests are being run in an automated manner, over and over and over and over again.
This does also lower our ongoing operational costs. So part of the Made Tech model is not for us to come in there and ingrain ourselves into a team or a squad and live there for the rest of time; it’s to come there, do the work we need to do to help to deliver the service in question, and then step away and let the team self-manage and take the service forward themselves, as they should be able to. And this helps us to do this, because while we collaborate on these tests in the squads, we are also building the ability to do full regression without us having to be there. So we, even within the DVLA, we will allow people to move on to other projects because they’re not having to maintain this thing the whole time, because quality is baked-in.
We also maintain… sorry we lower our organisational risk levels, and there’s a very particular way that we… I mean there are many ways, but there’s a very particular way that I really appreciate, and that’s that we have an automated rule that says you cannot deploy a live service if one of your tests is failing. It doesn’t matter if you’ve got 60,000 tests – we don’t have 60,000 tests – but even if you do, if one of them fails you can’t deploy. And this means that we always automatically check that regression risk before we go live. We really don’t want to discover that in a production environment, and this helps us to de-risk that. Of course you can never completely eradicate risk, but it helps us to manage it.
So, this looks great, we’ve got automated tests doing all these wonderful things, and we’ve got a collaborative team working together. So just to summarise that, what it all looks like together is this: Trusting collaborative team that shares knowledge and their workload effectively, and can focus on our main goal of building a high quality service. This is because we have this collaborative framework to work within. Another benefit we have in that framework is that we have these constantly increasing skill-sets, because of the way we collaborate. We’ve got cross-role learning happening the whole time, and we end up with a happier, more capable team overall. Part of the work we do together when we collaborate is we build automated tests end-to-end, which catch defects for us at all levels, which is wonderful.
And what we end up with as well, is a high quality codebase that’s pretty easy to work with. So this is a big thing, as a developer, I want to walk into a codebase and be able to understand it fairly quickly. Testing kind of enables you to do that, because in order to test code, you kind of need to make it look like something that a human can read. It’s just one of the side effects of testing, I’m not going to dive into that now, it’s just one of those subjects I will ramble on about for hours if you let me! So this facilitates quick delivery, because it’s easy to understand what’s going on and add things to it; and good engineering, because there’s this constant drive towards testing quality. So these two things work really well together: Agile collaborative frameworks and a drive towards what’s made to testing.
Now that’s great, but there’s one last thing, one last thought I want to leave you with, because we stepped into a team that was really doing this pretty darn well, and you may be looking at this and going “These are wonderful ideas and this sounds like a really great thing to do, sounds like Nirvana!” which it isn’t always of course, but it’s pretty great. But if you’re going to apply these things in your own organisation, I would be very careful of striving for perfection. No process or idea is perfect. There’s always stuff we can improve on. We mentioned how we always are improving our processes and use of the Agile framework. We’re always learning new things as developers and testers, so nothing will be perfect, especially from day one, because this this is typically easier if you have built your service up this way from day one. If you’re adding this sort of stuff on later it can be trickier as you manage the cultural shift, and as you start introducing the idea of learning, testing frameworks and all that sort of stuff to people.
So I would say the best approach is probably just to try and make it better than it was yesterday. If it’s a little bit easier for a developer to add a feature, if it takes a little bit less time to do your regression testing before go-live, that’s a big win. Maybe what you should do is start with a common pain-point. Maybe there’s some piece of code that everybody’s scared to touch because they feel like they’re going to break it because it’s complicated. Maybe you can start looking at adding some automated tests around that. Get your developers and your testers collaborating on that. So what I’m really talking about when I talk about iterating on our process, is kind of another way of saying: talking about the ‘build, measure, learn’ cycle from the lean start-up. So you build the thing, measure the outcomes, perhaps you measure how long it takes you to build features, or you measure how many defects get reported, whatever works in your context. You learn from that, and you iterate on your process, and this allows you to focus on compounding returns. So, robots filling up the swimming pool – for want of a better analogy – you’re adding more and more robots to fill up that swimming pool, you’re adding that water to make it easier for you to swim in the future.
I’ve really taken a lot of interest recently in this idea of infinite games. There’s a chap on YouTube called Simon Sinek – I’m not sure if I’m pronouncing his surname correctly – who speaks about this a lot. And he speaks about how business (and I think we can apply this to software development) is not a finite game. There’s no winning point. You’re always having to tweak software, change software, improve software, re-release software, all of these wonderful things. And it’s better to think about them as ‘infinite games’ where the purpose of the game is to keep the game going. We want to keep this ball rolling and continuously improve our services. I think it’s really good to think about them from that angle. These frameworks and this automation way of thinking really makes it a lot easier for you to do that.
And then finally… so we’re pretty passionate about this stuff at Made Tech. Like I said twice already, I think I’ll talk about this for hours if you give me a chance. So if you’re looking at this and going “this looks challenging” or “I’ve got lots of questions, could you maybe answer some of them?” please feel free to reach out to us. We’d love to talk to you about this stuff. Jack, that’s it from me. Thank you everybody for listening, I really appreciate it. I’m going to un-share my screen and hand over to Jack.
Jack: Wonderful stuff, thank you very much Scott. Keeping an eye on the time, we’ll dive straight into our Q&A. So the first question ties in quite nicely with another. The question is: “Are there ever times that you do manual testing?” – That also ties into another question of: “You’ve spoken specifically about feature testing. Do you do any other sorts of testing too?”
Scott: Yeah, so those are, you’re correct, quite right, those do tie in together. So there are times that we do manual testing. Why I say this, you can’t automate everything. Maybe you can. I have yet to do that effectively. So there’s certain types of testing that we do manually. So for example, if you want to do a full end-to-end test of a service like this, well at some point somebody has to receive a physical provisional driver’s licence. And so, in a production environment test, or even a pre-production environment test, you actually need to see a message go to that printing device – I’ve never seen one in real life – and actually see a driver’s licence with the correct details coming out. That’s pretty hard to automate.
So that’s the first part of the question. Second part was about other types of testing. Yeah, so there are other types of testing, and you’re quite right, what I’ve been speaking about is feature-level testing: Does the system perform as we expect? Does it do what we expect it to do? But there are other types to think about. And so for example, you could have performance testing, where you hammer your system really really hard and see how it reacts, see if it can cope with the load of, I don’t know, 5,000 people hitting it at the same time for example, which is something we do at the DVLA. We also do security and penetration testing as part of regular IT health-check audits. So that’s where we actually hire people to try break into our systems, and also have internal people doing checks on our various servers and infrastructure that we use, and they come back with findings which we resolve. We’ve got DAC audits. DAC stands for digital accessibility centre, and that’s a team that will go in and check whether your service meets accessibility standards, will come back with a bunch of findings for you to implement, for you to improve and tweak. And then obviously the end-to-end process testing that I mentioned. If anyone from DVLA is listening, when we’re back in the office, I would love to see that printing device – hint hint!
Jack: Lovely stuff. The next question, so here we go: “The codebases for the service I’m working on at the moment includes tests, but there are other tests that need to be run when we deploy beyond the dev environment. These tests are owned by another team, and we have no visibility or ability to run them at our own discretion. How do you approach removing the wall between developer teams and these other testers?”
Scott: That’s a tough one. So what I would imagine happens there is that you think that everything’s working as expected, but there’s this mysterious hidden box of tests that you don’t know whether you’ll pass or not. That sounds like a communication challenge to me. Off the cuff, I would say – I mean I don’t know enough about that person’s particular scenario to answer in a detailed manner – but I would say, I would approach this by actually going and talking to that team if you can, explaining the challenges that you face, and perhaps taking along your whole team with you to explain the challenge from all different angles, and see whether you can get some collaboration going on there. Maybe there’s a good reason why they keep those tests hidden from you, I can’t think of one offhand. I can’t think of why we would do that in this particular DVLA service that we’re working on. But yeah, I would say a friendly approach to those people to try and understand why that’s necessary, might be a good first step.
Jack: Cool, next question: “What do you think would be the top three key must-do’s for building a trusting collaborative team?”
Scott: So I’ll yammer on about communication again. When people feel unhappy about the way teams are performing or the way the team is run, often what comes up is communication. People feel like they don’t know what’s going on, like they don’t understand the greater goal of what we’re actually trying to achieve. So I would say, especially if you’re leading a team, comms is probably your number one starting point. You want to explain to people why they’re doing things… Okay, so get your comms channels open, start talking to people, because you can’t build trust if you’re not communicating with people.
I would say probably a good thing to do is to start understanding the sort of work people do. So I can’t really build trust with somebody and talk to them about their challenges if I don’t understand that. So what I would maybe do, if I use a tester as an example, and imagine I’m dev who doesn’t test – I would go talk to the testers, maybe sit with them for a little bit and understand what it is that they actually do all day, and how I could maybe help out there. And the third must-do, I think tied into the comms, is making sure… so, even though your leaders need to explain to you why you’re doing things… I read a book recently that really opened my eyes to this – you also have a responsibility to understand why you’re doing things. So you should be going up and questioning people to understand the ‘why’ of how things work, and you’ll find that – like I mentioned earlier – as you start facilitating that open communication, trust kind of emerges as a natural by-product. I think if you were to go in with the pure intention of just building trust, it might come across as disingenuous, and so I would be taking a genuine interest in understanding why we’re doing things, and what our strategic objectives are.
Jack: Excellent. The next question is: “How does this work in environments where there’s a lot of systematic failures, no processes or procedures? Have you been involved with projects with a lot of technical depth?”
Scott: Another subject I could go on about! Yeah, I mean, when you work in, not just with Made Tech, but over the course of my career, lots of places where our job is to go in and not only maybe add dev muscle to teams that just need more developers available, but also to help solve both organisational and technical problems, you run into a lot of that technical debt.
One of the biggest challenges people face in those scenarios is convincing people at higher levels. So imagine you’re a developer trying to convince people of this, that you should be allowed to try maybe add some automation to your test packs, or whatever the case may be, so that when… so we can maybe start reining things in a little bit. So one thing I would maybe suggest trying is, next time something goes wrong, I know it’s not a nice thing to say but next time something does go wrong, work with the people at a more leadership level to say “well, rather than just fixing it, can we put something in place that maybe makes sure this doesn’t happen again?” – So strike while the iron is hot and say like “we could add this automated test for example, that makes sure that we don’t ever make this mistake again”. And we can maybe then start looking at – why don’t we run these every day or every time we deport it to production? It ties into the point about starting slow. So you need to make sure that you take it one step at a time. You can’t just go throw like a TDD book at someone, at your boss, and say “hey we need to do this” – because they won’t listen to you. So take it slow. One small thing at a time. Choose your point when you’ve seen something going wrong and you think one of these techniques could fix it, and take it from there.
Jack: Excellent. “As the product grows, the automation test pack will grow. This will mean tests take a long time to run and slow progress. When should we run the tests or change how we run the tests to cope with this?”
Scott: So you’ve got a couple of options here: You can take the brute force option, which is – okay, we’re going to use a much bigger server to run the tests, the automated tests. We’re going to parallelise the test – so if anyone’s non-technical, what that means instead of running them one after the other we’re going to run five of the hundred at a time, on five different threads, and as the test pack gets bigger and bigger and bigger, we are just going to throw more and more hardware, add more and more CPU cycles and just let it run. That will probably work up to a certain scale. Eventually you’ll hit a problem, where you can keep throwing things at that test server, but it’s still taking a long time to run, and now you’ve got developers sitting around and testers sitting around going “I’m waiting for the server to run the test, so I can’t do anything until it’s finished, because I don’t want to context-switch to something else.”
Something I’ve seen happen really effectively at the DVLA in our project, is some of the guys and girls chose a couple of tests that they saw as the most important tests, the most commonly failing, or the ones that had integration points, or the ones that test core business features. And in certain scenarios when you’re doing day-to-day development, those tests are run automatically and will block you if things are going wrong, because you’ve clearly broken something and you need to resolve it. So they run a smaller subset of those tests, and then when we need to push to production or promote through environments, or do things that could be quite risky if something was wrong at that point. You run the massive test pack that takes say an hour or however long it is, so it doesn’t have to be all the tests all the time, just make sure you run all the tests before you push yourself to production.
Jack: Excellent. I think we have time for one more question: “Hi Scott, great talk. What is your view on increasingly quantity of tests slowing down build times and killing fun and flow. Bonus points for your views on flaky tests.”
Scott: I think I know who wrote that question! Yeah, so I have I have personal opinions on this, and I don’t know if they’re always the most objective, because I’m a bit of a zealot when it comes to testing. But I would say your test should be covering all of your business logic. What I mean by that is, if there’s a certain piece of behaviour that’s the key word in the system that is expected to work a certain way, like you click the Welsh language button and you see this translation on the screen, you better make sure that that works as expected. But that being said, you shouldn’t be scared to go in and delete tests that don’t make sense anymore. I’m a big stickler for – I enjoy large quantities of tests in my service – but I would also say if they’re not providing value, you should get rid of them. This is such a subjective question, I’m actually finding it hard to give any real decent advice here, other than: Be pragmatic about it. Make sure that you communicate with your team about what’s happening. Don’t just run around deleting tests, because you’re going to cause chaos!
Jack: Lovely. Well that’s all the time we have for questions. I just want to say a massive thank you again Scott, for taking the time to speak to us today.
Scott: Thank you Jack for having me.
Jack: As I mentioned at the beginning of the webinar, we will be sending out feedback forms to all of our attendees. Again, they’re very short and they help us understand what you’d like to see more of from our webinars in the future. Also, as promised, what we have coming up next. Our Head of Local Government, Glen Oksco, will be speaking on “The future of local digital services” at GovX Digital 2021, the UK’s biggest public sector transformation conference. You can catch that on Wednesday the 12th of May at 8.30am. I’m also very pleased to announce that we, earlier this week, we launched our new Made Tech podcast “Making Tech Better” – hosted by the wonderful Clare Sudbery, one of our Lead Engineers here at Made Tech. The first four episodes are already streaming on Apple podcasts and other major platforms. You can catch the interviews with Kit Collingwood, Paula Paul, Jon Skeet and Jessica Kerr. So yeah, if you get the time to find that, leave a review and tell us what you thought.
If you want to stay in touch with us, or if your question didn’t get answered by Scott during the Q&A, please feel free to reach out. Scott’s Twitter handle is up on the screen @scotteza or if you want to just keep up with all things Made Tech you can check us out. Our website is www.madetech.com or follow us on Twitter, and please do reach out. Our Twitter handle’s up on the screen: @madetech.
So once again, thank you to all of our attendees for coming today, and thank you again Scott for delivering another wonderful talk. Have a wonderful afternoon.
[recording ends]
Back to the episode
