Kyle: Hello and welcome to Making Tech Better, Made Tech’s fortnightly podcast bringing you content from all over the world on how to improve your software delivery. I’m Kyle Chapman, I go by he/him, and I work as a Senior Engineer at Made Tech, who are kindly sponsoring us.
This week’s guest is Sam Newman. Sam literally wrote the book on microservices. We’ll come back to that in a bit. He currently works as an independent technology consultant.
Kyle: Hello Sam.
Sam: Hello, how are you?
Kyle: I’m good. Where I want to start is with our main topic. You’ve written the book on this, you’ve run courses on this, so I want to talk about microservices. My first question is what is a microservice? What is a good definition of that?
Sam: It’s a design technique that maximises the amount of buzzword hyped technologies that you can cram into one project at any given point in time. That’s the slightly cynical version. I describe microservices as being a type of service oriented architecture which focuses on achieving independent deployability. The boundaries of microservices are typically designed and structured around a business domain. I describe them as being a subset of service oriented architectures, as there are lots of service oriented architectures where the individual services do not have independent deployability. So, I am trying to be very precise about that differentiation between that and other types of distributed systems that you might see.
Kyle: It’s like a subset of distributed systems, to you?
Sam: I think you could argue that you’ve got distributed systems as the overarching concept, then you’ve got service oriented architectures, and then you’ve got microservices. So, it’s a subset of a subset in many ways. The definition of a distributed system is – you can talk about it technically – which is any system where you’ve got two computers communicating over a non-local network.
If all of your programmes are communicating on the same machine over a local network, it’s not a distributed system. The moment you go over a network, you are building a distributed system whether you realise it or not.
Kyle: I think that happens accidentally quite a lot. People start building architectures that have those qualities without necessarily thinking about the trade offs that they are making.
Sam: I think something else happens as well, which is that people assume that what they are doing now isn’t distributed. Then that what they are going to move to is distributed, and there’s somehow a magical switch. That’s not really the case. I think most of us are already building some form of distributed system, it’s just a much simpler one. This is why I talk about this shift from say a monolithic single process based deployment topology to a microservice architecture. It’s more like degrees of complexity as you become more distributed.
On the one hand you get more complexity, more pain, more suffering. On the other hand, you hopefully create more options in terms of how you are going to solve certain problems. You give yourself a lot more flexibility. That’s the trade off you’re inherently playing here.
Kyle: There’s a theme I see coming out here, where it’s more subtle than a choice between monolith and microservices.
Sam: Yes. It’s probably worth also talking about at least how I define a monolith in that context, as well.
Kyle: Yes, please.
Sam: I describe a monolith as a deployment choice or a deployment constraint. If you deploy your entire system as a single unit, that’s a monolith. Of course, you can have the classical view of a monolithic deployment which is say a [00:03:28 Rubin route?] application, for example. Which is almost like the iconical example of a framework designed to build a simple web-based monolithic application. All my code runs on a single process. I deploy it as a single unit.
You then go to the other extreme, which is that I’ve got a service oriented architecture, I’ve got 20, 50, 100 services, but whenever we do a release, we have to release all of those services together. That’s a distributed monolithic application.
Kyle: Okay, so it doesn’t just refer to is my code in one repo or many repos, there’s more to it than that.
Sam: Yes, and that’s a whole other conversation, because that gets quite – there’s some religious wars about that whole thing. How you organise your code can be orthogonal to how you do your deployments.
Kyle: Yes, I see all this conflated quite a lot, I think. People tend to think of it as we are either over here or over there, there are only two options when it comes to microservices.
Sam: Yes. You start with the dial down to zero, where you don’t have any independent deployability and your system is maybe mostly not distributed. As you turn that dial up a little bit, you add one or two microservices. You are increasing the distribution of your system, so you are adding a bit of complexity there. You’re able to dip your toe in the water of microservices.
Then as you get more confident in that process, you can turn that dial more aggressively to the right, so you find the right spot for you, for your organisation.
Kyle: I’m struck by how similar this is to normal development, right? To work in small iterations and to get a fast feedback loop going. Is it just no different in this area, do you think the same patterns still apply?
Sam: I can’t speak for other people who are interested in the space of microservices. The reason I am, is because of continuous delivery. People have written a code; we need to get it into production. What are all the things that we need to do to make sure we can get that software into production as quickly and safely as possible? Then we’re moving into now, can we automatically deploy our software? Do we have test automation? Maybe we are looking at more aggressive infrastructure automation to remove the manual processes in shipping our software.
That’s what I’ve spent a good chunk of my career doing as a consultant at Thoughtworks. I realised that quite often you do all of those things, and quite often, the architecture of the software itself was on of the big inhibiting factors.
I still consider a lot of those things that I just said to be the low hanging fruit, you should do those things first. Once you’ve done those things you often realise you’ve got a system architecture which actually inhibits the desire to release more frequently.
Fundamentally, the easiest way to release more frequently is often to reduce the scope of each release. If I can make the scope of my releases small, I’m less likely to make a mistake. If I do make a mistake, it’s easier for me to work out what the problem is. I was thinking okay, what can we do architecturally to allow us to reduce the scope of a release. This really comes on to this idea of independent deployability. You might have a system of 10, 15, 20, 100 microservices, but if I’ve got a one line code change that I want to make to one microservice, I should be able to make that change to that microservice and release and deploy that service, without having to change any other part of my system.
Kyle: Yes, I want to dig into that a bit. This idea of independent deployment is quite interesting to me. It’s something I’ve seen you talk about before, these concepts of cohesion and coupling. Obviously, they are concepts already in system level architecture. I’m interested in how they relate to microservices and distributed architecture.
Sam: If we want independent deployability, we don’t want coupling. There are different types of coupling and different definitions of coupling, but the coupling I am talking about here is that problem you encounter, where you end up wanting to make a small change that somehow ripples across the system. You’ve got units of code, broadly speaking, which are coupled together. I change one thing and I have to change lots of things: coupling and cohesion being quite strongly linked.
When you have code which is highly coupled, cohesion tends to be low.
Kyle: What does cohesion mean, in this context?
Sam: I struggle with the definition of cohesion more than anything else. The definition I heard from someone, I can’t remember who from, was that the code that changes together, stays together.
Sam: To give you a very simplistic example, if I wanted to change how I manage the life cycle of an invoice, I would like to not have to do that in 15 different parts of my system. Because it’s more work, there’s more stuff to go wrong, there’s more things that might need to be done in conjunction with one another.
Kyle: I think I see cohesion as things belonging together for a good reason. If coupling is things belonging together for a bad reason, then cohesion is kind of the good side of that.
Sam: I like that definition as well. The important thing about that linkage though, is that it does mean you can tackle this from either point of view. Sometimes the way you can decrease coupling is by improving the cohesion. Sometimes though, you might have code that is collected together not for any good reason. You need to push that code away to give you the ability to change those things more independently. You will avoid the big ball of mud problem.
That was my focus because actually, during the first edition I wrote Building Microservices back in 2014, it came out 2015. The thing that crystallised my thinking even more than that, though, is this concept of information hiving, which is heavily linked to those ideas.
Kyle: Can you talk to me about information hiving? What’s that?
Sam: Basically, it’s a foundational principle of modular architecture. So, if we think about coupling and cohesion, you could apply those at different levels, and they have different meanings at different levels. Information hiding is a concept that was developed to guide us in terms of how we think about modular decomposition of software.
I said that microservices are a type of distributed system. They are also a type of modular architecture. If you think about what a module should do, we take a big programme and break it down into modules. Those modules should allow us to reason better about the problem space because they are easier to understand. Modules also theoretically give you the ability to work in parallel because I can have different teams working on different modules. Hopefully, if I do modular architecture right, I get the ability to change one module without having to change the rest of the system.
These are basically the three criteria for a good module that David Parnas outlined in his paper on the criteria to be used in decomposing systems into modules.
Kyle: That’s impressive, that you can quote that off the top of your head.
Sam: It’s the only one. Parnas was looking at okay, we’ve got this idea of a module, how do we find good modules? In that paper, he actually compares two different techniques to find module boundaries. In one, he takes almost a workflow based approach where he has a programme. He basically models each step of that programme and has a module for each step. What he found was that process often ends up in lots of changes having to be made that ripple across those module boundaries.
Kyle: The common things across each step?
Sam: Yes. It’s also like (? 00:10:06) described this to me once as like a sausage factory. One, two, three, four… everything is in a line. Every now and then, you do get problems that just are that sort of thing, but most programmes aren’t really like that. So, what is the information hiding part? Well, the idea is that inside a module boundary what we are trying to do is hide that code which changes frequently. Code that changes a lot should be hidden from the outside world so that within a module boundary, the stuff that is hidden can change freely.
If I can change that without changing the interface of the module, then the rest of the programme will continue to work and continue to operate. If this already sounds familiar, this is what we think of as encapsulation object oriented programming.
If you think about it, hide stuff in the modules. Okay, great. What’s that got to do with microservices? Well, let’s talk about independent deployability. If I want to make a change to a microservice, deploy a new version of that microservice into the production environment, I’ve got to make sure that I haven’t broken the consumers of that microservice. Because if I have, I’ve broken the system. And guess what? I don’t then really have safe, independent deployability.
If I want to be able to make a change to a microservice and deploy it into production, I’ve got to maintain backwards compatibility with the consumers of my microservice. That basically means I need a stable interface. It turns out, if you get good at hiding the stuff that changes from the stuff that doesn’t, you end up with stable interfaces, you maintain backwards compatibility. You’ll find that independent deployability is much easier to deliver on as a result.
Kyle: I’ve seen a little bit of how to influence this design through testing at the code level but not a lot about how to do it at a system level. What do you think the role of testing and observability is in microservices? How do you have that design pressure to make sure you are building things that way?
Sam: I think it actually starts with a mindset shift. You’ve got to recognise that 99 times out of 100, a microservice exists to be called by other things, whether it be a user interface, an API gateway, another microservice, whatever it is. You want to adopt a consumer centric mindset. When I’m thinking about information hiding in the context of microservices, my default position is, I’m not exposing anything unless someone absolutely needs it.
The reverse mindset I often see is that people say ah, I’ve got this data in my database, I’m not sure what people might want in the future…
Kyle: Someone might need it.
Sam: Exactly. I might need it, so I’m just going to stick that on an API, job done.
Kyle: Yes, it’s simpler, it’s easier, and then later, it isn’t.
Sam: Yes. Coming back to encapsulation as an analogy, thinking that you’ve done anything around hiding of information, or hiding of implementation detail, you haven’t in that situation. This is the equivalent of in object oriented systems, having private fields and public getters and setters. That’s not encapsulation. I think it starts with a culture or a mindset which is outside in, consumer first. If you think about your outside in design, I think it’s much easier to come up with an interface which only exposes what needs to be exposed. That also has the added benefit that your interface should be easier to use.
See it like a user interface. It’s just the user of your microservice interfaces are fellow developers that work at your company. I think it’s crazy that you’re not interested in making your fellow developer’s lives easy. They know where you live, right?
Kyle: They can come over to your desk.
Sam: They can come over to your desk, they can slag you off over Slack. So, it’s a really subtle thing. Don’t expose anything unless you have to. You can always expose things later on. If you think about what people need upfront, have conversations with them if you’re able to, that places you in a pretty good space to start at least getting that separation from a design point of view.
Kyle: I think you are reflecting on something cultural here. There’s maybe this image of a company that is using microservice architecture, or an organisation having very isolated pods for each microservice. What you are telling me here is that they need to talk to each other.
Sam: Yes. A pod can only completely be completely isolated if it never calls anybody, and no one calls it. The secret of autonomous teams often isn’t isolated teams. Often, what you’re trying to do when you are thinking about your organisational design or your system design, is not eliminate all cross-team communication coordination. It is to reduce the amount to those areas where it’s really required. If I’m working on the payment gateway and I’ve got a separate team that manages the fraud stuff, we should have a conversation. That is a required coordination. That coordination between our teams we would expect to be reflected in our system architecture.
We can’t eliminate coordination between teams, what we can do though is reduce it as much as possible. If you see a service, for example, that depends on lots of other teams downstream, that can sometimes be a sign of a potential problem. If you think about the team owning that microservice, they are going to potentially have to coordinate with lots of other people downstream. Sometimes your system architecture can actually show you potential organisational challenges that you might have.
Kyle: Definitely, yes. Most people I find have this intuition but then don’t necessarily know where to go with it when they say oh, we changed this and then this broke. It’s so strange. That’s a sign that those two things are too close together. Whereas, oh, we changed something about payments and then fraud broke, and people are like, oh, that makes a little bit more sense. Those two should have some relation.
I think that’s your intuition speaking about cohesion, about what should be together, or at least close.
Sam: Yes. You might not like it to have broken but at least you can understand why it might have. I think if you look at the organisations that have hundreds of microservices, they solve these problems without realising it sometimes. If you want independent deployability, you need to get good at not breaking other people, however you do it. Part of it could be by having a design ethos like I just shared, thinking outside in, thinking about information hiding. That can help, but it also is beholden upon you to look at what adequate safety guards you would put in place to try and catch those breakages before they occur. Or before they are seen by an end customer, an end user of your software.
This is where testing can be highly useful. I’m a big fan of things like consumer driven contracts in this space. This is another reason why I quite like the use of explicit schemas for my service interfaces. That actually allows me to check for structural backwards compatibility before I’ve even checked my code in. I think that’s an area where not enough work is being done. People are having to do an awful lot of checking with automated tests to make sure they haven’t broken compatibility, where actually a lot of what they are checking for are just simple structural issues. There’s an awful lot of people running around saying oh, we don’t need a schema, and it’s like well, no, you just don’t have an explicit schema.
Kyle: Yes. Just because you haven’t said it out loud, it still exists. There’s always an interface when you’re talking to each other, even if you haven’t defined it, there is always an interface.
Sam: Exactly. Martin Fowler was kind enough to review the second edition of Building Microservices. He picked me up on this because I used to talk about schema-based versus schema-less interaction. He said no, there’s always a schema, it’s just whether it’s explicit or not. If you send say a Jason payload over http, and you say, I’ve got no schema, well, the consumer of that payload has a set of expectations about what structure they expect that Jason to be in.
The thing is, if you don’t have an explicit schema, all of that is just completely implicit. I think having that explicit schema is just about being clearer to everybody.
Kyle: It’s another example of being consumer-led, right? You’re being kind to people by telling them hey, this is what we do. You are being open and transparent with them, in a way that hopefully helps them work better.
Sam: And ideally, that explicit schema is going to be one of the concrete outcomes of the conversation you had with your consumers.
Kyle: Yes, let’s work on it together. Let’s work out what you need, let’s write it down so we can both look at it and go yes, I think this is right for a first pass.
Sam: It’s got limitations, right? Because all a schema can do is talk, primarily, about structure. It can say that this end point takes two integers and returns a third integer, and that end point is called ‘calculate’. But I think when you are thinking about compatibility checking, the schema does the easy bit and then the test can focus on the hard bit. If you don’t have the schema, not only do you lack clarity, but you are also asking your tester to do more work that they shouldn’t have to do.
Kyle: Just a quick shout out to Made Tech, our sponsor. Made Tech are software delivery experts mainly working in the public sector. I’ve been working at Made Tech for a few years now, and I’ve learned more than I did anywhere else in such a short space of time. All the while being supported by a bunch of lovely humans.
You can find us on Twitter, that’s @madetech which is M-A-D-E-T-E-C-H. We are hiring across a number of regional offices across the UK. Go to madetech.com/careers to find out more about that.
We’ve also got some books. Head to madetech.com/resources and get yourself some free books.
Let’s get back to Sam.
Kyle: I’m interested in what you think about the possible organisational approaches to this. I’ve seen organisations where they are starting down this journey, and a particular team is an advocate for microservices and starts to build a few out, starts to sell that to other people in the organisation. I’ve also seen organisations where it is very up front, and someone says, we’re doing microservices now.
Sam: Let’s talk about the oxymoron of the Dev Ops team.
Kyle: Yes, let’s.
Sam: Let’s ignore that one because if I see an organisation creating a Dev Ops team, it’s like, do you understand what this is? Because I don’t think you do.
Kyle: We can be angry about this for ten minutes if you like. I’ll enjoy that.
Sam: I’ve got other places I’m angry. When I see a team think that microservices are a good idea, a lot of the time you can have limited impact because you are only going to be able to affect your own local world, typically. If you are adopting microservices just within your team structure, on the one hand you are keeping that experiment small scoped, it’s much easier to assess the impact. Almost by definition, that team is willing and is going to put the extra work in to try and make this idea work.
The flipside is that as you get to microservices at scale, there are a whole host of problems that are very difficult for one team to manage. Some people refer to this as the microservice tax.
There’s a portion of your capacity that just goes into oh, there’s lots of new sources of problems that we’ve got with microservices. That inherently is taking bandwidth away from feature delivery.
Kyle: I think I would call this the, you don’t need Kubernetes yet problem. If you do that straight up, then suddenly that is half of your time.
Sam: Exactly. I still think that method could work very well, but in that situation, you are not trying to say that this is going to work for everyone. You’re saying this works for ourselves, do other people want to use this idea? If they do, now we should potentially look at what things we might want to do together, to use this technology more efficiently at scale. I think that model can work. The issue is that often, developers don’t always have the best reasons for using microservices because they often only see a small part of the picture anyway, and it’s often based on fashion.
Also, they are often not a great advocate for organisational change. I’m a great believer in Dan Pink’s Three Pillars of Motivation for employees; Autonomy, Mastery and Purpose. I like that. A lot of developers who are trying to adopt and advocate for microservices in their organisation have been completely and utterly starved of mastery for a long time.
Kyle: I really like this, yes, that’s a good point.
Sam: They’ve not been given an opportunity to grow their skills. So, they’re like, this is fashionable, my boss might have heard of it, let’s do it.
Kyle: And it’s interesting, right? All these difficult challenges that we’re talking about, they’re like, ooh, puzzles to solve! Let me at them! I get that.
Sam: If that’s the only reason you’re doing it… we’ve got an awooga going off, we’re going to have conversations going off, bad things are going to happen. Even if they are using it for the right reasons, they are often not able to advocate to the wider organisation, necessarily, for why this should be happening.
They say how can I sell my boss on microservices? Well, your boss doesn’t care about microservices. The users of your software absolutely don’t. What you need to talk about is the problems that your organisation is facing, and how this approach is going to solve those problems. Then, your boss will care. Anchor that change you are going to make in terms of a benefit that you are trying to bring. Also, be really honest with yourself that there might be simpler ways to solve those problems.
Kyle: Yes, I was going to say that it usefully forces that question, as well, right? If you ask yourself, why are we doing this, how do I sell this? And the answer is not really, maybe this doesn’t give us a lot of benefits, then that’s telling you something. Maybe this isn’t the right way to go.
Sam: Absolutely. It’s about context as well, right? The flipside is that organisational – We’re going to microservices! I had a client about three years ago, a big Tel Co. The same day I arrived, the CEO put out a statement – this organisation is massive – the CEO put out a statement internally saying, “We’re moving to microservices and cloud native technologies, so that we can increase speed of delivery and reduce cost of delivery in the next 12 months.”
The only things I could say to that statement were that you are not going to go faster, and it’s going to cost you a lot more money. Effectively, these are implementation details. Microservices are an implementation detail. So, when you get those top down statements, it’s worrying. Often, they are not rooted in any real understanding about what they can do.
I did a workshop in Sydney about five years ago. I had these three people that had come from a marine research agency. In Australia that’s obviously sharks and stuff. That’s cool, right? Sharks and jellyfish, cool! What’s that got to do with microservices? They said, well, look, our boss told us to come here. I was like, okay, cool, why does your boss want you here? They said, ask our boss, he’s sitting behind us. So, I say, boss, why did you send your three minions here? I didn’t use the word minions. He said, well, our CTO said we are doing microservices, so I figured we had better find out what they are. Thou shalt, you will.
This problem is made significantly worse by the fact that these enterprise organisations would often have enough cash and resources to then fall into the next trap. We’re going to go into microservices, right? What are all the problems that people might face with microservices? Right, let’s solve them all now.
Kyle: Oh, no.
Sam: That’s when the consultants rain down from the sky. Lo, there are Kubernetes clusters built —
Kyle: And a Dev Op team managing those Kubernetes clusters.
Sam: No, Platform teams. Platform teams are more expensive that Dev Ops teams.
Kyle: Ah, yes, the Platform team.
Sam: The phrase I like is, “A plan never survives first contact with the enemy”. It’s an old saying. The Mike Tyson version of that is, “Everyone has got a plan until they get punched in the face.” There is that thing that happens when those enterprise organisations think, we’re going to have all these problems, let’s build a platform to solve those problems. They spend six months doing it, it actually takes them a year, then the first team to try and use it just find it to be an absolute nightmare. That’s the other extreme.
Kyle: Again, it wasn’t consumer-led, right? There was no feedback loop, there was no user interaction with what was being build. Just classic software problems, right? It’s not specific to microservices, it’s a lesson that you can learn from other arenas, and it seems to be just as appropriate here. It’s really fascinating to me that the same patterns work.
Sam: Absolutely. It’s also boring. This is the same things that we went through when we were talking about Dev Ops or continuous delivery or Agile enablement. I’ve been through all of those. The thing that microservices has even more so than those other organisational change type things I talked about, is a very vibrant technology space sector, with lots of VC money going into big tech companies who are pushing their products.
With Agile, what could you push? Consultancy, but not product, really. Dev Ops? Hmm, they changed it into being a tool-driven thing, when it was never about tools.
Kyle: As cool as Dev Ops.
Sam: Dev Ops in a box. With microservices, there are some legitimate problems that you will encounter at scale that technology can help with, but that’s different than saying we need to solve those problems today.
Kyle: Yes, that’s a really good point, Sam. I hadn’t thought about that, that there’s money involved in breaking it, as usual.
Sam: Vast amounts. We do have to recognise that underpinning this, structurally, we’ve got a lot of money put into tech start-ups. That money has to be converted into product development and sales. You’ve also got a lot of existing companies that work around that infrastructural operational space that need to continue to have those revenue streams. They are effectively what some people call microservices washing. Now with added microservices.
As a result, this is a bewildering world for organisations to navigate. It can be an attractive proposition to just say I’ll buy that, and it will solve my issues for me.
Kyle: Yes, especially when you have these large scale problems that are going to require engagement, strategy, and long-term work to fix. Someone coming along and saying, we can fix this for you, it will cost this much. Done. It’s very, very tempting.
Sam: If you want a little bit of insight into how crazy this world can be if you’re trying to think about it from the point of view of someone trying to work out what tech they should use, go to Google, type in the words “CNCF Landscape”, and go look at the picture on that.
Kyle: I’m going to do that now because I’m curious.
Sam: Yes, go to landscape.cncf.io.
Kyle: Yes, I’ve got it.
Sam: Then what you’ll have to do is zoom out.
Sam: Then zoom out again, and then zoom out again.
Kyle: We’ll definitely put this link in the description because that is amazing.
Sam: This is not a diss on CNCF, it’s not a diss on the Kubernetes at all. This is actually a reflection of how good a job the Kubernetes and the CNCF have done in creating an ecosystem where multiple different tools can play. What you will see on this website is basically the different categories of tools, technologies, or problem spaces that you might have in the area of building cloud-native software. I would say narrowly cloud-native software where Kubernetes is the bit underneath everything. You’ve got a box around platform; you’ve got a box around continuous integration. You’ve got a box for service mesh. In the service mesh box alone, there are 14 different choices for a service mesh.
There’s a separate question which is, do you even need a service mesh?
Kyle: It’s interesting to me because microservices I came to as a person who heard about them, I heard the hype. My initial reaction was definitely scepticism. I think if there’s lots of noise about it, it’s probably worth looking at, but also, slow down, think about what the benefits are, run small experiments and this kind of thing. Which again, is boring. It’s a standard technique that has kept working.
Sam: Arguably it’s a scientific method, at some level. I would wind all the way back to, what are the challenges that you are facing right now, that you can’t solve any other way? Or what are the things that your organisation is going to do in the near future that you can’t work out how to do without doing microservices? Focus on that goal. I spent a lot of time working with organisations moving to the cloud. Often, the reason for moving to the cloud was around gaining access to excess capacity, rather than a six month or more lead time to get new hardware.
Kyle: Do you remember when we had to build hardware ourselves, like animals? It was awful.
Sam: Rolling around with network cables in the dirt, in the caves.
Kyle: Yes, ah.
Sam: I try to explain to people that my default position is that you shouldn’t use microservices unless there is a good reason.
Kyle: Do they instantly say to you, but you wrote the book! You’ve written a book! What do you mean, don’t use them?
Sam: My response to this is yes, I wrote the book, which is why you should pay attention to me.
Kyle: Yes, I’ve looked at it a lot. We’ve talked a little bit about it not being good for things to be top down, and also the challenge of doing this as a single person who is interested in it. How do you help everyone get involved? How do we make architecture a thing that is for everyone, rather than just for the technical architects?
Sam: I think there are some things that need to be – not top-down, maybe, but widely communicated. Those things that are important are company vision and culture. I think that does need to happen. That’s the underpinning of where we are going and what we’re doing.
In terms of the architectural stuff, I start thinking about the architectural vision. We need a vision for our system architecture.
Kyle: Where are we going and what should it look like eventually?
Sam: Yes. And what do we need today? For me, that comes back to understanding where the company is going. If you’ve got no sense of the company direction, or the culture of the company, how we are going to go about getting in that direction, those two things are inputs to your architectural vision. So then, your architectural vision is based on and rooted against how we deliver on where the organisation wants to go, and what the organisation needs us to do.
Those things have to be connected. This is why the ivory tower is a problem, right? Gregor Hope has this analogy of the architect elevator. He thinks about a skyscraper. If you are going up to the bigwigs in the top office to talk about where we are going as a company, then you go back down to talk to the developers about well, I think we might need a system architecture to help us do these things, what shall we do? A lot of this is stylistic and collaborative as an architect, but I think you need to have a clear architectural vision. The process of how you come up with that architectural vision should be collaborative, but someone needs to be responsible for making sure we have one.
Sam: It might be one or two or three people, whatever. But multiple people might have an input into that. If we think about the classical – you’ve got these independent teams owning their stuff, blah blah, architecture will often need to span across those teams. In those types of organisation, I see the role of that architectural vision as being people who have got mental bandwidth to actually spend time with the teams, and also to look across the whole piece.
That is almost the architectural function as an enabling team, right?
Kyle: Yes. If you want to do this, this is how your teams are going to have to implement it, and the consequences that is going to have, that kind of thing.
Sam: Yes, but also more that if you tell me we’ve got to do things like this, it’s going to make my life miserable, so let’s have a conversation about that. There’s a term I first heard from Frank Bushman. He talked about this idea of developer habitability. He’s not the originator of the term, but I heard it from him first. It’s an idea that as somebody who might decide or a recommendation around say an architectural approach, the decisions that you make have a significant impact on the daily lives of the people implementing that architecture.
Sam: You have a responsibility to understand the impact that those decisions have, and you need to engage with people about that. I’m being really clear here, it’s not the job of an architect to come up with the vision. The job of the architect is to make sure there is an architectural vision. A lot of the time, that may well be simple things. Let’s get ten of us together, tech leads from each of the teams, so we can have a big chat about it and come with an idea. Okay, well I’m going to polish this, I’m going to explain this to the rest of the company, and we’ll work together to refine it. That could be quite a collaborative process.
Kyle: Yes, and probably it’s better if it is. If you’re managing to get more people’s input, generally, that leads to a better result.
Sam: Absolutely. The challenge is often where this stuff falls in the cracks, where people just assume it’s going to happen naturally. It might when you’ve got 20 or 30 people. It won’t with 100. That’s the unhappy balance some people find. It’s not top-down, but there are still responsibilities for anybody who has to ensure that technical vision is being carried out.
Kyle: We’ve got a few questions that we ask everyone. I’m interested in who in the industry you are inspired by?
Sam: I’m going to differentiate between being inspired by and inspire to be like. I think they are two different things. There are people that inspire me a lot, and there are people that seem to have boundless energy to help build community and really magnify their impact and bring other people with them. I’m thinking about Kelsey Hightower in the Kubernetes community. People like Brigitte Cromley in the Dev Ops space, who did so much work helping get conferences going. People like Angie Jones, in the Java and the testing world, just seem to have endless enthusiasm to bring people with them. They don’t seem to be overly cynical or snarky either, which is a trait I’ve never had. They get things going. I’m amazed at the work they do; I think it’s fantastic, it’s greatly inspirational. I couldn’t do what they do because I’m not that person, but I really admire the work and I’m glad we’ve got people like that in our industry. We would be much poorer for it if we didn’t.
I think there are other people that have set out a path which is more like what I can do. Probably the person that comes to mind there is Martin Fowler. I was fortunate enough to work with him at Thoughtworks, but a lot of what he seems to care about — we work in quite different ways – but this idea of collecting information, trying to explain that information in a good way, and then sharing that information with a much wider ecosystem. Again, I’m trying to have a wider impact than that, but maybe through writing or presentations or workshops. Martin is someone that I’ve learnt a lot from, just observing what he’s done. He’s given me advice in the past around that. Martin is a more aspirational person that I could be more like than the other three. The other three don’t seem to have a built-in cynical view of the world and seem to have a lot more energy than I do.
Kyle: It takes all kinds, though.
Sam: Yes, absolutely.
Kyle: It takes some of us to say, slow down, don’t use microservices. It’s good. That’s really nice, I think that habit of what have I learned? Then sharing it back, is really good. Cool. What is the best thing that has happened to you this month?
Sam: Well, the book got released and it started getting in to people’s hands. The best thing this month was definitely me finally getting my hands on the second edition of Building Microservices.
Sam: I had some copies sent to me from the US that took a while to arrive. That was really nice. Just holding it in my hands was very good.
Kyle: Yes, something special about that.
Sam: It is. Realising just how badly I had failed at my goal of writing a smaller book.
Kyle: Has it got bigger in the second edition?
Sam: Oh, yes. It’s significant. People seem to really like it, which is great. I’m getting nice feedback. I had a lot of feedback along the way. The only reason the book is as good as it is, is largely because of the feedback I had from lots of people that gave their time and helped with the book. Martin Fowler, Sarah Wells and Daniel Bryant put a lot of feedback into that book as well, my reviewers. Actually, seeing it, and seeing people now talking about it and seeming to like it, that’s been a big kick. So, that’s been the best thing that has happened to me this month.
Kyle: That’s great. I think that nicely covers our do you have anything you want to plug question, presumably? Although you do have courses as well, do you want to talk about those?
Sam: I work as an independent consultant. I mix my time between writing the books, so there’s information about building microservices over at my website. I also run private classes for companies about microservices and do private consultancy. I work with a number of companies guiding them through a microservices journey.
Kyle: So, people can reach out to you if they need help with this sort of thing in more detail, you can come along and help those organisations?
Sam: Absolutely. I also run a lot of public events at the learning platform at O’Reilly. So, if you go to learning.oreilly.com, you can sign up for a free trial there. You can read my books, I do regular webinar style training there, as well as running events on the infrastructure, operations side of things. Every two months we have a half-day conference. Every month I do a podcast, ask the experts thing, we’ve got some really interesting people there. If you subscribe to that platform, you get access to all of that. If you are an ACM member, you get access to those platforms for free, as well.
Kyle: Great, we’ll link those in the description, with any luck. Finally, can you tell me one truth and one lie about yourself?
Sam: I’m a really good rock climber and I used to babysit for a punk rock guitarist. Well, the kids. He was old at this time; it was the kids.
Kyle: Okay, well we are almost at time. Thank you so much for coming along, thank you for talking to me, it was really interesting. As usual, I’ve learned a bunch from you, and I could talk for hours more. Thanks a lot.
Sam: Thanks so much for having me, Kyle.
Kyle: That wraps up my chat with Sam Newman. I actually took quite a few things away from this, but I will try and stick to just a couple. The first is, there are real pressures driving you towards microservices, when they might not be the right choice. I don’t think I’m the only one sceptical of hype cycles in technology, but it was really interesting to dive into the causes of that hype, the money etcetera.
The second is maybe a more practical point. The lessons we have learned around technology are still relevant to microservices. Start small, experiment, change course if you need to. Think about the goal of what you are doing. Those are all things that are applicable to technology in general. I’ve sometimes not focused enough on those times in the past, to my peril.
Kyle: Every other episode, it’s story time. Stories are for everyone. They are a great way to share our experiences, inspire people or make us laugh. This week we’ve got Clare, who is telling us a story about a neighbour.
Clare: This is another story about assumptions. It’s a familiar story. The neighbour who uses power tools at all hours, the banging, the drilling, the scraping. There can be something uniquely stressful about a loud thudding or a whiny drill that seems to seep through the very fabric of the house. And how come it feels like it’s right at my elbow, when I know that there are at least two walls between me and it?
Then there are the hammer thuds that are intermittent. Just when you think they are done, another one comes. When you are confronted with this disembodied, faceless nuisance, it can be easy to demonise the source. They are so inconsiderate, they don’t even care about me, don’t they know what it’s like?
Once you’ve been through those thought processes, you can get to the point where you don’t even think about starting a dialogue with them about it. Maybe there have been some minor confrontations in the past, and as a result you’ve convinced yourself there is not even any point in appealing to reason.
In my case, my neighbour had been doing some refurbishments on and off for 25 years. We didn’t have a brilliant relationship. So, when he started on another prolonged project, and I had a podcast episode to record, I kept thinking it’s no use, I’ll just have to decamp to the office and record the episode there. I just don’t know when drills and banging might start up next door. But then I had to record a rearranged interview at short notice. It wasn’t possible for me to go elsewhere, so I just had to cross my fingers and hope for no noise.
Then, ten minutes before the interview was due to start, yep, the drilling started. It turned out my neighbour was right outside my window. I had no choice; I was going to have to throw myself at his mercy. We had had a few friendly conversations in recent work. He had been keen to tell me what his latest plans were. So, I went outside and I just hovered at his elbow, because he was too loud for me to make myself heard. Finally, he noticed that I was standing there. He looked at me and I thought oh god.
I said, “Look, I’m really sorry, but is there any chance that you could just take a break for an hour? It’s just that I have to record an interview, and I won’t be able to get a decent recording with this much noise in the background.”
He looked at me, there was a big pause, and then a giant smile. “Of course.” He said. He was planning to stop for something to eat anyway. He went back into the house; I went back into the house. Just as I was about to start recording, there was more drilling. I didn’t know what to do at this point. But it only lasted for thirty seconds, then there was blessed silence. I recorded the interview with no hitch at all.
What’s the lesson? Well, it made me realise just how easy it is to demonise people. The less you communicate with them, the easier it is to imagine in your head that they are some kind of evil presence. It’s actually easier to cast them as a demon in your head than it is to go and just do the communication, because then you don’t have to make that effort. But that negative image in your head, and the idea that they are somehow an enemy, creates a whole lot more stress and misery.
You can convince yourself you know them because they are making an unpleasant noise, for instance, that is causing you discomfort. Therefore, you assume some kind of malicious intent. Of course, there is no reason to think that. All I had to do was be friendly, apply a little bit of charm, explain my situation, ask for sympathy and a bit of a favour, and that way he got to be the good guy, and we were both happy.
Kyle: For this final segment, Making Life Better, myself and my colleagues make some suggestions for small things we can do to help out. It could be helping the person next to us, or a little further afield. Today’s advice comes from Owen Piggot, one of our Senior Engineers, who has some advice on making the effort to help people who are struggling, staying calm and showing empathy. Owen, do you want to tell us a little bit more about that?
Owen: Sure. When we are all working together and trying to deliver something, it’s easy for one person to get lost from a group and be struggling silently to deliver a task that they may be working on solo, or as a pair. It’s really good to identify that because essentially, we are working on one complete picture for the customer. If one person is struggling, the whole thing can suffer. It can start to spread as well. If one person is stressed, another person can get stressed. I think the best way to deal with that is to stay calm yourself, and show empathy to those around you.
Kyle: Do you have any advice for bridging that gap, if you do notice someone who maybe isn’t themselves that day? Do you have any advice on taking that first step?
Owen: Generally, stick to statements that are questions. Try to get as quickly as you can, what the problem is out of them. Don’t tell them what you think the problem is. So that people can give you the information in their thoughts.
Kyle: I’m completely with you there. Most of the time it’s having that moment of realizing that there is someone who is there to listen. Not to provide a solution, but someone who is there to hear it all out and who cares.
Owen: Yes, exactly. And to help put problems into perspective, as well. It’s easy to have a small task that is going wrong. We think it might be the end of the world, or the end of the project. When you speak to somebody else, it’s suddenly put into perspective that we can solve this together, it can be fixed quickly. There are always problems, but there are always solutions.
Kyle: Couldn’t agree more. Thank you very much, Owen.
Owen: Thank you.
Kyle: Well, that’s almost it for this episode, but I have a little treat for you. We have three copies of Sam’s book to give away, kindly provided by O’Reilly, his publisher. These are DRM free eBook copies. If you would like to win one, send us your entry to firstname.lastname@example.org. You don’t need to include anything in the email body, but please make the subject line, “Building Microservices Giveaway”. If you prefer to read these details, that is all in the episode description. We will then pick three out of a hat and send you some free stuff.
You can find my guest Sam Newman, me – Kyle, and our sponsor Made Tech all on Twitter. Sam is at @samnewman. I am @kjdchapman and Made Tech are @madetech, which are all on the episode notes as well.
If you’ve enjoyed this, we would love a rating or review on whatever podcast platform you were using. It really helps us to hear how we are doing, and it will help more people find it, as well.
Thank you to Rose, our editor, Gina Cady, our virtual assistant, Viv Andrews, our transcriber, Richard Murray for the music – there’s a link in the description – and to the rest of our internal Made Tech team; Clare Sudbery, Jack Harrison, Karsyn Robb and Lara Plaga. We publish new episodes every fortnight on a Tuesday morning.
Clare: Hello. I’m just popping in to say because of the Christmas break we won’t be publishing an episode on Tuesday 21st December. Instead, the next episode will go out on Tuesday 4th January. I hope you all have a fantastic holiday and a great new year.
Kyle: We’ll see you next time, thanks for listening.
[Recording Ends]Back to the episode