Finding the balance between consistency and innovation
For many large organisations – in particular those in highly regulated industries such as finance – it often feels like developing and releasing new software and applications is harder and more complicated than it ever has been. However, at the same time, the industry is being flooded with new tools, technologies, and processes that aim to “speed up the delivery of new applications” and “make your organisation more secure, cost-effective, and adaptable”, says Jonathan Sidney, Cloud Platform Lead at Synthesis.
On the one hand, the ecosystem seems to have all the resources to enable quicker and more effective software development and delivery. Yet, on the other hand, thousands of software and IT professionals are struggling to get any valuable work done. What is clearly obvious is that there are some fundamental paradigm changes that are required to move forward in this industry.
“To understand how the industry evolved into this situation, a holistic view of the evolution of modern Software Engineering over the last 10 years would be instructive” explains Jonathan.
Prior to the explosion of the popularity of Cloud computing, Agile methodologies, and DevOps, software projects were delivered through several highly specialised teams. For example, there would be a team that only did database administration, a team that only built user interfaces, and so on. This meant that engineers could specialise their skills in very specific sub-domains and specialties. Silos and hand-offs were created that delayed the delivery of working software as well as introduced various bugs and errors because of these hand-offs and poor communications. It was precisely these issues that the DevOps revolution hoped to address when it introduced the concept of a team owning the end-to-end solution (from development all the way through to deployment and operation of a single system). If all the subdomains in an organisation were represented in a singular team focused on a specific value stream, then software could be built and maintained far more effectively. This can be seen in the fictional DevOps transformation described in the Phoenix Project.
Nevertheless, as Cloud technology has become more ubiquitous and concepts such as Infrastructure as Code (IaC) became industry standard practice, some holes in the DevOps methodologies have been uncovered. Even though teams should be comprised of people with different specialties that can cover all the various areas of the Software Development Life Cycle (SDLC), practice teams usually lack the skills to successfully cover all the various requirements. At a very high level, this situation has led to several damaging situations within teams that are illustrated in the book Team Topologies.
Jonathan highlights some of these situations below:
Unreasonably high Team Cognitive Load. Cognitive load can be thought of as the amount of knowledge that a human brain – and by extension a team of people – is managing at a given point. In the case of developing and implementing software, this cognitive load will include topics such as “How to implement a good user interface” and “How to design an appropriate database schema”. However, since teams are now expected to own the entire SDLC, they must also know how to implement new things such as deployment pipelines, network security, data encryption, cloud infrastructure development, logging, monitoring, and more. The result of this is teams that cannot handle all the tasks that they are expected to accomplish. Delivery, therefore, slows to a virtual crawl as teams struggle to learn and deliver on these diverse requirements that are so new and foreign to them.
Badly designed and implemented methods of inter-team communication. Since teams are finding it impossible to manage the cognitive load being placed on them, they are quickly reverting to the outdated strategy of having highly specialised teams providing specific skills and expertise. However, since this is happening in such an ad-hoc fashion, these specialised teams are overburdened and cannot service the rest of the organisation effectively. Yet again, delivery is slowed down even further. Additionally, the animosity and unhappiness that results from this slowdown are dangerous for the long-term success of the overall enterprise.
Lack of consistency and standards across teams. Once teams become fed-up with the slow pace with which specialist teams (such as security, DevOps, networking, etc.) action their requirements, they decide to “go it alone” and implement the various requirements themselves. The end-to-end ownership of a problem is admirable but, teams create a massive sprawl of tools, technologies, and processes that can oftentimes create far more headaches down the line. It makes it almost impossible for operations teams to maintain systems in the long run without significant rework and even the most generic problem will often have hundreds of different solutions across an organisation. Additionally, this lack of standards is an ideal breeding ground for cost overruns and security vulnerabilities to grow. This results in teams entering an organization and making mistakes that would have been easily avoided by people with those specialised skills.
With all this background, a possible solution can be introduced.
“As has been described above, the goal of any solution will be to balance the need for software teams to own the end-to-end solutions that they are building whilst adequately balancing the cognitive load placed on them” advises Jonathan.
In this breach, Team Topologies introduce the concept of an Internal Developer Platform (IDP). The goal of this tool is to fundamentally behave as an internal shopfront of best practices, patterns, and tooling that an organisation can expose to its teams.
Before discussing how an IDP can solve the problems mentioned above, Jonathan highlights a few characteristics of an IDP.
- The IDP is not simply a central repository of documentation, “how to’s” and “Read me’s”. It is a collection of working software templates that can be used repeatedly to account for the repetitive – yet complicated – foundations of a specific type of software architecture.
- The goal of any IDP must be to improve the experience of the teams and developers in each organisation. Just as product development places a massive focus on User Experience (UX), so too must an IDP place great importance on the developer experience (DevEx).
- Lastly, although the IDP is a product that is exposed to the organisation, it cannot be allowed to ignore the feedback from its users. As with any product, the feedback that the IDP receives from its “customers” is what it allows it to evolve with the organisation and continue to provide a useful service to the organisation.
Based on all this context, Jonathan explains how the IDP aims to solve the three critical issues listed above:
- The goal of the IDP is to provide patterns and templates of the basic – yet complicated – components of any modern, mature application. Additionally, since these patterns are working codebases, they allow a large amount of complexity to be abstracted away from teams. This results in a lower amount of expert knowledge required to build and configure these components. As such, teams can rely on a “lower” level of understanding of these components whilst still being guaranteed that they are being implemented at the highest level of quality and best practice. The upshot of all of this is that teams can focus more on delivering real business value through applications and systems, rather than focusing on the details of areas they are not well-versed in.
- As a rule, the modern SDLC is governed using Application Programming Interfaces (APIs). These allow the modern developer to harness a myriad of tools and systems using a very standard, easy-to-use, interface instead of writing all the code to implement them from the ground up. In this vein, an IDP should be – as far as possible – an API drive. In other words, a crucial requirement for the IDP is for it to be accessible through a very well-defined, easy-to-use method. This should often take the form of an API, but where this is not possible, it should be as easy to use and consistent as possible. Whilst it is important to stress that the IDP should never replace open methods of communication via emails, instant messaging, and the like; the ability to utilise the patterns and tools in the IDP in the easiest manner possible must be treated with the utmost importance when building it out.
- Arguably the most obvious positive outcome of an IDP is increased consistency across multiple teams. A key assumption of the IDP is that in most cases, the foundation, low-level configuration of many application environments can be similar and utilise very similar settings. From an organisational perspective, this allows for far greater consistency and adherence to standards between disparate value streams and teams. Additionally, it allows teams to deliver with far greater consistency and efficiency since there is evidence and learnings that can be harnessed from previous implementations.
The concept of the internal developer platform is still very new and is still maturing across the industry. However, it is also clear that it does provide a possible solution to some of the most pressing and difficult problems facing our industry. In its latest “State of DevOps Report”, the configuration management provider Puppet clearly demonstrates some of the initial positive outcomes resulting from the adoption of Platform Engineering and the IDP.
It is also critical to remember the words of Frederick P Brooks Jr who cautioned against believing in perfect solutions to the complexity and difficulty of software engineering in 1986: “There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity”. We must all be ready to adapt our preconceptions and solutions to meet the coming complexity and be ready to pivot in order to deliver as much value to all our customers, users, and stakeholders as efficiently, securely, and cost-effectively as possible, concludes Jonathan.
Author: Jonathan Sidney.
Ends
For more information on the innovative work Synthesis has done for its clients, contact us on 087 654 3300
About Synthesis
Synthesis uses innovative technology solutions to provide businesses with a competitive edge today. Synthesis focuses on banking and financial institutions, retail, healthcare and telecommunications sectors in South Africa and other emerging markets.
In 2017 Capital Appreciation Limited, a JSE-listed Fintech company, acquired 100 percent of Synthesis. Following the acquisition, Synthesis remains an independent operating entity within the Capital Appreciation Group providing Cloud, Digital, Payments and RegTech services as well as corporate learning solutions through the Synthesis Academy.
