March 29, 2023
min read
Internal Developer Portal: What It Is and Why You Need One
Internal Developer Portal Guide: What It Is and Why You Need One.
Internal Developer Portal Guide: What It Is and Why You Need One.
What does the term “developer portal” bring to mind? For most software engineers, it evokes an external-facing knowledge base aimed at third-party software engineers who consume your APIs. However, recently, a different kind of developer portal has emerged: the internal developer portal ("IDP").
The increasingly widespread adoption of distributed architectures means that vital sociotechnical information about your applications, services, environments and resources is fragmented across your tools, clouds and teams. This fragmentation, in turn, affects key performance indicators ("KPIs") like velocity, reliability, security, cloud costs and compliance, and it ultimately hurts the developer experience.
If you’re at one of the tens of thousands of companies facing pain from sprawl — be it a sprawling number of services, environments, cloud accounts and/or an increasingly heterogenous toolchain — an internal developer portal is essential to tame this problem.
In a nutshell, a well-architected internal developer portal is like a data lake containing all of the sociotechnical knowledge about your software development operation. On top of this data lake, internal developer portals have built scenario-specific workflow and analytics solutions to alleviate frequent pain points, all within a unified platform offering a consistent user experience (even though your underlying toolchain may be constantly evolving).
The goal of a developer portal, according to Gartner, is to “increase developer productivity, along with reducing the cognitive load.”
In this post, we’ll dive deeper into the pain points solved by internal developer portals and also unpack the solution landscape.
In a word, the problem internal developer platforms aim to solve is fragmentation. This is primarily due to the adoption of distributed systems. Despite their many benefits, they often split vital information among various tools, cloud providers and teams. This causes pain across the organization that affects velocity, reliability and other important measures.
For engineers, fragmentation slows velocity and harms reliability, as well as the developer experience. For instance, fragmentation causes developers and ops professionals to be less prepared to respond to incidents as crucial information, such as troubleshooting checklists, service owners and key information about the service, might be hard to locate. This leads to stress.
Fragmentation also makes it slower to ramp up new engineers given documentation, architecture, ownership and other key information is scattered. It can even lead to a loss of focus on what matters when KPIs, tickets and organizational improvement goals are scattered across tools and dozens or hundreds of dashboards.
Fragmentation can also lead to duplicated work: engineers can end up writing components that already exist in the organization because they weren’t easily discoverable.
Ops professionals also struggle because information about resources is fragmented across multiple clouds and cloud accounts, and cloud consoles are notoriously clunky to traverse. Further, it’s not always clear what environments and ultimately applications depend on a given resource as a result of information fragmentation. And when it comes to understanding cloud costs at the service level, there aren’t any good solutions today.
Finally, management and leadership also suffer from this fragmentation. It becomes more challenging for them to gather information and answer basic questions about their systems. They struggle when collecting metrics related to security, stability, architectural standards and drift, cloud costs and other necessary measures.
Often a project management organization is necessary to help management measure and manage compliance and related initiatives, and even then, a PMO team can only provide infrequent information to management and often requires engineering team interruptions to gather inputs.
The solution for the fragmentation problem is an internal developer portal (IDP).
An internal developer portal (IDP) is a self-service application and data store that lets developers and managers track and organize everything their engineering teams build and operate.
Consider it a living sociotechnical knowledge map of your operation: all of the knowledge about your applications, services, environments, resources — key information about them sourced from your entire toolchain and the people who build and operate them, in one discoverable place.
This universal catalog also serves as an abstraction layer to enable developers and managers to accomplish frequently occurring tasks without needing to log-in to many different tools, to manually source and analyze data and to accomplish this all on a self-service basis with flexible guardrails in place.
An internal developer portal (IDP) is a single source of truth on all matters regarding the development ecosystem in an organization.
An IDP has three primary functions:
An internal developer portal enhances your developer experience and improves outcomes for your department, primarily by alleviating the cognitive load associated with common tasks like on-call rotations, deployments, new-hire onboarding and resource lookups.
Further, by joining previously disparate data sets together, leaders can understand their operations with never-before-possible ease and granularity. Use it to measure and manage teams’ compliance with architectural, reliability and other standards. Ask and answer questions about security, staffing measures and more without interrupting teams. Ultimately, it is a tool to help the organization improve by enabling transparency.
IDPs have two main components: the discovery layer and the self-service layer. Let’s cover each one in turn.
As we’ve mentioned earlier, one of the main functions of an IDP is to serve as a universal catalog, a single source of truth regarding the organization’s software ecosystem. The catalog can’t realize its full potential unless it knows applications, services, environments and infrastructure. That way, the catalog can deliver benefits for virtually all actors in the organization.
As a living sociotechnical knowledge map of the team and system, an IDP allows developers to be more agile and efficient. They can quickly find crucial information about any aspect of an application, service, environment or resource, such as ownership, documentation, tickets, commits, recent CI/CD runs, reliability measures, APM and security metrics, costs, on-call rotations and more. This information is incredibly useful in organizations that run distributed systems in order to help engineers:
The universal catalog can also be of invaluable help to operations professionals. It offers ops teams:
IDPs enable leaders to measure and manage their organization’s compliance with internal best practices in never-before-possible detail and speed. Leaders can analyze metrics and programmatically organize improvement initiatives related to:
These analytics and initiatives (sometimes called scorecards) are powerful tools to help organizations understand and improve, all without requiring a project management organization to interrupt teams to gather inputs and then manually compute results.
Another, less frequently used feature of an IDP is a scaffolder. Organizations can codify best practices around the creation of a service via a template or blueprint while still allowing developers the flexibility to go “off-roading.” The scaffolder then enables a developer to select an appropriate template, make configuration choices and stand up the new service with key resources like CI/CD, databases, etc., all properly connected in code automatically.
It can be a time saver for organizations building bespoke new services, though inquire how frequently this applies in your case. IDPs may serve as an abstraction layer for a scaffolder and/or merchandise blueprints in the catalog so developers can easily discover and use them (via the abstraction layer or directly in a scaffolder).
Finally, some platform engineering teams are investing in making it simpler to manage their infrastructure. The more infrastructure you have, across many accounts and clouds, the more value your organization will get from integrated management capabilities.
These capabilities may enable you to change application or service configuration, add resources to a service, create a new environment and perform deployment by tag and rollbacks.
Many companies have invested in infrastructure-as-code solutions such as CloudFormation and Terraform as well as CI/CD systems to handle these tasks, and IDPs can expose those controls right in the portal via plugins.
Much like with scaffolding, platform engineering teams should consider the frequency of these types of changes, compatibility with their infrastructure-as-code investments and how many users within the overall engineering team are affected by these scenarios when planning their investments.
It’s possible to create your own portal from scratch, though that would probably be an arduous endeavor, requiring integration with many third-party APIs and services
As is common in the software industry, the main question comes down to building versus buying when it comes to picking an IDP solution.
So, the first option we’ll analyze is at one extreme of this dilemma: building your own. This path would offer you the most flexibility to tailor a solution to the needs of your organization. However, this would be an arduous endeavor requiring building integrations with many third-party APIs and services as well as the IDP application. Therefore, this would be a large level of effort that would result in a higher total cost of ownership and significantly delayed time to market relative to available alternative approaches.
Since IDPs are, in general, quite complicated pieces of software, there aren’t many open source options to choose from. The two most popular are Backstage and Gimlet. The first one, however, is more of a DIY platform for building developer portals, which means you’ll still need to spend quite a lot of time getting something that actually benefits your company.
It also lacks true infrastructure awareness, struggles in terms of search quality, requires significant effort to configure and maintain, and its strongest attribute may be its sprawling vision for the future.
In many ways, Backstage is reminiscent of how Jenkins was novel at first, but was eclipsed by paid solutions with superior features and service. You can read more about evaluating Backstage in this strategic guide.
Gimlet is quite popular too, but it focuses on Kubernetes-based environments. Both are great tools — don’t get me wrong — but if you’re looking for a faster time to solution via something ready to use and feature-rich that likely offers a lower cost of ownership and superior return on investment versus costly-to-configure-and-maintain open source software (see this post for more), consider the third option.
The final, often economically superior option, is to leverage an existing, ready-to-use solution like configure8.
When considering a ready-to-use solution, be sure to consider the following dimensions:
An internal developer portal is an emerging product category that aims to solve the problem of information fragmentation faced by engineering organizations that run a distributed system. The heart of this solution lies in the breadth and depth of the data lake it is built on, and then the quality of individual scenario-specific solutions to address the needs of your internal developers, leadership, DevOps and FinOps teams. Ready-to-use solutions often offer the most compelling return on investment and features to improve the developer experience and leadership’s ability to help teams understand and improve along important dimensions.
Copyright © 2024 configure8, Inc. All rights reserved.
.png)
