RFC: Heroku-like experience for Internal Developer Platforms with Backstage

[Taras Mankovski]

Summary

This RFC proposes creating a Backstage plugin for distributing a CLI and a Backstage UI plugging for managing runtime environments, secret management, environment management, and observability tooling. The CLI and Backstage UI will use a pluggable Platform API.

Motivation

Heroku-like experience in the Cloud Native ecosystem has been an elusive mirage. Those familiar with the experience of using Heroku often refer to it as the experience developers want while using their Internal Developer Platforms. Many innovations made Heroku experience memorable. Developers could create their applications and push code to a Git repository, where the code would be automatically built and deployed to the Heroku platform. When it took months for Ops to set up a database, Heroku gave developers the ability to add databases in minutes. Developers could add a database to their application, and the Heroku platform would automatically make the connection string for the application available through the environment variables. Developers could manage secrets for their applications that would be added as environment variables without storing sensitive information in the source code. Heroku made running ambitious applications easy and enjoyable. This is why it's been a lasting inspiration for developers building Internal Developer Platforms in the Cloud Native ecosystem.

Many features of the Heroku platform are now a standard architecture of Internal Developer Platforms. Most IDPs allow developers to push their code to a Git repository provider such as GitHub, GitLab, or BitBucket. When a new commit is pushed into a branch, it's automatically built by a Continuous Integration system like GitHub Actions, GitLab CI or BitBucket Pipelines. The CI workflow creates a container image and pushes it into a container registry. A Continuous Deployment service monitors the container registry and automatically deploys the container to the platform. Once the application runs on the platform, it's autoscaled by Kubernetes, which is managed by the platform engineering team. The platform is instrumented with observability tooling that allows developers to see their logs and metrics in tools like Kibana and Prometheus. Mass adoption of git-based workflows makes it possible to provision resources like databases using configurated stored as code with standards PAWS - Platform-Agnostic Workload Specification. Modern IDPs can now automatically provision resources based on configurations in a paws.yaml file.

The Cloud Native ecosystem matured to a point where in some respects, the developer experience of using the platform is superior to that of Heroku, but there is still a gap. This gap is in the interfaces that developers have to use to interact with their Internal Developer Platforms. Heroku allowed developers to interact with their platform using a single CLI and a unified UI. CLI and UI provided feature parity in features that were important to developers. A developer could create an application, clone environments, set environment variables, and view logs from the CLI. The CLI experience was project-code-centric - the project's working directory sets the CLI's context. When a developer invoked commands from the CLI, those commands automatically assumed the Heroku application based on the code in the working directory.

The experience of interacting with the IDP from the CLI with a context set by the working directory and the ability to perform all of the same operations via a Web UI is the last remaining piece in realizing a Heroku-like experience on Internal Developer Platforms.

The emergence of Backstage as a standard developer portal is the last missing piece in the Heroku-like experience puzzle. Before Backstage, each company created its portal using a variety of technologies and different architectures. Creating a reusable toolkit that companies could use off the shelf without a consistent architecture is difficult. Backstage provides a UI framework that can be used to create a reusable user experience in a web application and a pluggable server architecture that can be used to create a reusable server backend for the web application and CLI tool.

Approach

Features

The most common features of a Heroku-like DX are the ability to manage environments, manage secrets, view logs and metrics, and manage releases.

Managing Environments

Runtime environments for services are an important part of the developer's everyday life. Developers need to see where the application is running, know how to access the service, manipulate secrets and see logs for each environment. They need to be able to create ephemeral environments that automatically deploy from branches and provision new resources. In all of these use cases, the runtime environment is the context for each activity. This necessitates for Backstage to treat Runtime Environment as a first-class concept.

Manage Secrets

Most services require credentials and tokens to connect to external integrations. Each platform has a different way of managing these secrets. Some Internal Developer Platforms, especially those going through change, may have multiple secret stores or a different secret store for each environment. Managing these secrets can be a time consuming and error prone activity. To simplify the process of managing secrets and reduce error, developers should be able to manage secrets from the Backstage portal or the CLI.

View logs and metrics

When a service is failing or when doing troubleshoot, developers need to be able to find all of the information associated with a service without jumping around different systems. Logs and metrics are critical to debugging and learning about runtime behaviour of a service. This information needs to be available at their finger tips in Backstage or in the command line interface. They should be able to switch between different components and see the observability data immediately without doing any digging.

Manage releases

Shipping software is perhaps the most important activity in the entire software development lifecycle. How that software makes it to the users varies from organization to organization, from team to team and even from software to software. Empowering developers to ship their software in an reliable and fail-safe way without introducing unnecessary meetings and obstacles is perhaps one of the biggest contributors to improving organizational DevOps maturity and the experience of shipping software on an Internal Developer Platform. Backstage and the CLI tool have the opportunity to make it easier for organizations to ship their software by providing a flexible interface to deployment requirements on the Internal Developer Platform.

Architecture

One of this problem's biggest challenges is that each Internal Developer Platform is different. One platform may use AWS Secret Manager, and another may use Hashicorp Vault. Depending on the maturity of the IDP, it's not unusual to see two different technologies being used for the same purpose on different projects. Even though the goal is to standardize, not two platforms are alike, and their implementation changes over time. This requires designing the architecture in a way that allows maximizing reuse while being flexible to the decisions of each Internal Developer Platform.

We can define a clear interface between the client and the server. The clients - the Backstage UI components and the CLI, will use a clearly defined schema to communicate to the platform API. The platform API will expose an API that implements the schema, but the implementation of each Platform API will be platform specific. For example, regardless of whether the platform uses Hashicorp Vault or AWS Secrets Manager, the UI components and CLI will make the same requests to the Platform API. The Platform API will be responsible for making appropriate calls to the Hashicorp Vault or AWS Secrets Manager, depending on the platform's use.

Adaptors

Some tools in the Cloud Native ecosystem offer a lot of value when building an Internal Developer Platform. These tools typically provide their Platform APIs and accelerate the creation of a modern IDP. Humanitec is a perfect example of such a tool. It does the hard work of dynamically generating configuration, provisioning resources, and deploying applications. For many companies, platforms like Humanitec make up a big percentage of their Internal Developer Platform. The Adapter API will allow these tools to provide a shortcut to implementing the Platform API. Each Adapter will be able to provide one or more features.

TODO

1. Research: How do we compile the binary?
   1. Download the deno binary from npm
   2. Checks for existence of compiled files
   3. If not there, compiles the files
2. How do we authenticate it?
   1. Login command
   2. Open browser
   3. Follow authentication flow
   4. Get token
   5. Store it on file system
   6. Use it to make requests
3. Implement Commands
   • version - version of the CLI, URL of Backstage instance, version of Backstage used to compile it
   • project
     • info - read catalog info entity idp info [--component]
     • environments
     • variables
     • releases
   • templates

• search

Workflows

1. Install Plugin
   • Frontend
     • Download CLI Page
       • provide curl command
       • execute curl command from local machine
       • Download the binary (make it executable)
   • Backend
     • Platform REST API

CLI Commands

• version - version of the CLI, URL of Backstage instance, version of Backstage used to compile it
• project
  • info - read catalog info entity
  • environments
  • variables
  • releases
• templates
• search

Unknowns

1. How do we compile the binary?
   1. Download the deno binary from npm
   2. Checks for existence of compiled files
   3. If not there, compiles the files
2. How do we authenticate it?
   1. Login command
   2. Open browser
   3. Follow authentication flow
   4. Get token
   5. Store it on file system
   6. Use it to make requests
3. How do make determine context server configurable?
   • As a developer working on the Backstage portal, I need to be able to configure how the CLI determines what is the component associated with the current working directory. Ideally, this would happen without having to release a new version of the CLI.