Leveraging MongoDB Atlas in your Internal Developer Platform (IDP)

DevOps, a portmanteau of “Developer” and “Operations”, rose to prominence around the early 2010s and established a culture of incorporating automated processes and tools designed to deliver applications and services to users faster than the traditional software development process. A significant part of that was the movement to "shift left" by empowering developers to self-serve their infrastructure needs, in theory offering them more control over the application development lifecycle in a way that reduced the dependency on central operational teams.

While these shifts towards greater developer autonomy were occurring, the proliferation of public clouds, specific technologies (like GitHub, Docker, Kubernetes, Terraform), and microservices architectures entered the market and became standard practice in the industry. As beneficial as these infrastructure advancements were, these technical shifts added complexity to the setups that developers were using as a part of their application development processes. As a result, developers needed to have a more in-depth, end-to-end understanding of their toolchain, and more dauntingly, take ownership of a growing breadth of infrastructure considerations.

This meant that the "shift left" drastically increased the cognitive load on developers, leading to inefficiencies because self-managing infrastructure is time-consuming and difficult without a high level of expertise. In turn, this increased the time to market and hindered innovation. Concurrently, the increasing levels of permissions that developers needed within the organization led to a swath of compliance issues, such as inconsistent security controls, improper auditing, unhygienic data and data practices increased overhead which ate away at department budgets, and incorrect reporting. Unsurprisingly, the desire to enable developers to self-serve to build and ship applications hadn't diminished, but it became clear that empowering them without adding friction or a high level of required expertise needed to become a priority.

With this goal in mind, it became clear that investment was required to quickly and efficiently abstract away the complexities of the operational side of things for developers. From this investment comes the rise of Platform Engineering and Internal Developer Platforms (whether companies are labeling it as such or not).

Platform engineering and the rise of internal developer platforms

Within a developer organization, platform engineering (or even a central platform team) is tasked with creating golden paths for developers to build and ship applications at scale while keeping infrastructure spend and cognitive load on developers low. At the core of the platform engineering ethos is the goal of optimizing the developer experience to accelerate the delivery of applications to customers. Like teaching someone to fish, platform teams help pave the way for greater developer efficiency by providing them with pipelines that they can take and run with, reducing time to build, and paving the way for greater developer autonomy without burdening developers with complexity.

To do this, platform teams strive to design toolchains and workflows based on the end goals of the developers in their organization. Therefore, it’s critical for the folks tasked with platform engineering to understand the needs of their developers, and then build a platform that is useful to the target audience. The end result is what is often (but not exclusively) known as an Internal Developer Platform.

What is an IDP?

An IDP is a collection of tools and services, sourced and stitched together by central teams to create golden paths for developers who will then use the IDP to simplify and streamline application building. IDPs reduce complexity and lower cognitive load on developers - often by dramatically simplifying the experience of configuring infrastructure and services that are not a direct part of the developer's application. They encourage developers to move away from spending excess time managing the tools they use and allow them to focus on delivering applications at speed and scale. IDPs enable developers the freedom to quickly and easily build, deploy, and manage applications while reducing risk and overhead costs for the organization by centralizing oversight and iteration of development practices.

An IDP is tailored with developers in mind and will often consist of the following tools:

1. Infrastructure platform that enabled running a wide variety of workloads with the highest degree of security, resilience, and scalability, and a high degree of automation (eg. Kubernetes)

2. Source code repository system that allows teams to establish a single source of truth for configurations, ensuring version control, data governance, and compliance. (eg. Github, Gitlab, BitBucket)

3. Control interface that enables everyone working on the application to interact with and manage its resources. (eg. Port or Backstage)

4. Continuous integration and continuous deployment (CI/CD) pipeline that applies code and infrastructure configuration to an infrastructure platform. (eg. ArgoCD, Flux, CircleCI, Terraform, CloudFormation)

5. Data layer that can handle changes to schemas and data structures. (eg. MongoDB Atlas)

6. Security layer to manage permissions in order to keep compliance. Examples of this are roles-based compliance tools or secrets management tools (eg. Vault).

While some tools have overlap and not all of them will be a part of a specific IDP, the goal of platform engineering efforts is to build an IDP for their developers that is tightly integrated with infrastructure resources and services to maximize automation, standardization, self-service, and scale for developers, as well as maximizing security whilst minimizing overhead for the enterprise.

While there will be many different terms that different organizations and teams use to refer to their IDP story, at its core, an IDP is a tailored set of tech, tools, and processes, built and managed by a central team, and used to provide developers with golden paths that enable greater developer self-service, lower cognitive load, and reduce risk.

How does MongoDB Atlas fit into this story?

Developers often cite working with data as one of the most difficult aspects of building applications. Rigid and unintuitive data technologies impede building applications and can lead to project failure if they don’t deliver the data model flexibility and query functionality that your applications demand. A data layer that isn’t integrated into your workflows slows deployments, and manual operations are a never-ending drag on productivity. Failures and downtime lead to on-call emergencies – not to mention the enormous potential risk of a data breach.

Therefore, making it easy to work with data is critical to improving the developer experience. IDPs are in part about giving developers the autonomy to build applications. For this reason, MongoDB’s developer data platform is a natural fit for an IDP because it serves as a developer data platform that can easily fit into any team’s existing toolstack and abstracts away the complexities associated with self-managing a data layer.

MongoDB’s developer data platform is a step beyond a traditional database in that it helps organizations drive innovation at scale by providing a unified way to work with data that address transactional workloads, app-driven analytics, full-text search, vector search, stream data processing, and more, prioritizing an intuitive developer experience and automating security, resilience, and performance at scale. This simplification and broad coverage of different use cases make a monumental difference to the developer experience.

By incorporating MongoDB Atlas within an IDP, developer teams have a fully managed developer data platform at their disposal that enables them to build and underpin best-in-class applications. This way teams won’t have to worry about adding the overhead and manual work involved in self-hosting a database and then building all these other supporting functionality that come out of the box with MongoDB Atlas.

Lastly, MongoDB Atlas can be hosted on more cloud regions than any other cloud database in the market today with support for AWS, Azure, and Google Cloud.

How can I incorporate MongoDB Atlas into my IDP?

MongoDB Atlas’ Developer Data Platform offers many ways to integrate Atlas into their IDP through many tools that leverage the MongoDB Atlas Admin API. The Atlas Admin API can be used independently or via one of these tools/integrations and provides a programmatic interface to directly manage and automate various aspects of MongoDB Atlas, without needing to switch between UIs or incorporate manual scripts. These tools include:

• Atlas Kubernetes Operator

• HashiCorp Terraform Atlas Provider

• AWS CloudFormation Atlas Resources

• Atlas CDKs

• Atlas CLI

• Atlas Go SDK

• Atlas Admin API

With the Atlas Kubernetes Operator, platform teams are able to seamlessly integrate MongoDB Atlas into the current Kubernetes deployment pipeline within their IDP allowing their developers to manage Atlas in the same way they manage their applications running in Kubernetes. First, configurations are stored and managed in a git repository and applied to Kubernetes via CD tools like ArgoCD or Flux. Then, Atlas Operator's custom resources are applied to Atlas using the Atlas Admin API and support all the building blocks you need, including projects, clusters, database users, IP access lists, private endpoints, backup, and more.

For teams that want to take the IaC route in connecting Atlas to their IDP, Atlas offers integrations with HashiCorp Terraform and AWS CloudFormation which can also be used to programmatically spin up Atlas services off the IaC integrations built off the Atlas Admin API in the Cloud environment of their choice.. Through provisioning with Terraform, teams can deploy, update, and manage Atlas configurations as code with either the Terraform Provider or the CDKTF. MongoDB also makes it easier for Atlas customers who prefer using AWS CloudFormation to easily manage, provision, and deploy MongoDB Atlas services in three ways: through resources from the CloudFormation Public Registry, AWS Quick Starts, and the AWS CDK.

Other programmatic ways that Atlas can be incorporated into an IDP are through

• Atlas CLI, which interacts with Atlas from a terminal with short and intuitive commands and accomplishes complex operational tasks such as creating a cluster or setting up an access list interactively

• Atlas Go SDK which provides platform-specific and Go language-specific tools, libraries, and documentation to help build applications quickly and easily

• Atlas Admin API provides a RESTful API, accessed over HTTPS, to interact directly with MongoDB Atlas control plane resources.