Monitoring and alerting are key ingredients of the DevOps methodology put into practice. The systems that you run or the applications on them will generate a ton of telemetry data that you can use as actionable insight to inform your database monitoring and management strategy. Whether it's the number of connections currently being made to a database or the amount of RAM in use, this monitoring data can help us proactively prevent problems that can lead to poor customer experience
MongoDB Atlas, the database as a service platform, provides you deep visibility into database and platform monitoring, with full integration into services relied upon by DevOps teams such as Slack or PagerDuty. You can ensure notifications are also available via email or SMS.
Configuring proper monitoring and ensuring the correct members of your team are notified can keep your database performing at optimal levels, and increase uptime of your application. By monitoring your application for specific metrics that are key to performance, you can ensure you've properly scaled your managed database environment, and can re-scale at any time, so you minimize your costs accordingly.
In this blog post I'll discuss the reasons why configuring alerts is important, and show you how to:
- Review the existing MongoDB Atlas default alerts
- Set up Slack as an outbound alert endpoint
- Configure your alerts
- Notify your team via SMS
Why do we need Alerts for our DBaaS?
Even when using a Database-as-a-Service (DBaaS) such as MongoDB Atlas, there is still a need to implement a monitoring policy. MongoDB Atlas comes with a number of default alerts to notify your team of specific triggered events; of course these alerts are also configurable. This is a great start for most teams, but there maybe times in which a custom alert for your environment is required.
Proactive Maintenance
Being proactive is important to ensure the success of an application once it's been deployed to a production environment. If our MongoDB database were to run short on disk space, we certainly need to be notified before our disk is full, to ensure uptime. By configuring proper thresholds to alert us when our disk is near full, we can proactively provision more disk capacity, without interruption to the application. If your database is saturated by connections, you may have errors that require intervention by your team, and the need to scale your cluster. These types of preventative measures can be configured for many specific conditions relative to disk performance, IOPS, CPU and available memory. By utilizing alerts that notify you before your database cluster has reached any performance limit, you can plan a strategy for scaling up and down elastically as usage dictates.Change notification
If you are working in an environment with multiple administrators, there could be many people concurrently making changes to your application. This could require change notification alerts to inform your team and help avoid confusion about why modifications are occurring to your Atlas deployment. MongoDB Atlas makes it easy to configure a range of custom alerts that can be as granular as triggering when new users being added to a group.
Performance Indication
Custom alerts can be used to ensure you are getting the best possible performance from your MongoDB cluster. Some good examples of alerts that can help you maximize your database's performance include:
- Monitoring request latency per host
- Configure scanned objects alerts that notify you if queries are scanning more than a certain number of objects in a single query. This can assist in assuring appropriate fields are indexed
- If CPU, memory, or IOPS consumption is close to its limits
- Connections to your database are reaching a maximum threshold
The great thing about MongoDB Atlas is when you get these alerts, you can easily and seamlessly do something about them. If you need more CPU or RAM, just upsize your instances, or add more of them; more or faster disks, scale the disks; taking a break for a while, drop your costs to a minimum by removing excess capacity. And all of these reconfigurations can be accomplished with no application downtime. All the cluster needs do is perform one a rolling replica set election, orchestrated by a couple of mouse clicks (or API calls) in the clean Atlas interface.
Notifications
Monitoring and alerts are no good to us without the actual notification reaching us on our mobile device if we are on-call. If I don't get a notice about a potential incident, how am I ever to fix it? MongoDB Atlas comes with some solutions to solving this problem by permitting you a number of ways to notify your team.
The following are all outbound notification options for MongoDB Atlas:
- Email - to any standard email client
- SMS - Text message to alert your phone
- HipChat / Slack / Flowdock - chat notifications pushed via the MongoDB Atlas API directly to a specified channel
- PagerDuty - integration with on-call schedule and alerting
- Webhooks - Sends an HTTP POST request to an endpoint for programmatic processing
We can use any or all of the above to ensure that our team is notified of changes or potential issues with our MongoDB Atlas cluster.
You can find the configuration section for the service-based alerts in the "Settings" section of your MongoDB Atlas UI. You will need the API keys from your different service providers.
Notify via Slack
Slack has a direct integration with MongoDB Atlas that sends database alerts to the Slack channel of your choosing. To get started, make sure you have an existing Slack account with permission to add an API key.
In this example, we’ve created a channel for our team called "#alerts" that will receive notifications of any alert status provided by MongoDB Atlas.
Next, we will generate an API key to place in my Settings for my Slack integration. Go to the Slack API App Token generator, then click the "Create an App" button.
Select the team, and then title the name of the app. In this case we have just used "mongodb-alerts" to make it easy to identify the purpose of the created token in the future. Now we will need to add permissions for this, so let's click the "Bot Users" button, give the bot the username @mongodbalerts, and then enable it:
Once this is finished, we will click “Save changes”. Now let's finally "Install" the new app by clicking the "Install App" link. We'll be brought to a section that will ask us for final authorization; clicking “OK” will bring us to the API Token Section. We have obscured our full token, but you can see that we are provided with two different tokens. The token required for MongoDB Atlas is the "Bot User OAuth Access Token"; copy it and then open a new browser window.
In the new browser tab, bring up the "Settings Section" and then scroll to the bottom of the page and find the "Slack Integration" tab. The "#alerts" channel is where the token and the channel for notifications are specified.
After you click the "Post test message to Slack" you should see a notification into the #alerts channel:
Create an alert
MongoDB Atlas's comprehensive monitoring and alerting system is highly customizable for your team. You can get started with creating an alert by going to the Alerts section in your MongoDB Atlas UI.
Here we see our Alert History, a section which lets us know what alerts were triggered in the past, and who acknowledged this alert.
Next, go to the upper right hand corner and click the green "ADD" button. You'll now have a section permitting you to create an alert. If you wanted to create an alert for any time a host with the type “primary” has a connection number totaling over 500 per second, that is then pushed out to all members of our group via SMS and sends a notification to Slack, you'd follow the method as shown in this GIF:
-x4uv6e3wyc.gif)
Click SAVE and you're all set. Your team will be SMS'd any time your Atlas database has over 500 concurrent connections. You can now customize your metrics for your database to ensure you're always properly notified.
With the appropriate alerts now configured and our team's notification endpoints set, you'll be able to sleep soundly at night. By using correct monitoring settings and alerting appropriate team members, your application can remain resilient and provide a great experience for your users.
Get started with MongoDB Atlas today, try our free tier by signing up now!