You may be using MongoDB Cloud Manager to monitor, back up, and automate your MongoDB deployment. If you like some of the functionality in Cloud Manager but were looking for a more managed experience, you can migrate your deployment to MongoDB Atlas, the database as a service from MongoDB, Inc.
In this post, I’ll walk you through migrating a node-todo app running on a MongoDB deployment managed by Cloud Manager and hosted on AWS to a fully managed Atlas cluster, also hosted in AWS. Node provides the RESTful API. Angular provides the frontend and accesses the API. MongoDB will store the todo items.
Before we jump into the tutorial below, it’s worth pointing out that there are a few options for how you approach running your deployment on AWS:
- Rely on MongoDB as a service: MongoDB Atlas takes care of the operational heavy lifting so you can focus on building your apps. This is the easiest way to run MongoDB on AWS as many of the operational recommendations and best practices are “built-in”.
- Use management tools designed for MongoDB: Cloud Manager is a powerful management tool that simplifies and automates tasks like monitoring, backups, installation, upgrades, and more.
- Manage everything yourself: If you have an unusual configuration or you want to “roll your own,” there are many best practices for running on AWS.
Now, let’s dive in.
Cloud Manager (Migration Source)
Tip: Before you begin the migration process, check the upgrade path and compatibility of your existing MongoDB deployment in the "Live Migration" documentation.To get started, here’s an overview of my existing application infrastructure:
In a production app, you will likely have more than a single web front end server, and possibly a load balancer. To ensure we're working with an easy example, I will stick with just a single instance for my application.
I will be migrating my data from my existing database deployment managed by Cloud Manager to a MongoDB Atlas cluster. With MongoDB Atlas, a failure to one of my nodes will no longer require manual intervention to recover to a fully operational three-node replica set. This is a major change as Cloud Manager does require you to inspect and fix your failed nodes manually.
Tip: Your migration source must be an existing MongoDB replica set. You cannot utilize Live Migration on a standalone. Please review the "Convert a Standalone to a Replica Set" documentation for instructions.
MongoDB Atlas (Migration Target)
MongoDB Atlas provides you with almost all the features included in Cloud Manager such as comprehensive monitoring and continuous backups with point-in-time recovery. It also includes features that are not included in Cloud Manager such as automated deployments, elastic scalability with the push of a button, automated bug fixes and security patches, and much more.To get started, I have created a new MongoDB Atlas Group and launched my new cluster, the target of my migration.
I have established a peering connection between my existing host VPC in AWS and my unique MongoDB Atlas VPC. This will ensure that our network connection from our application to our Atlas cluster uses the same private IP network.
I'll go ahead and then provide a whitelist entry to my host server's private IP and then connect to our MongoDB Atlas cluster via the shell to ensure we have everything working properly:
[root@ip-172-16-0-77 ~]# host records-shard-00-00-x8fks.mongodb.net
records-shard-00-00-x8fks.mongodb.net is an alias for ec2-52-44-134-173.compute-1.amazonaws.com.
ec2-52-44-134-173.compute-1.amazonaws.com has address 192.168.254.198
[root@ip-172-16-0-77 ~]# mongo "mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/test?replicaSet=records-shard-0" --authenticationDatabase admin --ssl --username admin --password
MongoDB shell version v3.4.4
Enter password:
connecting to: mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/test?replicaSet=records-shard-0
2017-05-04T15:29:57.901+0000 I NETWORK [thread1] Starting new replica set monitor for records-shard-0/records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017
MongoDB server version: 3.4.4
records-shard-0:PRIMARY>
I have ensured that the EC2 instance hosting my app is able to access my new MongoDB Atlas database and am now ready to take the next steps.
Prepping for the Migration
I will need to provide appropriate network connectivity between the MongoDB Atlas Live Migration service and my existing deployment managed by Cloud Manager. I’ll also need to modify the default MongoDB settings related to the network interfaces and make sure that they are listening to traffic on our mongod port.Open the Firewall
In the AWS Security Group containing the database servers managed by Cloud Manager, allow access to two IP ranges used by the MongoDB Atlas migration service.
You can see I’ve done so in the photo below, granting Atlas access.
Allow Cloud Manager to accept remote network connections:
By default, MongoDB does not bind to all of your server's network interfaces to restrict unwanted network-based intrusions. I will allow the MongoDB Atlas migration service to access my deployment managed by Cloud Manager via the public network.
To allow remote connections, go to the Cloud Manager Group and click the "Modify" button to the right:
In the modify section, I’ll find the "ADVANCED OPTIONS" menu to modify the bindIP of my MongoDB cluster. In this case, I've entered: 0.0.0.0, allowing my server being managed by Cloud Manager to accept network connections to all ethernet interfaces.
Next, I will click the apply button in the blue bar at the top of my Cloud Manager window. Cloud Manager will deploy rolling configuration changes to my cluster. When the configuration change is complete, I can validate my web host EC2 instance can still connect to the Cloud Manager database server:
[root@ip-172-16-0-77 ~]# mongo mongodb-app-2.jaygordon10gen.9433.mongodbdns.com:27001
MongoDB shell version v3.4.4
connecting to: mongodb-app-2.jaygordon10gen.9433.mongodbdns.com:27001
MongoDB server version: 3.4.4
myapp:PRIMARY>
Start the Migration to MongoDB Atlas To start the migration, I can kick off the process by finding the "Migrate Data to this Cluster" button in the tile representing my Atlas cluster.
A new window will now provide the next steps to take to begin the import of data from my source cluster. I will be prompted for information regarding the hostname of my source primary as well as any user authentication I may have enabled on my original database.
Ensuring that live updates are synced to my target cluster
Before I actually click the "I'm ready to migrate" button to begin the process, I am going to simulate some web traffic to replicate a live website as closely as possible.
I have recreated the experience one would have doing a live cutover by simulating some active traffic that posts to my API endpoint for my app. Once the cutover to my new database is completed, I should still see POSTs to my endpoint continue. Without doing this test, I would not be able to fully confirm a live migration from my source database to my new cluster in MongoDB Atlas. I wrote a curl command I'm just running through watch on my local laptop:
bash-3.2$ watch -n 0 bash looper && watch -n 0 bash looper
The script, looper is just a bash script that runs curl against the todo API:
curl 'http://todo.jaydestro.org/api/todos' -H 'Origin: http://todo.jaydestro.org' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, */*' -H 'Referer: http://todo.jaydestro.org/' -H 'Cookie: keystone.sid=s%3AhcBc4UboMZ9snnudz0P-Pvy6nDNOMLCm.uNzl8r8Y8kKEIkLl6%2FelJjxVPDtdqijL9b6Yqg5ZAO8' -H 'Connection: keep-alive' --data-binary '{"text":"bdsfj"}' --compressed
Now that I’ve prepped all the right parts, it's time to click "I'm ready to migrate" and get to work!
Migrate to MongoDB Atlas
MongoDB Atlas asks me for a few details about my source. Let’s say for the sake of simplicity that I wasn’t sure about authentication when I built my app and only used network-based security. There won't be a username and password, nor SSL from the Cloud Manager side to enter. I can just enter the hostname of the primary in my Cloud Manager cluster and then click "Validate" at the bottom of the window.
My primary is the "mongodb-app-2" node — this will be the primary I provide to the live migration service. MongoDB Atlas will read our oplog and migrate data as our app continues to accept new data from my "curl" script.
After I click "Validate", I will be given a green window stating that the oplog is readable from the migration service and that I’m ready to migrate my data.
While the data is being migrated, I can watch its progress in the tile (shown above) representing my MongoDB Atlas cluster. I’ll be notified of the completion of the initial sync between the source Cloud Manager cluster and my MongoDB Atlas cluster.
When the initial sync completes, I'll have a 72 hour window to complete my cutover.
During this period, changes to the oplog of my primary being managed by Cloud Manager will continue to be synced to my new Atlas cluster. There are just a few final steps left to complete my migration to MongoDB Atlas.
Database User
MongoDB Atlas will allow me to configure a "least privilege access control" user account for my app. I can navigate within my MongoDB Atlas menu to Security > Users and create a user for our web app with access only to the database "demodb", which is where I write new entries for my app.
I'll take this user information and my connection string from MongoDB Atlas and begin the process of cutting over my application. I click CONNECT on the tile representing my Atlas cluster and get the default login for my cluster, but I'll modify it slightly to reflect our database's name and the database user we created for it. You can always test this first by accessing your user from the shell:
[root@ip-172-16-0-77 ~]# mongo "mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/demodb?replicaSet=records-shard-0" --authenticationDatabase admin --ssl --username todo --password
MongoDB shell version v3.4.4
Enter password:
connecting to: mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/demodb?replicaSet=records-shard-0
2017-05-04T17:07:15.705+0000 I NETWORK [thread1] Starting new replica set monitor for records-shard-0/records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017
MongoDB server version: 3.4.4
records-shard-0:PRIMARY>
Cutover and update connection string
It's time to finalize my database migration. I can now click the "Start Cutover" button that will provide me details on how to complete the migration. I’m provided with instructions to stop my app; in our case, I'll just stop node.Next, I’m going to wait for the optime gap to reach zero in the window, which means my sync is up to date with the latest changes to the oplog associated with my source:
l enter in our new connection string in my app's config/database.js file with our new credentials:
module.exports = {
remoteUrl : '',
localUrl: 'mongodb://todo:$PASSWORD@records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/demodb?ssl=true&replicaSet=records-shard-0&authSource=admin'
};
Finally I can click "I'M DONE" in my import window. Once I’ve done that, I can restart my node app. I'm then taken to the tile for my Atlas cluster, which confirms that the migration process has been completed!
My app is now online with my database running in MongoDB Atlas.
Conclusion
Now that my migration is completed, I can download my latest backup snapshots from MongoDB Cloud Manager. Then I can terminate the instances and processes used for my source database. I now have the benefits of a fully managed database service, meaning I can spend less time on database administration work.Sign up for MongoDB Atlas today and easily migrate your data with the live migration service!