Cloud providers like Amazon Web Services provide you with a large array of options for hosting MongoDB. But in our experience supporting development teams, long term management of the data layer, whether that is MongoDB, or any other database, adds operational overhead, and can also distract available resources from more strategic initiatives. MongoDB Atlas provides you with a managed database service, run by MongoDB, that allows you to seamlessly deploy and expand your data layer, reducing the cycles your team spends on undifferentiated database administration work.
Instance Configuration Considerations
AWS provides users with many options when it comes to compute and disk storage, but selecting the right options can be tricky if you’re not experienced. And as your application grows, increased management complexity may become an issue.
If your instance size is no longer the right option for you in AWS, resizing may require you to make manual modifications. Long term, this may also require you to further manage the day to day aspects of the health of your EBS volumes, which involves creating new alerts for every EBS volume that you create each time, new tagging of information related to the disks, and further inventory documentation. You could manage this by yourself and indeed many organizations have built up the expertise and internal processes to do so. Or you could move to a more managed model for your data layer, one where operational and security best practices are handled for you.
Selecting MongoDB Atlas as the platform for your MongoDB database means that rather than concerning yourself with scaling, you can point and click to select the right cluster for your data; Atlas automatically handles the upgrade. Alerting, IOPS management, and long term management is simplified for you with MongoDB Atlas.
Secure in the cloud
Security should always be top of mind for a production application. When running a MongoDB deployment, it's important to follow best practices. Is your database's network properly configured to prevent unwanted users from accessing it? Rather than configuring this yourself, the easiest way to ensure your data is secure is by using MongoDB Atlas, and letting the service handle it for you.
"We need speed. Speed's what we need. Greasy, fast speed!" - Mickey in Rocky II
Innovation requires the ability to move quickly, and self hosted environments require more work by the user to make major modifications to their database server's disk size or speed.Here are just some of the reasons you might consider using a database as a service like MongoDB Atlas rather than managing databases yourself:
- A self-managed database may eventually run out of local disk space, which will require manual intervention; MongoDB Atlas allows you to expand disk on the fly.
- A self-managed database will require you to set up and maintain a monitoring suite or pay for a third-party hosted monitoring service; MongoDB Atlas comes out of the box with charts tracking relevant metrics, so you get insights faster, and get notifications when it is time to adjust capacity
- A self-managed database solution requires you to define the processes and infrastructure for data durability; MongoDB Atlas offers continuous backups with point in time recovery and queryable snapshots.
- A self-managed database may buckle under a large influx of new traffic until you manually intervene; MongoDB Atlas allows you to dynamically scale your database while allowing your front end app to handle new incoming traffic.
The very basic examples above demonstrate why many companies are migrating from their self-managed instances to database services with built in operational and security best practices. In this blog post, I'll discuss the process of taking a standalone MongoDB server and migrating it to MongoDB Atlas, the fully managed MongoDB database as a service using the data import service.
Migrating to Database as a Service (DBaaS)
MongoDB Atlas live migration supports the following migration paths for specific MongoDB versions.
| Source Replica Set Version | Destination Cluster version |
|---|---|
| 3.0 | 3.2 |
| 3.2 | 3.2 |
| 3.4 | 3.4 |
Before I get started, I want to be certain to verify my MongoDB version is supported. Later in this post I'll verify what my existing version is.
Standalone (Migration Source)
Today I will be working with data that is housed on my development Linux server in the cloud. Because the MongoDB installation was for development, it was created as a standalone instance. I want to run a db.serverStatus() to collect some basic details before I get my migration started. I can go to the command line of this standalone database server and run the following command, db.serverStatus(); here are some truncated results:
[root@myappdev people-bson]# echo "db.serverStatus()" | mongo | more
MongoDB shell version v3.4.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.4
{
"host" : "myappdev",
"version" : "3.4.4",
"process" : "mongod",
"pid" : NumberLong(3031),
"uptime" : 534,
"uptimeMillis" : NumberLong(533489),
"uptimeEstimate" : NumberLong(533),
"localTime" : ISODate("2017-05-04T20:32:33.082Z"),
"asserts" : {
"regular" : 0,
"warning" : 0,
"msg" : 0,
"user" : 0,
"rollovers" : 0
},
The hostname above myappdev is part of my /etc/hosts file. I'll need to have a fully qualified domain name as part of my hostname that resolves in order to convert my standalone. I’m running MongoDB version 3.4.4, which is supported by Atlas live migration.
[root@myappdev people-bson]# ping myappdev
PING localhost6 (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=255 time=0.013 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=255 time=0.023 ms
Convert the standalone to a replica set
Tip: If you’re already running a MongoDB replica set, just skip ahead to "MongoDB Atlas (Migration Target)" section.
In order to migrate data with Atlas live migration, my MongoDB database must have an oplog. My standalone server is not configured with an oplog, so I'll have to convert it to a replica set. To do this, I'll need to make some changes to my MongoDB configuration — I can start mongod from a configuration file, /etc/mongod.conf, and edit accordingly. As a replica set, my cluster will have an oplog, which our MongoDB Atlas data migration service will use in the migration process. This ensures that the data being written to my application will also be written to my destination cluster in MongoDB Atlas as the migration process is happening.
First, I open the file in vim and state that my database is now a replica set. I can do this by opening the /etc/mongod.conf file and adding the following two lines to the very bottom of the file:
replication:
replSetName: myapp
After I add this change, I'll need to restart mongod quickly. I can use my local restart script that was provided when I installed MongoDB:
[root@myappdev people-bson]# service mongod restart
Stopping mongod: [ OK ]
Starting mongod: [ OK ]
I can log into this cluster and initialize the replica set with the rs.initiate() command:
[root@myappdev people-bson]# mongo
MongoDB shell version v3.4.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.4
Server has startup warnings:
2017-05-04T21:15:59.900+0000 I STORAGE [initandlisten]
2017-05-04T21:15:59.900+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-05-04T21:15:59.900+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-05-04T21:16:00.126+0000 I CONTROL [initandlisten]
2017-05-04T21:16:00.126+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-05-04T21:16:00.126+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-05-04T21:16:00.126+0000 I CONTROL [initandlisten]
> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "myappdev:27017",
"ok" : 1
}
myapp:PRIMARY> show databases
admin 0.000GB
local 0.000GB
people-list 0.163GB
Our standalone MongoDB instance now is a single node replica set with an oplog, permitting us to start the migration process.
Firewall Rules
In order to permit access to my single node replica set for the migration to take place, I will need to make just a few more changes. I will need to permit inbound network connections to my single node replica set in both MongoDB and on my firewall. If I did not happen to be using a firewall, I would only have to modify the network interfaces the mongod process binds itself to.
I will be adding IP's 4.71.186.128/25 and 4.35.16.128/25 to my firewall to access my MongoDB port (27017) as directed by the live import documentation. If my source server was in AWS, I would be making this modification to my MongoDB development server's security group.
bindIP
By default, this is just localhost or 127.0.0.1, which will not permit remote services access to the mongod port, 27017. Once again, it's time to return to the /etc/mongod.conf file and make a modification.
Open the /etc/mongod.conf file and find the line:
bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.
Add a hash sign before this line as the directions say. It should look like this:
# bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.
Then restart mongod and confirm your database is back online:
[root@myappdev people-bson]# mongo
MongoDB shell version v3.4.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.4
Server has startup warnings:
2017-05-04T21:32:16.631+0000 I STORAGE [initandlisten]
2017-05-04T21:32:16.631+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-05-04T21:32:16.631+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-05-04T21:32:16.856+0000 I CONTROL [initandlisten]
2017-05-04T21:32:16.856+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-05-04T21:32:16.856+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-05-04T21:32:16.856+0000 I CONTROL [initandlisten]
myapp:PRIMARY>
MongoDB Atlas (Migration Target)
To get ready for my migration, I first need to sign up for MongoDB Atlas, create a group and then create a new cluster. You will need to use a production MongoDB Atlas cluster in order to migrate your data as the M0 free tier is not currently supported as a migration target.
I have launched an M10 cluster to become my migration target. As a reminder, I can modify this at any time to make it larger, or add different disk properties.
I'll go ahead and then provide a whitelist entry to my web app host server's IP and then connect to my MongoDB Atlas cluster via the shell to ensure my database is working properly.
This will make sure I am able to make a network connection as soon as I am ready to cut over to my new MongoDB Atlas cluster.
Now I can begin this process of importing my data into Atlas. I've gone ahead and selected the menu option "Migrate Data to Cluster", which will begin the process of importing my data.
A new window providing me details on required actions is surfaced next. This window will detail some of the requirements such as the authentication details, the primary hostname, and a CAFile if I’m using SSL on my single node replica set.
I've prepped all the right parts, it's time to click "I'm ready to migrate" and get to work!
Migrate to MongoDB Atlas
I am now brought to a window which asks me for a few details about my source. Since I was really not certain about authentication when I built my app, I only used network-based security. There won't be a username and password, nor SSL on my single node replica set to enter.
I only have a single node, so by default this is the primary I’ll provide to the dialog box. After I click "Validate", I will be given a green window stating that the oplog is readable from the import service, and that I’m ready to migrate my data.
I will now go ahead and click "Start Migration" and begin the process of importing my data to MongoDB Atlas.
While the data is migrated, I will see the progress in my MongoDB Atlas cluster continue until I am provided with an indication of the completion of the initial sync from my “remote” replica set to my MongoDB Atlas cluster.
Setting Up Database Users
With MongoDB Atlas’ user management capabilities, I can configure a "least privilege access control" for my app. I will go to Security->Users and create a user for my webapp with access only to the database "people-list", which is where new writes will be stored.
myapp:PRIMARY> show databases
admin 0.000GB
local 0.000GB
people-list 0.163GB
Now I'll take this user information and my connection string from MongoDB Atlas and begin the process of switching over in my app. I click config and get the default login for my cluster, but I'll modify it slightly to reflect our database's name and the database user I created for it. You can always test this first by accessing your user from the shell:
[root@myappdev people-bson]# mongo "mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/people-list?replicaSet=records-shard-0" --authenticationDatabase admin --ssl --username peopleuser --password
MongoDB shell version v3.4.4
Enter password:
connecting to: mongodb://records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/people-list?replicaSet=records-shard-0
2017-05-04T21:59:40.506+0000 I NETWORK [thread1] Starting new replica set monitor for records-shard-0/records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017
MongoDB server version: 3.4.4
records-shard-0:PRIMARY>
Cutover and update connection string
It's time to finalize my migration. I can now click the "Start Cutover" button, which will provide me details on how to complete the import to Atlas. You're provided with instructions to stop your app; in our case I'll just stop NodeJS.
Next I’m going to wait for the optime gap to reach zero in the window; this means my sync is up to date with the last changes to the oplog:
l enter in my new connection string in my app's configuration file:
module.exports = {
remoteUrl : '',
localUrl: 'mongodb://peopleuser:$PASSWORD@records-shard-00-00-x8fks.mongodb.net:27017,records-shard-00-01-x8fks.mongodb.net:27017,records-shard-00-02-x8fks.mongodb.net:27017/people-list?ssl=true&replicaSet=records-shard-0&authSource=admin'
};
Finally I can click "I'M DONE" in my import window. Once I click I'm done, I restart my NodeJS app. I'm brought to a window confirming the process has been completed!
My app is online and running on MongoDB Atlas; the migration process only required me to stop and update one file and restart my node process. The replacement of my connection string and restarting of node was done manually. I could have performed the change by updating a git repository and having my DevOps tools such as Chef, Puppet or Ansible perform the required change and restart.
Conclusion
I am now fully in production and ready to grow. By using MongoDB Atlas, I have the full benefits of a fully managed database service for my app in production!
Get started using MongoDB's Live Import tool, sign up for Atlas today!