How does a MongoDB driver discover and monitor a single server, a set of mongos servers, or a replica set? How does it determine what types of servers they are? How does it keep this information up to date? How does it discover an entire replica set given an initial host list, and how does it respond to stepdowns, elections, reconfigurations, network error, or the loss of a server?

In the past each MongoDB driver answered these questions a little differently, and mongos differed a little from the drivers. We couldn't answer questions like, "Once I add a secondary to my replica set, how long does it take for the driver to start using it?" Or, "How does a driver detect when the primary steps down, and how does it react?"

To standardize our drivers, I wrote the Server Discovery And Monitoring Spec, with David Golden, Craig Wilson, Jeff Yemin, and Bernie Hackett. Beginning with this spring’s next-generation driver releases, all our drivers conform to the spec and answer these questions the same. Or, where there's a legitimate reason for them to differ, there are as few differences as possible and each is clearly explained in the spec. Even in cases where several answers seem equally good, drivers agree on one way to do it.

The spec describes how a driver monitors a topology:

The spec covers all MongoDB topologies, but replica sets are the most interesting. So I’ll explain the spec’s algorithm for replica sets by telling the story of your application as it passes through life stages: it starts up, discovers a replica set, and reaches a steady state. Then there is a crisis—I spill coffee on your primary server’s motherboard—and a resolution—the replica set elects a new primary and the driver discovers it.

At each stage we’ll observe a typical multi-threaded driver, PyMongo 3.0, a typical single-threaded driver, the Perl Driver 1.0, and a hybrid, the C Driver 1.2. (I implemented PyMongo's server discovery and monitoring. David Golden wrote the Perl version, and Samantha Ritter and Jason Carey wrote the one in C.)

To conclude, I'll tell you our strategy for verifying spec compliance in ten programming languages, and I'll share links for further reading.

Startup

When your application initializes, it creates a MongoClient. In Python:

In Perl:

In C, you can either create a client directly:

Or create a client pool:

A crucial improvement of the next gen drivers is, the constructor no longer blocks while it makes the initial connection. Instead, the constructor does no network I/O. PyMongo launches a background thread per server (two threads in this example) to initiate discovery, and returns control to your application without blocking. Perl does nothing until you attempt an operation; then it connects on demand.

In the C Driver, if you create a client directly it behaves like the Perl Driver: it connects on demand, on the main thread. But the C Driver's client pool launches one background thread to discover and monitor all servers.

The spec's "no I/O in constructors" rule is a big win for web applications that use our next gen drivers: In a crisis, your app servers might be restarted while your MongoDB servers are unreachable. Your application should not throw an error at startup, when it constructs the client object. It starts up disconnected and tries to reach your servers until it succeeds.

Discovery

The initial host list you provide is called the "seed list":

The seed list is the stepping-off point for the driver's journey of discovery. As long as one seed is actually an available replica set member, the driver will discover the whole set and stay connected to it indefinitely, as described below. Even if every member of the set is replaced with a new host, like the Ship of Theseus, it is still the same replica set and the driver remains connected to it.

I tend to think of a driver as a tiny economy of information about your topology. Monitoring supplies information, and your application's operations demand information. Their demands are defined in David Golden's Server Selection Spec, while the method of supplying information is defined here, in the Server Discovery And Monitoring Spec. In the beginning, there is no information, and the monitors rush to supply some. I'll talk more about the demand side later, in the "Crisis" section.

Multi-threaded

Let's start with PyMongo. In PyMongo, like other multi-threaded drivers, the MongoClient constructor starts one monitor thread each for "hostA" and "hostB".

Each monitor connects to its assigned server and executes the "ismaster" command. Ignore the command's archaic name, which dates from the days of master-slave replication, long superseded by replica sets. The ismaster command is the client-server handshake. Let's say the driver receives hostB's response first:

hostB confirms it belongs to your replica set, informs you that it is a secondary, and lists the members in the replica set config. PyMongo sees a host it didn't know about, hostC, so it launches a new thread to connect to it.

If your application threads are waiting to do any operations with the MongoClient, they block while awaiting discovery. But since PyMongo now knows of a secondary, if your application is waiting to do a secondary read, it can now proceed:

Meanwhile, discovery continues. PyMongo waits for ismaster responses from hostA and hostC. Let's say hostC responds next, and its response includes "ismaster": true:

Now PyMongo knows the primary, so all reads and writes are unblocked. PyMongo is still waiting to hear back from hostA; once it does, it can use hostA for secondary reads as well.

Single-threaded

Multithreaded Perl code is problematic, so the Perl Driver doesn't launch a thread per host. How, then does it discover your set? When you construct a MongoClient it does no I/O. It waits for you to begin an operation before it connects. Once you do, it scans the hosts serially, initially in random order.

Let's say the driver begins with hostB, a secondary. Here's a detail I didn't show you earlier: replica set members tell you who they think the primary is. HostB's reply includes "primary": "hostC:27017":

The Perl Driver uses this hint to put hostC next in the scan order, because connecting to the primary is its top priority. It checks hostC and confirms that it's primary. Finally, it checks hostA to ensure it can connect, and discovers that hostA is another secondary. Scanning is now complete and the driver proceeds with your application's operation.

Hybrid

The C driver has two modes for server discovery and monitoring: single-threaded and pooled. Single-threaded mode is optimized for embedding the C Driver within languages like PHP: PHP applications deploy many single-threaded processes connected to MongoDB. Each process uses the same connections to scan the topology as it uses for application operations, so the total connection count from many processes is kept to a minimum.

Other applications should use pooled mode: as we shall see, in pooled mode a background thread monitors the topology, so the application need not block to scan it.

C Driver's single-threaded mode

The C driver scans servers on the main thread, if you construct a single client:

In single-threaded mode, the C Driver blocks to scan your topology periodically with the main thread, just like the Perl Driver. But unlike the Perl Driver's serial scan, the C Driver checks all servers in parallel. Using a non-blocking socket per member, it begins a check on each member concurrently, and uses the asynchronous "poll" function to receive events from the sockets, until all have responded or timed out. The driver updates its topology as ismaster calls complete. Finally it ends the scan and returns control to your application.

Whereas the Perl Driver's topology scan lasts for the sum of all server checks (including timeouts), the C Driver's topology scan lasts only the maximum of any one check's duration, or the connection timeout setting, whichever is shorter. Put another way, in single-threaded mode the C Driver fans out to begin all checks concurrently, then fans in once all checks have completed or timed out. This "fan out, fan in" topology scanning method gives the C Driver an advantage scanning very large replica sets, or sets with several high-latency members.

C Driver's pooled mode
To activate the C Driver's pooled mode, make a client pool:

The pool launches one background thread for monitoring. When the thread begins, it fans out and connects to all servers in the seed list, using non-blocking sockets and a simple event loop. As it receives ismaster responses from the servers, it updates its view of your topology, the same as a multi-threaded driver like PyMongo does. When it discovers a new server it begins connecting to it, and adds the new socket to the list of non-blocking sockets in its event loop.

As with PyMongo, when the C Driver is in background-thread mode, your application's operations are unblocked as soon as monitoring discovers a usable server. For example, if your C code is blocked waiting to insert into the primary, it is unblocked as soon as the primary is discovered, rather than waiting for all secondaries to be checked too.

Steady State

Once the driver has discovered your whole replica set, it periodically re-checks each server. The periodic check is necessary to keep track of your network latency to each server, and to detect when a new secondary joins the set. And in some cases periodic monitoring can head off errors, by proactively discovering when a server is offline.

By default, the monitor threads in PyMongo check their servers every ten seconds, as does the C Driver's monitor in background-thread mode. The Perl driver, and the C Driver in single-threaded mode, block your application to re-scan the replica set once per minute.

If you like my supply-and-demand model of a driver, the steady state is when your application's demand for topology information is satisfied. The driver occasionally refreshes its stock of information to make sure it's ready for future demands, but there is no urgency.

Crisis

So I wander into your data center, swirling my cappuccino, and I stumble and spill it on hostC's motherboard. Now your replica set has no primary. What happens next?

When your application next writes to the primary, it gets a socket timeout. Now it knows the primary is gone. Its demand for information is no longer in balance with supply. The next attempt to write blocks until a primary is found.

To meet demand, the driver works overtime. How exactly it responds to the crisis depends on which type of monitoring it uses.

Multi-threaded: In drivers like PyMongo, the monitor threads wait only half a second between server checks, instead of ten seconds. They want to know as soon as possible if the primary has come back, or if one of the secondaries has been elected primary.

Single-threaded: Drivers like the Perl Driver sleep half a second between scans of the topology. The application's write operation remains blocked until the driver finds the primary.

C Driver Single-Threaded: In single-threaded mode, the C Driver sleeps half a second between scans, just like the Perl Driver. During the scan the driver launches non-blocking "ismaster" commands on all servers concurrently, as I described above.

C Driver Pooled: Each time the driver's monitor thread receives an ismaster response, schedules that server's next ismaster call on the event loop only a half-second in the future.

Resolution

Your secondaries, hostA and hostB, promptly detect my sabotage of hostC, and hold an election. In MongoDB 3.0, the election takes just a couple seconds. Let's say hostA becomes primary.

A half second or less later, your driver rechecks hostA and sees that it is now the primary. It unblocks your application's writes and sends them to hostA. In PyMongo, the monitor threads relax, and return to their slow polling strategy: they sleep ten seconds between server checks. Same for the C Driver's monitor in background-thread mode. The Perl Driver, and the C Driver in single-threaded mode, do not rescan the topology for another minute. Demand and supply are once again in balance.

Compliance Testing

I am particularly excited about the unit tests that accompany the Server Discovery And Monitoring Spec. We have 38 tests that are specified formally in YAML files, with inputs and expected outcomes for a range of scenarios. For each driver we write a test runner that feeds the inputs to the driver and verifies the outcome. This ends confusion about what the spec means, or whether all drivers conform to it. You can track our progress toward full compliance in MongoDB's issue tracker.

Further Study

The spec is long but tractable. It explains the monitoring algorithm in very fine detail. You can read a summary, and the spec itself, here:

- A summary of the spec.
- The Server Discovery And Monitoring Spec.
- The spec source, including the YAML test files.

Its job is to describe the demand side of the driver's information economy. For the supply side, read my colleague David Golden's article on his Server Selection Spec.

Posters are a great way to present an interesting area of work and is commonplace at academic and professional conferences. We’re excited to announce that we’re bringing them to the largest MongoDB event of the year, MongoDB World 2015.

What exactly is a poster?

Think of posters as a mix between talking to your audience and publishing a paper.

• Posters summarize the outcomes of an academic or commercial project
• Posters include results - whether they were successful or not - and the lessons learned
• Posters are usually made up of a set of slides that an audience can browse at their own pace
• One of more of the authors of a poster is on hand to engage on the details

Open to any MongoDB user, posters will give you the opportunity to show off the amazing things you’ve been doing with MongoDB to a captive audience of over 2,000 technology enthusiasts.

What should you submit?

In particular we’re looking for:

• The application of new features to old problems
• Benchmarking results
• Scalability testing results
• Interesting approaches to known challenges
• New challenges and proposed solutions with experimental findings

We’re also open to anything you find particularly interesting in the work you’ve been doing with MongoDB.

Submit your poster application now. If it’s accepted, our team will help you produce a great presentation for the conference. Good luck and see you at MongoDB World.

SUBMIT YOUR APPLICATION NOW

PyMongo 3.0 is a partial rewrite of the Python driver for MongoDB. More than six years after the first release of the driver, this is the biggest release in PyMongo’s history. Bernie Hackett, Luke Lovett, Anna Herlihy, and I are proud of its many improvements and eager for you to try it out. I will shoehorn the major improvements into four shoes: conformance, responsiveness, robustness, and modernity.

Conformance

The motivation for PyMongo’s overhaul is to supersede or remove its many idiosyncratic APIs. We want you to have a clean interface that is easy to learn and, more importantly, closely matches the interfaces of our other drivers.

CRUD API

Mainly, “conformance” means we have implemented the same interface for create, read, update, and delete operations as the other drivers have, as standardized in the Driver CRUD API. My colleague Jeremy Mikola will explain the full spec in a separate article soon, but here is the gist. The familiar old methods work the same in PyMongo 3, but they are deprecated:

• save()
• insert()
• update()
• remove()
• find_and_modify()

These methods were vaguely named. For example, “update” updates or replaces some or all matching documents depending on its arguments. The arguments to “save” and “remove” are likewise finicky, and the many options for “find_and_modify” are intimidating. Other MongoDB drivers do not have exactly the same arguments in the same order for all these methods. If you or other developers on your team are using a driver from a different language, it makes life a lot easier to have consistent interfaces.

The new CRUD API names its methods like “update_one”, “insert_many”, “find_one_and_delete”: they say what they mean and mean what they say. Even better, all MongoDB drivers have exactly the same methods with the same arguments. See the spec for details.

One Client Class

In the past we had three client classes: Connection for any one server, and ReplicaSetConnection to connect to a replica set. We also had a MasterSlaveConnection that could distribute reads to slaves in a master-slave set. In November 2012 we created new classes, MongoClient and MongoReplicaSetClient, with better default settings, so now PyMongo had five clients. Even more confusingly, MongoClient could connect to a set of mongos servers and do hot failover.

As I wrote on my blog, the fine distinctions between the client classes baffled users. And the set of clients we provided did not conform with other drivers. But since PyMongo is among the most-used of all Python libraries we waited long, and thought hard, before making major changes.

The day has come. MongoClient is now the one and only client class for a single server, a set of mongoses, or a replica set. It includes the functionality that had been split into MongoReplicaSetClient: it can connect to a replica set, discover all its members, and monitor the set for stepdowns, elections, and reconfigs. MongoClient now also supports the full ReadPreference API. MongoReplicaSetClient lives on for a time, for compatibility’s sake, but new code should use MongoClient exclusively. The obsolete Connection, ReplicaSetConnection, and MasterSlaveConnection are removed.

The options you pass to MongoClient in the URI or as arguments now completely control the client’s behavior:

This is exciting because PyMongo applications are now so easy to deploy: your code simply loads a MongoDB URI from an environment variable or config file and passes it to a MongoClient. Code and configuration are cleanly separated. You can move smoothly from your laptop to a test server to the cloud, simply by changing the URI.

Non-Conforming Features

PyMongo 2 had some quirky features it did not share with other drivers. For one, we had a “copy_database” method that only one other driver had, and which almost no one used. It was hard to maintain and we believe you want us to focus on the features you use, so we removed it.

A more pernicious misfeature was the “start_request” method. It bound a thread to a socket, which hurt performance without actually guaranteeing monotonic write consistency. It was overwhelmingly misused, too: new PyMongo users naturally called “start_request” before starting a request, but in truth the feature had nothing to do with its name. For the history and details, including some entertaining (in retrospect) tales of Python threadlocal bugs, see my article on the removal of start_request.

Finally, the Python team rewrote our distributed-systems internals to conform to the new standards we have specified for all our drivers. But if you are a Python programmer you may care only a little that the new code conforms to a spec; it is more interesting to you that the new code is responsive and robust.

Responsiveness

PyMongo 3’s MongoClient can connect to a single server, a replica set, or a set of mongoses. It finds servers and reacts to changing conditions according to the Server Discovery And Monitoring spec, and it chooses which server to use for each operation according to the Server Selection Spec. David Golden and I explained these specs in general in the linked articles, but I can describe PyMongo’s implementation here.

Replica Set Discovery And Monitoring

In PyMongo 2, MongoReplicaSetClient used a single background thread to monitor all replica set members in series. So a slow or unresponsive member could block the thread for some time before the thread moved on to discover information about the other members, like their network latencies or which member is primary. If your application was waiting for that information—say, to write to the new primary after an election—these delays caused unneeded seconds of downtime.

When PyMongo 3’s new MongoClient connects to a replica set it starts one thread per mongod server. The threads fan out to connect to all members of the set in parallel, and they start additional threads as they discover more members. As soon as any thread discovers the primary, your application is unblocked, even while the monitor threads collect more information about the set. This new design improves PyMongo’s response time tremendously. If some members are slow or down, or you have many members in your set, PyMongo’s discovery is still just as fast.

I will explain and demonstrate the new design in my MongoDB World talk, “Drivers And High Availability: Deep Dive.”

Mongos Load-Balancing

Our multi-mongos behavior is improved, too. A MongoClient can connect to a set of mongos servers:

The behavior in PyMongo 2 was “high availability”: the client connected to the lowest-latency mongos in the list, and used it until a network error prompted it to re-evaluate their latencies and reconnect to one of them. If the driver chose unwisely at first, it stayed pinned to a higher-latency mongos for some time. In PyMongo 3, the background threads monitor the client’s network latency to all the mongoses continuously, and the client distributes operations evenly among those with the lowest latency. See "mongos Load Balancing" for more information.

Throughput

Besides PyMongo’s improved responsiveness to changing conditions in your deployment, its throughput is better too. We have written a faster and more memory efficient pure python BSON module, which is particularly important for PyPy, and made substantial optimizations in our C extensions.

Robustness

Disconnected Startup
The first change you may notice is, MongoClient’s constructor no longer blocks while connecting. It does not raise ConnectionFailure if it cannot connect:

The constructor returns immediately and launches the connection process on background threads. Of course, foreground operations might time out:

Meanwhile, the client’s background threads keep trying to reach the server. This is a big win for web applications that use PyMongo—in a crisis, your app servers might be restarted while your MongoDB servers are unreachable. Your applications should not throw an exception at startup, when they construct the client object. In PyMongo 3 the client can now start up disconnected; it tries to reach your servers until it succeeds.

On the other hand if you wrote code like this to check if mongod is up:

This will not work any more, since the constructor never throws ConnectionFailure now. Instead, choose how long to wait before giving up by setting “serverSelectionTimeoutMS”:

One Monitor Thread Per Server

Even during regular operations, connections may hang up or time out, and servers go down for periods; monitoring each on a separate thread keeps PyMongo abreast of changes before they cause errors. You will see fewer network exceptions than with PyMongo 2, and the new driver will recover much faster from the unexpected.

Thread Safety

Another source of fragility in PyMongo 2 was APIs that were not designed for multithreading. Too many of PyMongo’s options could be changed at runtime. For example, if you created a database handle:

… and changed the handle’s read preference on a thread, the change appeared in all threads:

Making these options mutable encouraged such mistakes, so we made them immutable. Now you configure handles to databases and collections using thread-safe APIs:

Modernity

Last, and most satisfying to the team, we have completed our transition to modern Python.

While PyMongo 2 already supported the latest version of Python 3, it did so tortuously by executing “auto2to3” on its source at install time. This made it too hard for the open source community to contribute to our code, and it led to some absurdly obscure bugs. We have updated to a single code base that is compatible with Python 2 and 3. We had to drop support for the ancient Pythons 2.4 and 2.5; we were encouraged by recent download statistics to believe that these zombie Python versions are finally at rest.

Intro

Tailable cursors, and in particular tailing MongoDB’s oplog, is a popular feature with many uses, such as real-time notifications of all the changes to your database. A tailable cursor is conceptually the same as the Unix "tail -f" command. Once you've reached the end of the result set, the cursor will not be closed, rather it will continue to wait forever for new data and when it arrives, return that too.

Tailing the oplog is very simple for replica sets, but when it comes to sharded clusters things are a little more complex. In this post we explain how to tail MongoDB’s oplog in a sharded cluster.

Why tail the oplog?

Tailable cursors can be used on capped collections and are often used for Publish-Subscribe type of data flows. In particular, MongoDB's Oplog, that we use internally for replication, is a capped collection and secondaries will use a tailable cursor to get the operations that are to be replicated.

Also 3rd party tools in the ETL or heterogeneous replication domain can read events from the MongoDB oplog. For example the Mongo Connector or the MongoDB ElasticSearch River do exactly that.

But which such a powerful interface, there's more we can do than just replication! Reactive programming has become the dominant paradigm especially in HTML5 / JavaScript applications. Several modern JavaScript frameworks will update the user interface immediately and automatically as you change some value in your data model.

Tailing a MongoDB collection, or the entire database by way of tailing the oplog, is a perfect match for such a programming model! It means the application server will be notified real-time of any changes happening in the entire database.

In fact, one wonderful JavaScript framework is already doing this: Meteor. They have a cool video demo on their site, check it out! This makes Meteor a full stack reactive platform: changes propagate automatically all the way from database to the UI.

Reading the oplog with a tailable cursor

Here's an example how to do a tailable cursor from the mongo shell:

shard01:PRIMARY>c = db.oplog.rs.find( { fromMigrate : { $exists : false } } ).addOption(  DBQuery.Option.tailable ).addOption(DBQuery.Option.awaitData)
{ "ts" : Timestamp(1422998530, 1), "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } }
{ "ts" : Timestamp(1422998574, 1), "h" : NumberLong("-6781014703318499311"), "v" : 2, "op" : "i", "ns" : "test.mycollection", "o" : { "_id" : 1, "data" : "hello" } }
{ "ts" : Timestamp(1422998579, 1), "h" : NumberLong("-217362260421471244"), "v" : 2, "op" : "i", "ns" : "test.mycollection", "o" : { "_id" : 3, "data" : "hello" } }
{ "ts" : Timestamp(1422998584, 1), "h" : NumberLong("7215322058367374253"), "v" : 2, "op" : "i", "ns" : "test.mycollection", "o" : { "_id" : 5, "data" : "hello" } }
shard01:PRIMARY>c.hasNext()
true
shard01:PRIMARY>c.next()
{
    "ts" : Timestamp(1423049506, 1),
    "h" : NumberLong("5775895302295493166"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 12,
        "data" : "hello"
    }
}
shard01:PRIMARY>c.hasNext()
false

As you can see, when used from the shell, the cursor will not wait forever, rather will timeout after a few seconds. Then you can use the hasNext() and next() methods to check if any new data has arrived. And it has!

You can of course apply any filter to find() to capture only the events you want. For example, this is what a tailing cursor from Meteor looks like:

meteor:PRIMARY> db.currentOp()
{
    "inprog" : [
        {
            "opid" : 345,
            "active" : true,
            "secs_running" : 4,
            "op" : "getmore",
            "ns" : "local.oplog.rs",
            "query" : {
                "ns" : {
                    "$regex" : "^meteor\\."
                },
                "$or" : [
                    {
                        "op" : {
                            "$in" : [
                                "i",
                                "u",
                                "d"
                            ]
                        }
                    },
                    {
                        "op" : "c",
                        "o.drop" : {
                            "$exists" : true
                        }
                    }
                ],
                "ts" : {
                    "$gt" : Timestamp(1422200128, 7)
                }
            },

Tailing the Oplog on sharded clusters

But what happens when you use sharding? Well, first of all you'll have to tail each oplog on each shard separately.

That's still doable, but there are more complications. In a sharded cluster the MongoDB balancer will occasionally be moving data from one shard to another. This means that on one shard you will see a bunch of deletes, and on the next one you'll simultaneously see a corresponding bunch of inserts. But these are purely a MongoDB internal matter. If you were tailing the oplog to capture changes in the database, most likely you wouldn't want to see these and might even be confused by these internal events. For example, a Meteor app tailing oplogs on a sharded cluster might mysteriously delete some data all of a sudden!

Let me illustrate. First let's setup a sharded cluster using the mlaunch utility:

$ mlaunch --sharded 2 --replicaset
launching: mongod on port 27018
launching: mongod on port 27019
launching: mongod on port 27020
launching: mongod on port 27021
launching: mongod on port 27022
launching: mongod on port 27023
launching: config server on port 27024
replica set 'shard01' initialized.
replica set 'shard02' initialized.
launching: mongos on port 27017
adding shards. can take up to 30 seconds...

Now I'll connect to the mongos, shard a collection and insert some data into it:

$ mongo
MongoDB shell version: 2.6.7
connecting to: test
mongos> sh.enableSharding( "test" )
{ "ok" : 1 }
mongos> sh.shardCollection( "test.mycollection", { _id : 1 }, true )
{ "collectionsharded" : "test.mycollection", "ok" : 1 }
mongos> db.mycollection.insert( { _id : 1, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 3, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 5, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 7, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 9, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 11, data : "hello" } )
WriteResult({ "nInserted" : 1 })

And if I connect to the mongod on shard01, we can see that all data is there. We can also see the insert events from the oplog:

$ mongo --port 27018
MongoDB shell version: 2.6.7
connecting to: 127.0.0.1:27018/test
shard01:PRIMARY> show collections
mycollection
system.indexes
shard01:PRIMARY> db.mycollection.find()
{ "_id" : 1, "data" : "hello" }
{ "_id" : 3, "data" : "hello" }
{ "_id" : 5, "data" : "hello" }
{ "_id" : 7, "data" : "hello" }
{ "_id" : 9, "data" : "hello" }
{ "_id" : 11, "data" : "hello" }
shard01:PRIMARY> use local
switched to db local
shard01:PRIMARY> show collections
me
oplog.rs
slaves
startup_log
system.indexes
system.replset
shard01:PRIMARY> db.oplog.rs.find().pretty()
{
    "ts" : Timestamp(1422998530, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422998574, 1),
    "h" : NumberLong("-6781014703318499311"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998579, 1),
    "h" : NumberLong("-217362260421471244"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 3,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998584, 1),
    "h" : NumberLong("7215322058367374253"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 5,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998588, 1),
    "h" : NumberLong("-5372877897993278968"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 7,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998591, 1),
    "h" : NumberLong("-243188455606213719"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 9,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998597, 1),
    "h" : NumberLong("5040618552262309692"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 11,
        "data" : "hello"
    }
}

On shard02 there's so far nothing, because there's still so little data that the balancer didn't run. Let's split the data into 2 chunks, this will trigger a balancer round:

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "version" : 4,
    "minCompatibleVersion" : 4,
    "currentVersion" : 5,
    "clusterId" : ObjectId("54d13c0555c0347d23e33cdd")
}
  shards:
    {  "_id" : "shard01",  "host" : "shard01/hingo-sputnik:27018,hingo-sputnik:27019,hingo-sputnik:27020" }
    {  "_id" : "shard02",  "host" : "shard02/hingo-sputnik:27021,hingo-sputnik:27022,hingo-sputnik:27023" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "test",  "partitioned" : true,  "primary" : "shard01" }
        test.mycollection
            shard key: { "_id" : 1 }
            chunks:
                shard01 1
            { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard01 Timestamp(1, 0) 

mongos> sh.splitAt( "test.mycollection", { _id : 6 } )
{ "ok" : 1 }

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "version" : 4,
    "minCompatibleVersion" : 4,
    "currentVersion" : 5,
    "clusterId" : ObjectId("54d13c0555c0347d23e33cdd")
}
  shards:
    {  "_id" : "shard01",  "host" : "shard01/hingo-sputnik:27018,hingo-sputnik:27019,hingo-sputnik:27020" }
    {  "_id" : "shard02",  "host" : "shard02/hingo-sputnik:27021,hingo-sputnik:27022,hingo-sputnik:27023" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "test",  "partitioned" : true,  "primary" : "shard01" }
        test.mycollection
            shard key: { "_id" : 1 }
            chunks:
                shard02 1
                shard01 1
            { "_id" : { "$minKey" : 1 } } -->> { "_id" : 6 } on : shard02 Timestamp(2, 0) 
            { "_id" : 6 } -->> { "_id" : { "$maxKey" : 1 } } on : shard01 Timestamp(2, 1) 

mongos>

As you can see, the collection is split into 2 chunks now, and the balancer has done its job and spread them evenly across the shards. If we go back to shard01, we can see how half of the records disappeared ( { "op" : "d"} are deletions):

shard01:PRIMARY> use test
switched to db test
shard01:PRIMARY> db.mycollection.find()
{ "_id" : 7, "data" : "hello" }
{ "_id" : 9, "data" : "hello" }
{ "_id" : 11, "data" : "hello" }
shard01:PRIMARY> use local
switched to db local
shard01:PRIMARY> db.oplog.rs.find().pretty()
{
    "ts" : Timestamp(1422998530, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422998574, 1),
    "h" : NumberLong("-6781014703318499311"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998579, 1),
    "h" : NumberLong("-217362260421471244"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 3,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998584, 1),
    "h" : NumberLong("7215322058367374253"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 5,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998588, 1),
    "h" : NumberLong("-5372877897993278968"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 7,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998591, 1),
    "h" : NumberLong("-243188455606213719"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 9,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998597, 1),
    "h" : NumberLong("5040618552262309692"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 11,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998892, 1),
    "h" : NumberLong("3056127588031004421"),
    "v" : 2,
    "op" : "d",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 1
    }
}
{
    "ts" : Timestamp(1422998892, 2),
    "h" : NumberLong("-7633416138502997855"),
    "v" : 2,
    "op" : "d",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 3
    }
}
{
    "ts" : Timestamp(1422998892, 3),
    "h" : NumberLong("1499304029305069766"),
    "v" : 2,
    "op" : "d",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 5
    }
}
shard01:PRIMARY>

And on shard02 we can see the same records appearing:

$ mongo --port 27021
MongoDB shell version: 2.6.7
connecting to: 127.0.0.1:27021/test
shard02:PRIMARY> db.mycollection.find()
{ "_id" : 1, "data" : "hello" }
{ "_id" : 3, "data" : "hello" }
{ "_id" : 5, "data" : "hello" }
shard02:PRIMARY> use local
switched to db local
shard02:PRIMARY> db.oplog.rs.find().pretty()
{
    "ts" : Timestamp(1422998531, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422998890, 1),
    "h" : NumberLong("-6780991630754185199"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.system.indexes",
    "fromMigrate" : true,
    "o" : {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test.mycollection"
    }
}
{
    "ts" : Timestamp(1422998890, 2),
    "h" : NumberLong("-165956952201849851"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998890, 3),
    "h" : NumberLong("-7432242710082771022"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 3,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998890, 4),
    "h" : NumberLong("6790671206092100026"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 5,
        "data" : "hello"
    }
}

If we again insert some more data...

mongos> db.mycollection.insert( { _id : 2, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 4, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 6, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 8, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.insert( { _id : 10, data : "hello" } )
WriteResult({ "nInserted" : 1 })
mongos> db.mycollection.find()
{ "_id" : 1, "data" : "hello" }
{ "_id" : 7, "data" : "hello" }
{ "_id" : 3, "data" : "hello" }
{ "_id" : 9, "data" : "hello" }
{ "_id" : 5, "data" : "hello" }
{ "_id" : 11, "data" : "hello" }
{ "_id" : 2, "data" : "hello" }
{ "_id" : 6, "data" : "hello" }
{ "_id" : 4, "data" : "hello" }
{ "_id" : 8, "data" : "hello" }
{ "_id" : 10, "data" : "hello" }

...then these inserts appear as expected on shard01...

shard01:PRIMARY> use local
switched to db local
shard01:PRIMARY> db.oplog.rs.find().pretty()

...beginning is the same as above, omitted for brevity ...

{
    "ts" : Timestamp(1422998892, 3),
    "h" : NumberLong("1499304029305069766"),
    "v" : 2,
    "op" : "d",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 5
    }
}
{
    "ts" : Timestamp(1422999422, 1),
    "h" : NumberLong("-6691556866108433789"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 6,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999426, 1),
    "h" : NumberLong("-3908881761176526422"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 8,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999429, 1),
    "h" : NumberLong("-4997431625184830993"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 10,
        "data" : "hello"
    }
}
shard01:PRIMARY>

...and shard02:

shard02:PRIMARY> use local
switched to db local
shard02:PRIMARY> db.oplog.rs.find().pretty()
{
    "ts" : Timestamp(1422998531, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422998890, 1),
    "h" : NumberLong("-6780991630754185199"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.system.indexes",
    "fromMigrate" : true,
    "o" : {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test.mycollection"
    }
}
{
    "ts" : Timestamp(1422998890, 2),
    "h" : NumberLong("-165956952201849851"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998890, 3),
    "h" : NumberLong("-7432242710082771022"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 3,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998890, 4),
    "h" : NumberLong("6790671206092100026"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 5,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999414, 1),
    "h" : NumberLong("8160426227798471967"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 2,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999419, 1),
    "h" : NumberLong("-3554656302824563522"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 4,
        "data" : "hello"
    }
}
shard02:PRIMARY>

Separating internal operations from application operations

So if an application like Meteor was reading the above, it would certainly be challenging to figure out what the end state of the data model is. If we simply combine the oplog events from boths shards, it seems there has been these inserts and deletes:

So, given the above sequence, do _id's 1, 3 and 5 exist in the data or not?

Fortunately, it is possible to distinguish cluster-internal operations from application operations. You may have noticed that the operations caused by the migrations have a fromMigrate flag set:

{
    "ts" : Timestamp(1422998890, 2),
    "h" : NumberLong("-165956952201849851"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "fromMigrate" : true,
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}

Since we are only interested in operations that actually alter the database state when taking the cluster as a whole, we can filter out everything where this flag is set. Note that the correct way is to use $exists, rather than false:

shard01:PRIMARY> db.oplog.rs.find( { fromMigrate : false } ).pretty()
shard01:PRIMARY> db.oplog.rs.find( { fromMigrate : { $exists : false } } ).pretty()
{
    "ts" : Timestamp(1422998530, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422998574, 1),
    "h" : NumberLong("-6781014703318499311"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 1,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998579, 1),
    "h" : NumberLong("-217362260421471244"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 3,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998584, 1),
    "h" : NumberLong("7215322058367374253"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 5,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998588, 1),
    "h" : NumberLong("-5372877897993278968"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 7,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998591, 1),
    "h" : NumberLong("-243188455606213719"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 9,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422998597, 1),
    "h" : NumberLong("5040618552262309692"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 11,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999422, 1),
    "h" : NumberLong("-6691556866108433789"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 6,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999426, 1),
    "h" : NumberLong("-3908881761176526422"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 8,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999429, 1),
    "h" : NumberLong("-4997431625184830993"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 10,
        "data" : "hello"
    }
}
shard01:PRIMARY>

And on shard02:

shard02:PRIMARY> db.oplog.rs.find( { fromMigrate : { $exists : false } } ).pretty()
{
    "ts" : Timestamp(1422998531, 1),
    "h" : NumberLong(0),
    "v" : 2,
    "op" : "n",
    "ns" : "",
    "o" : {
        "msg" : "initiating set"
    }
}
{
    "ts" : Timestamp(1422999414, 1),
    "h" : NumberLong("8160426227798471967"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 2,
        "data" : "hello"
    }
}
{
    "ts" : Timestamp(1422999419, 1),
    "h" : NumberLong("-3554656302824563522"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.mycollection",
    "o" : {
        "_id" : 4,
        "data" : "hello"
    }
}
shard02:PRIMARY>

Exactly what we want!

If you’re interested in learning more about the operational best practices of MongoDB, download our guide:

DOWNLOAD OPS BEST PRACTICE

Many applications need to be alerted when new documents are inserted or modified in the database. These include real-time apps, or features such as notifications, messaging, or chat.

It is possible to build this kind of application with MongoDB, using features that are native to the database: Capped Collections and Tailable Cursors!

Details and code samples that illustrate this technique are provided in How to Create a Pub/Sub application with MongoDB. There’s also a screencast that shows the application built in the blog post:

Application Code is available on GitHub.

Introduction

In this 2-part blog series, you will learn how to use MongoDB, Mongoose Object Data Mapping (ODM) with Express.js and Node.js. These technologies use a uniform language - JavaScript - providing performance gains in the software and productivity gains for developers.

In this first part, we will describe the basic mechanics of our application and undertake data modeling. In the second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application.

No prior experience with these technologies is assumed and developers of all skill levels should benefit from this blog series. So, if you have no previous experience using MongoDB, JavaScript or building a REST API, don’t worry - we will cover these topics with enough detail to get you past the simplistic examples one tends to find online, including authentication, structuring code in multiple files, and writing test cases.

Let’s begin by defining the MEAN stack.

What is the MEAN stack?

The MEAN stack can be summarized as follows:

• M = MongoDB/Mongoose.js: the popular database, and an elegant ODM for node.js.
• E = Express.js: a lightweight web application framework.
• A = Angular.js: a robust framework for creating HTML5 and JavaScript-rich web applications.
• N = Node.js: a server-side JavaScript interpreter.

The MEAN stack is a modern replacement for the LAMP (Linux, Apache, MySQL, PHP/Python) stack that became the popular way for building web applications in the late 1990s.

In our application, we won’t be using Angular.js, as we are not building an HTML user interface. Instead, we are building a REST API which has no user interface, but could instead serve as the basis for any kind of interface, such as a website, an Android application, or an iOS application. You might say we are building our REST API on the ME(a)N stack, but we have no idea how to pronounce that!

What is a REST API?

REST stands for Representational State Transfer. It is a lighter weight alternative to SOAP and WSDL XML-based API protocols.

REST uses a client-server model, where the server is an HTTP server and the client sends HTTP verbs (GET, POST, PUT, DELETE), along with a URL and variable parameters that are URL-encoded. The URL describes the object to act upon and the server replies with a result code and valid JavaScript Object Notation (JSON).

Because the server replies with JSON, it makes the MEAN stack particularly well suited for our application, as all the components are in JavaScript and MongoDB interacts well with JSON. We will see some JSON examples later, when we start defining our Data Models.

The CRUD acronym is often used to describe database operations. CRUD stands for CREATE, READ, UPDATE, and DELETE. These database operations map very nicely to the HTTP verbs, as follows:

• POST: A client wants to insert or create an object.
• GET: A client wants to read an object.
• PUT: A client wants to update an object.
• DELETE: A client wants to delete an object.

These operations will become clear later when define our API.

Some of the common HTTP result codes that are often used inside REST APIs are as follows:

• 200 - “OK”.
• 201 - “Created” (Used with POST).
• 400 - “Bad Request” (Perhaps missing required parameters).
• 401 - “Unauthorized” (Missing authentication parameters).
• 403 - “Forbidden” (You were authenticated but lacking required privileges).
• 404 - “Not Found”.

A complete description can be found in the RFC document, listed in the resources section at the end of this blog. We will use these result codes in our application and you will see some examples shortly.

Why Are We Starting with a REST API?

Developing a REST API enables us to create a foundation upon which we can build all other applications. As previously mentioned, these applications may be web-based or designed for specific platforms, such as Android or iOS.

Today, there are also many companies that are building applications that do not use an HTTP or web interface, such as Uber, WhatsApp, Postmates, and Wash.io. A REST API also makes it easy to implement other interfaces or applications over time, turning the initial project from a single application into a powerful platform.

Creating our REST API

The application that we will be building will be an RSS Aggregator, similar to Google Reader. Our application will have two main components:

1. The REST API
2. Feed Grabber (similar to Google Reader)

In this blog series we will focus on building the REST API, and we will not cover the intricacies of RSS feeds. However, code for Feed Grabber is available in a github repository, listed in the resources section of this blog.

Let’s now describe the process we will follow in building our API. We will begin by defining the data model for the following requirements:

• Store user information in user accounts
• Track RSS feeds that need to be monitored
• Pull feed entries into the database
• Track user feed subscriptions
• Track which feed entry a user has already read

Users will need to be able to do the following:

• Create an account
• Subscribe/unsubscribe to feeds
• Read feed entries
• Mark feeds/entries as read or unread

Modeling Our Data

An in-depth discussion on data modeling in MongoDB is beyond the scope of this article, so see the references section for good resources on this topic.

We will need 4 collections to manage this information:

• Feed collection
• Feed entry collection
• User collection
• User-feed-entry mapping collection

Let’s take a closer look at each of these collections.

Feed Collection

Lets now look at some code. To model a feed collection, we can use the following JSON document:

If you are familiar with relational database technology, then you will know about databases, tables, rows and columns. In MongoDB, there is a mapping to most of these Relational concepts. At the highest level, a MongoDB deployment supports one or more databases. A database contains one or more collections, which are the similar to tables in a relational database. Collections hold documents. Each document in a collection is, at a highest level, similar to a row in a relational table. However, documents do not follow a fixed schema with pre-defined columns of simple values. Instead, each document consists of one or more key-value pairs where the value can be simple (e.g., a date), or more sophisticated (e.g., an array of address objects).

Our JSON document above is an example of one RSS feed for the Eater Blog, which tracks information about restaurants in New York City. We can see that there are a number of different fields but the key ones that our client application may be interested in include the URL of the feed and the feed description. The description is important so that if we create a mobile application, it would show a nice summary of the feed.

The remaining fields in our JSON document are for internal use. A very important field is _id. In MongoDB, every document must have a field called _id. If you create a document without this field, at the point where you save the document, MongoDB will create it for you. In MongoDB, this field is a primary key and MongoDB will guarantee that within a collection, this value is unique.

Feed Entry Collection

After feeds, we want to track feed entries. Here is an example of a document in the feed entry collection:

Again, we can see that there is a _id field. There are also some other fields, such as description, title and summary. For the content field, note that we are using an array, and the array is also storing a document. MongoDB allows us to store sub-documents in this way and this can be very useful in some situations, where we want to hold all information together. The entryID field uses the tag format to avoid duplicate feed entries. Notice also the feedID field that is of type ObjectId - the value is the _id of the Eater Blog document, described earlier. This provides a referential model, similar to a foreign key in a relational database. So, if we were interested to see the feed document associated with this ObjectId, we could take the value 523b1153a2aa6a3233a913f8 and query the feed collection on _id, and it would return the Eater Blog document.

User Collection

Here is the document we could use to keep track of users:

A user has an email address, first name and last name. There is also an sp_api_key_id and sp_api_key_secret - we will use these later with Stormpath, a user management API. The last field, called subs, is a subscription array. The subs field tells us which feeds this user is subscribed-to.

User-Feed-Entry Mapping Collection

The last collection allows us to map users to feeds and to track which feeds have been read.

We use a Boolean (true/false) to mark the feed as read or unread.

Functional Requirements for the REST API

As previously mentioned, users need to be able to do the following:

• Create an account.
• Subscribe/unsubscribe to feeds.
• Read feed entries.
• Mark feeds/entries as read or unread.

Additionally, a user should be able to reset their password.

The following table shows how these operations can be mapped to HTTP routes and verbs.

In a production environment, the use of secure HTTP (HTTPS) would be the standard approach when sending sensitive details, such as passwords.

Real World Authentication with Stormpath

In robust real-world applications it is important to provide user authentication. We need a secure approach to manage users, passwords, and password resets.

There are a number of ways we could authenticate users for our application. One possibility is to use Node.js with the Passport Plugin, which could be useful if we wanted to authenticate with social media accounts, such as Facebook or Twitter. However, another possibility is to use Stormpath. Stormpath provides User Management as a Service and supports authentication and authorization through API keys. Basically, Stormpath maintains a database of user details and passwords and a client application REST API would call the Stormpath REST API to perform user authentication.

The following diagram shows the flow of requests and responses using Stormpath.

In detail, Stormpath will provide a secret key for each “Application” that is defined with their service. For example, we could define an application as “Reader Production” or “Reader Test”. This could be very useful when we are still developing and testing our application, as we may be frequently adding and deleting test users. Stormpath will also provide an API Key Properties file.

Stormpath also allows us to define password strength requirements for each application, such as:

• Must have >= 8 characters.
• Must include lowercase and uppercase.
• Must include a number.
• Must include a non-alphabetic character

Stormpath keeps track of all of our users and assigns them API keys, which we can use for our REST API authentication. This greatly simplifies the task of building our application, as we don’t have to focus on writing code for authenticating users.

Node.js

Node.js is a runtime environment for server-side and network applications. Node.js uses JavaScript and it is available for many different platforms, such as Linux, Microsoft Windows and Apple OS X.

Node.js applications are built using many library modules and there is a very rich ecosystem of libraries available, some of which we will use to build our application.

To start using Node.js, we need to define a package.json file describing our application and all of its library dependencies.

The Node.js Package Manager installs copies of the libraries in a subdirectory, called node_modules/, in the application directory. This has benefits, as it isolates the library versions for each application and so avoids code compatibility problems if the libraries were to be installed in a standard system location, such as /usr/lib, for example.

The command npm install will create the node_modules/ directory, with all of the required libraries.

Here is the JavaScript from our package.json file:

Our application is called reader-api. The main file is called server.js. Then we have a list of the dependent libraries and their versions. Some of these libraries are designed for parsing the HTTP queries. The test harness we will use is called frisby. The jasmine-node is used to run frisby scripts.

One library that is particularly important is async. If you have never used node.js, it is important to understand that node.js is designed to be asynchronous. So, any function which does blocking input/output (I/O), such as reading from a socket or querying a database, will take a callback function as the last parameter, and then continue with the control flow, only returning to that callback function once the blocking operation has completed. Let’s look at the following simple example to demonstrate this.

In the above example, we may think that the output would be:

one two

but in fact it might be:

two one

because the line that prints “one” might happen later, asynchronously, in the callback. We say “might” because if conditions are just right, “one” might print before “two”. This element of uncertainty in asynchronous programming is called non-deterministic execution. For many programming tasks, this is actually desirable and permits high performance, but clearly there are times when we want to execute functions in a particular order. The following example shows how we could use the async library to achieve the desired result of printing the numbers in the correct order:

In the above code, we are guaranteed that function two will only be called after function one has completed.

Wrapping Up Part 1

Now that we have seen the basic mechanics of node.js and async function setup, we are ready to move on. Rather than move into creating the application, we will instead start by creating tests that validate the behavior of the application. This approach is called test-driven development and has two very good features:

• It helps the developer really understand how data and functions are consumed and often exposes subtle needs like the ability to return 2 or more things in an array instead of just one thing.
• By writing tests before building the application, the paradigm becomes “broken / unimplemented until proven tested OK” instead of “assumed to be working until a test fails.” The former is a “safer” way to keep the code healthy.

If you’re interested in learning more about the architecture of MongoDB, download our guide:

DOWNLOAD ARCHITECTURE GUIDE

About the Author - Norberto

Norberto Leite is Technical Evangelist at MongoDB. Norberto has been working for the last 5 years on large scalable and distributable application environments, both as advisor and engineer. Prior to MongoDB Norberto served as a Big Data Engineer at Telefonica.

One of the more notable challenges with maintaining a suite of drivers across many languages has been following individual language idioms while still keeping their APIs consistent with each other. For example, the Ruby driver should feel like any other Ruby library when it comes to design and naming conventions. At the same time, the behavior for API calls should be the same across all of the drivers.

Towards the end of 2014, a handful of MongoDB driver developers started working on a CRUDAPI specification for our next generation drivers. The CRUD acronym refers to create, read, update, and delete operations, which are commonly found on each driver's Collection interface. In truth, the spec covers a bit more than those four methods:

• Create
• Read
• Update
• Delete
• Count
• Replace
• Aggregate
• Distinct
• Bulk, One or Many
• Find and Modify

For obvious reasons, we decided to do without the full CRUDCRADBOoMFaM acronym and stick with CRUD.

Compared to the Server Selection and SDAM specifications, which deal with internal driver behavior, the CRUDAPI is a high-level specification; however, the goal of improving consistency across our drivers is one and the same. To ensure that multiple language viewpoints were considered in drafting the spec, the team included Craig Wilson (C#), Jeff Yemin (Java), Tyler Brock (C and C++), and myself (representing PHP and other dynamic languages).

What's in a Name?

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

The spec's position on function and option names perhaps best illustrates the balancing act between language idiomaticity and cross-driver consistency. While the spec is flexible on style (e.g. snake_case or camelCase, common suffixes), certain root words are non-negotiable. The spec doesn't attempt to define an exhaustive list of permitted deviations, but it does provide a few examples for guidance:

• batchSize and batch_size are both acceptable, but batchCount is not since "batch" and "size" are root words.
• maxTimeMS can be abbreviated as maxTime if the language provides a data type with millisecond precision (e.g. TimeSpan in C#), but maximumTime is too verbose.
• If a driver's find() method needs a typed options class (e.g. Java) in lieu of a hash literal (e.g. JavaScript) or named parameters (e.g. Python), FindOptions or FindArgs are both OK, but QueryParams would be inconsistent.
• Some languages may prefer to prefix a boolean options with "is" or "has", so a bulk write's ordered option could be named isOrdered.

Several Options for Handling Options

In addition to naming conventions, the spec acknowledges that each language has its own conventions for expressing optional parameters to functions. Ruby and Python support named parameters, JavaScript and PHP might use hash literals, C++ or C# may use an options class, and Java could opt for a fluent builder class. Ultimately, we decided not to require method overloading, since it was only supported by a few languages.

Required parameters, such as the fieldName for a distinct command or the pipeline for an aggregation, must always be positional arguments on the CRUD method. This ensures that all drivers will present a consistent public API for each method and their essential inputs.

Query Modifiers and Cursor Flags

The query API found in our legacy drivers differentiates between query modifiers and wire protocol flags. Commonly used query modifiers include $orderBy, for sorting query results, or $hint, for suggesting an index. Wire protocol flags, on the other hand, might be used to instruct the server to create a tailable cursor. Depending on the driver, these options might be specified via arguments to find() or any of various setter methods on a mutable Cursor object. The CRUDAPI now enforces consistent naming for these options and ensures they will all be specified in the same manner, be it an options structure for find() or a fluent interface.

Ultimately, users should never have to think about whether these query options are modifiers within the query document or bit flags at the protocol level. That distinction is an implementation detail of today's server API. Similar to how MongoDB 2.6 introduced write commands and deprecated write operations in the wire protocol, we expect a future version of the server to do the same for queries. In fact, progress for find and getMore commands has already begun in SERVER-15176. By abstracting away these details in the CRUDAPI, we can achieve a bit of future-proofing for our drivers and the applications that use them.

A Step Towards Self-documenting Code

One of the common pain points with our legacy API, especially for beginners, was that update operations affected only a single document by default while deletes would remove everything matching the criteria. The inconsistency around the name of this limit option (is it multi, multiple, or justOne?) was icing on the cake. This is definitely something we wanted to fix in the CRUD spec, but one has to tread carefully when changing the behavior of methods that can modify or delete data.

In the interest of not surprising any users by silently changing defaults, we opted to define some new, more descriptive methods:

• deleteOne(filter)
• deleteMany(filter)
• replaceOne(filter, replacement, options)
• updateOne(filter, update, options)
• updateMany(filter, update, options)

The most striking change is that we've moved the limit option into the name of each method. This allows drivers to leave their existing update() and delete() (or remove()) methods as-is. Secondly, delete operations will now require a filter option, which means it will take a bit more effort to inadvertently wipe out a collection (deleteMany({}) instead of remove()). And lastly, we wanted to acknowledge that the difference between replacing an entire document and updating specific fields in one or many documents. By having each method check if the document contains atomic modifiers, we hope to help users avoid the mistake of clobbering an entire document when they expected to modify specific fields, or vice versa.

Less is More

Some things are better left unsaid. While the CRUD spec contains a lot of detail, there are a few subjects which aren't addressed:

• Read preferences
• Write concerns
• Fluent API for bulk writes
• Explaining queries

With regard to read preferences and write concerns, we noted that not every driver allows those options to be specified on a per-operation basis. For some, read preferences and write concerns are only set on the Client, Database, or Collection objects. Nevertheless, the spec happily permits drivers to support additional options on its read and write methods.

The Bulk API, which first appeared in the MongoDB shell and select drivers around the time MongoDB 2.6 was released, was left alone. The CRUD spec defines a single bulkWrite() method, that receives an array of models each describing the parameters for insert, update, or delete operations. We felt this method was more versatile, as it does not impose a fluent API (with all of its method calls) upon the user, nor does it hide the list of operations within a builder object. Users can create, examine, or modify the list however they like before executing it through the new method, or even re-use it entirely in a subsequent call.

Lastly, we spent a fair amount of time discussing (and bikeshedding) the API for explaining queries, aggregation pipelines, and any other operations that might be supported by MongoDB 3.0 and beyond (e.g. SERVER-10448). Ultimately, we determined that explain is not a typical use case for drivers, in contrast to the shell. We also did not want to effectively double the public API of the CRUD specification by defining explainable variants of each method. That said, all drivers will continue to provide the necessary tools to execute explains (either through queries or command execution).

Wrapping Up

If you're interested in digging deeper into any of the topics discussed in this article (and some that weren't, such as error reporting), do give the CRUDAPI spec a look. We've also published a set of standardized acceptance tests in YAML and JSON formats, which are being used by many of our next generation drivers that implement the spec.

To learn more about what's new in MongoDB 3.0, download the white paper below.

SEE WHAT'S NEW IN 3.0

About the Author - Jeremy

Jeremy Mikola is a software engineer at MongoDB's NYC office. As a member of the driver and evangelism team, he helps develop the PHP driver and contributes to various OSS projects, such as Doctrine ODM and React PHP. Jeremy lives in Hoboken, NJ and is known to enjoy a good sandwich.

In the first part of this blog series, we covered the basic mechanics of our application and undertook some data modeling. In this second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application.

Write the tests first

Let’s begin by defining some small configuration libraries.

file name: test/config/test_config.js

Our server will be running on port 8000 on localhost. This will be fine for initial testing purposes. Later, if we change the location or port number for a production system, it would be very easy to just edit this file.

To prepare for our test cases, we need to ensure that we have a good test environment. The following code achieves this for us. First, we connect to the database.

file name: test/setup_tests.js

Next, we drop the user collection. This ensures that our database is in a known starting state.

Next, we will drop the user feed entry collection.

Next, we will connect to Stormpath and delete all the users in our test application.

Next, we close the database.

Finally, we call async.series to ensure that all the functions run in the correct order.

Frisby was briefly mentioned earlier. We will use this to define our test cases, as follows.

file name: test/create_accounts_error_spec.js

We will start with the enroll route in the following code. In this case we are deliberately missing the first name field, so we expect a status reply of 400 with a JSON error that we forgot to define the first name. Let’s “toss that frisby”:

In the following example, we are testing a password that does not have any lower-case letters. This would actually result in an error being returned by Stormpath, and we would expect a status reply of 400.

In the following example, we are testing an invalid email address. So, we can see that there is no @ sign and no domain name in the email address we are passing, and we would expect a status reply of 400.

Now, let’s look at some examples of test cases that should work. Let’s start by defining 3 users.

file name: test/create_accounts_spec.js

In the following example, we are sending the array of the 3 users we defined above and are expecting a success status of 201. The JSON document returned would show the user object created, so we can verify that what was created matched our test data.

Next, we will test for a duplicate user. In the following example, we will try to create a user where the email address already exists.

One important issue is that we don’t know what API key will be returned by Stormpath a priori. So, we need to create a file dynamically that looks like the following. We can then use this file to define test cases that require us to authenticate a user.

file name: /tmp/readerTestCreds.js

In order to create the temporary file above, we need to connect to MongoDB and retrieve user information. This is achieved by the following code.

file name: tests/writeCreds.js

In the following code, we can see that the first line uses the temporary file that we created with the user information. We have also defined several feeds, such as Dilbert and the Eater Blog.

file name: tests/feed_spec.js

Previously, we defined some users but none of them had subscribed to any feeds. In the following code we test feed subscription. Note that authentication is required now and this is achieved using .auth with the Stormpath API keys. Our first test is to check for an empty feed list.

In our next test case, we will subscribe our first test user to the Dilbert feed.

In our next test case, we will try to subscribe our first test user to a feed that they are already subscribed-to.

Next, we will subscribe our test user to a new feed. The result returned should confirm that the user is subscribed now to 2 feeds.

Next, we will use our second test user to subscribe to a feed.

The REST API

Before we begin writing our REST API code, we need to define some utility libraries. First, we need to define how our application will connect to the database. Putting this information into a file gives us the flexibility to add different database URLs for development or production systems.

file name: config/db.js

If we wanted to turn on database authentication we could put that information in a file, as shown below. This file should not be checked into source code control for obvious reasons.

file name: config/security.js

We can keep Stormpath API and Secret keys in a properties file, as follows, and need to carefully manage this file as well.

file name: config/stormpath_apikey.properties

Express.js overview

In Express.js, we create an “application” (app). This application listens on a particular port for HTTP requests to come in. When requests come in, they pass through a middleware chain. Each link in the middleware chain is given a req (request) object and a res (results) object to store the results. Each link can choose to do work, or pass it to the next link. We add new middleware via app.use(). The main middleware is called our “router”, which looks at the URL and routes each different URL/verb combination to a specific handler function.

Creating our application

Now we can finally see our application code, which is quite small since we can embed handlers for various routes into separate files.

file name: server.js

We define our own middleware at the end of the chain to handle bad URLs.

Now our server application is listening on port 8000.

Let’s print a message on the console to the user.

Defining our Mongoose data models

We use Mongoose to map objects on the Node.js side to documents inside MongoDB. Recall that earlier, we defined 4 collections:

1. Feed collection.
2. Feed entry collection.
3. User collection.
4. User feed-entry-mapping collection.

So we will now define schemas for these 4 collections. Let’s begin with the user schema. Notice that we can also format the data, such as converting strings to lowercase, and remove leading or trailing whitespace using trim.

file name: app/routes.js

In the following code, we can also tell Mongoose what indexes need to exist. Mongoose will also ensure that these indexes are created if they do not already exist in our MongoDB database. The unique constraint ensures that duplicates are not allowed. The “email : 1” maintains email addresses in ascending order. If we used “email : -1” it would be in descending order.

We repeat the process for the other 3 collections.

The following is an example of a compound index on 4 fields. Each index is maintained in ascending order.

Every route that comes in for GET, POST, PUT and DELETE needs to have the correct content type, which is application/json. Then the next link in the chain is called.

Now we need to define handlers for each combination of URL/verb. The link to the complete code is available in the resources section and we just show a few examples below. Note the ease with which we can use Stormpath. Furthermore, notice that we have defined /api/v1.0, so the client would actually call /api/v1.0/user/enroll, for example. In the future, if we changed the API, say to 2.0, we could use /api/v2.0. This would have its own router and code, so clients using the v1.0 API would still continue to work.

Starting the server and running tests

Finally, here is a summary of the steps we need to follow to start the server and run the tests.

• Ensure that the MongoDB instance is running
• mongod
• Install the Node libraries
• npm install
• Start the REST API server
• node server.js
• Run test cases
• node setup_tests.js
• jasmine-node create_accounts_error_spec.js
• jasmine-node create_accounts_spec.js
• node write_creds.js
• jasmine-node feed_spec.js

MongoDB University provides excellent free training. There is a course specifically aimed at Node.js developers and the link can be found in the resources section below. The resources section also contains links to good MongoDB data modeling resources.

Resources

HTTP status code definitions

Chad Tindel’s Github Repository

M101JS: MongoDB for Node.js Developers

Data Models

Data Modeling Considerations for MongoDB Applications

Interested in learning more about how to build node.js apps with MongoDB? Register for our upcoming webinar here:

REGISTER FOR WEBINAR

<< Read Part 1

About the Author - Norberto

Norberto Leite is Technical Evangelist at MongoDB. Norberto has been working for the last 5 years on large scalable and distributable application environments, both as advisor and engineer. Prior to MongoDB Norberto served as a Big Data Engineer at Telefonica.

This year at MongoDB World, join us and 2000+ tech enthusiasts for two days of comprehensive technical sessions, networking opportunities, and one-on-one time with engineers. The most cutting edge thought leaders will take the stage at MongoDB World. They come from industry powerhouses and the most disruptive forces in the market. Come see what they’re excited about.

Featured Keynote Speakers including:

• Mark Callaghan, Member of Technical Staff of Facebook
• Ben Golub, CEO of Docker
• Hilary Mason, Data Scientist and Founder of Fast Forward Labs
• Scott Dietzen, CEO of Pure Storage
• R “Ray” Wang, Principal Analyst, Founder and Chairman of Constellation Research, Inc.
• Dev Ittycheria, President and CEO of MongoDB
• Eliot Horowitz, Co-Founder and CTO of MongoDB

With over 100+ sessions to choose from, you will walk away with best practices, tips and tricks you need to know to build and manage modern applications.

Sign up today and save $400 on onsite ticket prices with Early Bird Pricing. Hurry, prices increase after Friday, May 1st.

REGISTER NOW

The JVM drivers team is pleased to announce the release of the 3.0 Java Driver. This release is highlighted by a number of new features: an extensible Codec API that allows applications to encode and decode their own classes directly to and from BSON; a new CRUD API, based on the cross-driver CRUD specification; and finally, a callback-based asynchronous version of the CRUD API.

The road to 3.0

Since the 2.0 version of the Java driver was released nearly five years ago, there have been 13 minor releases, with very few binary compatibility breaks. While this stability has been important in allowing users to freely upgrade to the latest release, significant technical debt has accumulated in both its internal design and its public API.

We addressed some of the internal design debt early in the 3.0 development cycle by re-writing three key components of the driver: the connection pool, the server monitor, and the server selector. These were completed a year ago, and, in order to get them into production as fast as possible, they were backported and included in the 2.12 release that shipped along side MongoDB 2.6. This provided the initial field testing of some of the ideas in what later became the cross-driver Server Discovery and Monitoring and Server Selection specifications.

That left the technical debt in our public API and we could only address that by breaking backwards compatibility. In accordance with semantic versioning that required a major version bump -- and thus 3.0 was born.

We worked hard to limit the number of breaking changes so that upgrading applications to 3.0 would be relatively smooth. Here are the questions we asked ourselves about each breaking change that we were considering:

1. Is this class or method likely to be used by applications? If the answer was no, we were more likely to consider removing it, especially if it no longer served any purpose or had been replaced by something better. There were dozens of candidates that, when examined, turned out to be dispensable.
2. Would the continued existence of this class or method force us to sacrifice important design principles of the driver? Again, if the answer was yes, we were more likely to consider removing it. Happily, there weren’t many of these, but a good example is the WriteResult.getLastError method. This method allows an application to execute an unacknowledged write, and at an unbounded future time request acknowledgement. This method is nearly impossible to support correctly, and since there is an alternative (acknowledge the write immediately), the method has been removed.
3. Does the class or method have ill-defined or inconsistent semantics? Again, if yes, we considered removal. There weren’t many of these either, but a prominent one in this category is DB.requestStart , and so it’s been removed as well.

For a more complete run-down of the most significant breaking changes in 3.0, please refer to the Upgrade Guide.

New Database and Collection Classes

One of our main goals in releasing the 3.0 driver was to address some fundamental issues with the existing DB and DBCollection classes:

• Parameters that represent BSON documents are all of type DBObject, so applications which use domain-specific classes must first translate instances to DBObject instances, which reduces efficiency.
• Their properties are mutable and there is only one instance created per database or collection name, so different parts of an application can easily step on each other by, say, changing the default read preference or write concern.
• The classes are concrete so are difficult to wrap. Many have asked that they be changed to interfaces.
• There is a surfeit of overloaded methods for various combinations of optional arguments.

At the same time, we didn’t want to abandon the DB/DBCollection API, as that would make upgrading existing applications to 3.0 a nightmare, discouraging its adoption by existing users. So we decided to leave these classes alone in 3.0, and introduced a new set of classes that addressed all these problems: MongoDatabase and MongoCollection.

So for users of the 2.x driver series that are upgrading to 3.0, the driver still has the MongoClient.getDB("myDB") method and, although it has been deprecated, rest easy -- it won't be removed in the 3.x series. The deprecation is intended to nudge new applications towards the MongoDatabase/MongoCollection API, which is accessible via the new MongoClient.getDatabase("myDB") method. This way we retain backwards compatibility while still offering an improved API that users can migrate to at their own pace.

The MongoCollection interface contains just 57 methods, down from 107 in the DBCollection class, and at the same time gains functionality and improved consistency across methods. To help future proof the API we've used Options classes for methods arguments, which allows us to extend the API without cluttering it with more method overloads as functionality is added to future versions of MongoDB.

MongoCollection and MongoDatabase are defined as interfaces, and instances are immutable. Overloads are limited to situations where the extra parameter changes the return type of the method.

A popular feature request has been to provide an easier way to convert the results from the database straight into domain objects. With the introduction of a Codec registry and the use of generics in the API, working directly with domain objects is now possible. So if the new Document class isn't to your taste, then register a custom codec for your domain object and simply use the API and pass in the class of the domain object you want to use. For example:

If you always want to use a default document class with your collection you can easily change it:

Methods that return streams of values, like find, aggregate, mapReduce, and distinct, all present fluent APIs that are defined as implementations of Java’s Iterable interface.

There are two core benefits to the above implementation: the API is consistent and also future-compatible, as additions to the fluent interfaces can be made without having to increase the number of methods to the underlying collection API.

Async Driver

You asked, we answered. There's now a brand new, asynchronous driver as part of 3.0. When abstracting the core library we ensured that it supported both synchronous and asynchronous execution, on top of which we’ve written a new asynchronous library. The async driver follows the idioms set by the new API in the Java driver but uses callbacks to provide non-blocking behavior.

In the following example we get all the names out of the collection and in the callback we print those names:

Getting Started

To get started with using the new Java driver, have a look at the getting started with the Java Driver quick tour. We look forward to getting your feedback!

SUBMIT FEEDBACK HERE

When: June 4th, 9am to 4:30 pm
Where: MongoDB office, 229 W 43rd Street, 5th Floor, New York
Who: For engineers and architects implementing MongoDB storage engines
Questions: Contact henrik.ingo a t mongodb.com

To facilitate collaboration with the MongoDB team in charge of creating the MongoDB Storage Engine API, and with people implementing storage engines that consume that API, we are inviting you to a summit in our New York office. This is a full-day event taking place on June 4th.

Please RSVP to the event by May 20th, using this form, so that we can reserve enough seats and lunch. Note: MongoDB employees participating must RSVP too!

Below is a preliminary agenda for the day.

Agenda

Storage Engine API in MongoDB 3.0

• Most participants will already be familiar with it, so this is just a short recap to allow for questions.
• Feedback from community on the API in MongoDB 3.0.

Best practices for working with storage engine partners, by Mark Callaghan

• From people with experience doing the same with MySQL, we'll be examining some best practices worth repeating and some mistakes to avoid in working with 3rd party storage engine implementers

MongoDB 3.1 Storage Engine API

• Share current plans/design

MongoDB 3.1 Oplog and Replication enhancements

• Change notification
• Read committed

Community wishlist for MongoDB 3.1 Storage Engine API

• Roundtable or just multiple speakers taking turns
• From 3rd party engines to MongoDB

Unstructured time

• Unconference and/or hackathon style

Lunch will be provided.

RSVP HERE

The short answer

As the June 30, 2015 leap second event approaches, I have received a number of questions about how MongoDB is expected to behave during a leap second event.  The short answer is “just fine.” MongoDB treats the observation of leap seconds similarly to the observation of clock skew between machines or the observation of other time-setting events, like manual clock adjustment.

In more detail

To understand why MongoDB is robust to leap seconds, it helps to think about how leap seconds affect the observation of wall clock time, especially the case where it can make it appear to processes that time has gone backwards, and about how MongoDB uses wall clock time.

Leap seconds come in one of two forms: either an extra second added at the end of the last minute of a specific calendar day or the omission of the last second at the end of the last minute of a specific calendar day in UTC. So, this can lead to a time 23:59:60Z on a day with a leap second in the first case, or to time transitioning from 23:59:58Z to 00:00:00Z on a day with a leap second in the second case. Unfortunately, the time standard used by almost all computers defines a calendar day as being composed of 86,400 seconds.

Two techniques are used to deal with this discrepancy.  The cool but by far less common one is to make all the computer-reported seconds for a period of time leading up to the end of the leap-second day slightly longer or shorter than true seconds, “smearing” the leap second over several hours. Google apparently does this. The more mundane technique is for the OS clock to have the last second occur two times, from the point of view of observing processes, or to skip the last second, depending on the type of leap second. When the last second of the day occurs twice, an observer reading time with subsecond granularity could observe 23:59:59.800Z and subsequently observe 23:59:59:200Z, making it seem as though time has moved backwards. When the last second of the day is omitted, a process might believe that two seconds have passed when in fact only one has, because it observes 23:59:58Z and then 00:00:00Z.

With this information about the observable effects of leap seconds in hand, we can now look at how this might affect MongoDB’s use of wall clock time. MongoDB uses wall clock time for the following:

• To generate diagnostic information, such as log messages;
• To record the wall clock time in fields of documents via the $currentDate update operator and related operators, and to generate OIDs;
• To generate “optime” fields in replication oplogs;
• To schedule periodic events, such as replication heartbeats or cursor expirations.

Impact on Diagnostic Information

Diagnostic data is used by human beings and tools such as MMS Monitoring to monitor the health of a MongoDB cluster, or to perform a forensic analysis after an observed failure.  In these cases, the accuracy of the reported wall clock time aids in diagnosis, but is not required for correct operation of the cluster or for the analytic task.  This must be so, because MongoDB clusters are distributed over asynchronous networks, and tight synchronization of clocks among the components of the system cannot be assured.

One caveat in the forensics and monitoring use case is that, if your operating system might allow MongoDB to observe time moving backwards, some diagnostic log messages may indicate that an operation took a very long time when it in fact did not.  These false positives for slow operations are typically easy to identify because they report absurdly long or negative durations (frequently on the order of two weeks, positive or negative).  This can also occur if you manually reset your system clock during MongoDB operation.

Impact on $currentDate et al

When a client application requests a document be updated with the server’s notion of the current date and time, MongoDB simply asks the operating system for the current wall clock time and records that value in a client document. Any impact of clock adjustments for leap seconds or otherwise will effectively be passed through to the client application. Applications that require second-granularity precision of timestamps should be examined, whether or not they use MongoDB, as should the time synchronization technology used in support of that application (typically NTP).

Impact on the replica set oplog

MongoDB replica sets use a replicated operation log, or oplog, to inform secondary nodes of changes to make in order to stay consistent with the primary node.  These changes are kept in a total order, described by an “optime”, sometimes called the timestamp. This optime is composed of wall clock time paired with an "increment", an integer which uniquely identifies operations that execute during the same wall clock time. For example, the first operation recorded at 23:59:59Z would be recorded as optime (23:59:59Z,1) and the third operation would have optime (23:59:59Z,3). But wall clock time is not used indiscriminately, because system clocks can drift, or be reset. The time portion of the optime is actually the maximum of the current observed time and the greatest previous observation. If MongoDB records operation A with an optime of (23:59:59Z,1), and then observes a time of 23:59:58Z when it attempts to log a subsequent operation B, it will act as if operation B occurred during 23:59:59Z, and thus log it with an optime of (23:59:59Z,2).In addition to leap seconds, unsynchronized clocks between replica set members may cause the optime to be ahead of any one node’s local wall clock time.  This situation is common and does not negatively affect replication operation.

Impact on the scheduling of periodic tasks

The final way that MongoDB uses wall clock time is to schedule periodic activities, like sending heartbeats to replica set nodes, cleaning up expired cursors or invalidating caches that use age-based invalidation policies.  These activities are typically scheduled to run after some amount of wall clock time has elapsed, rather than at specific absolute wall clock times;  the difference is not material.  In either event, the introduction of a positive leap second may cause an event to occur later than it otherwise would have, and the introduction of a negative leap second may cause an event to occur sooner than it otherwise would have.  MongoDB’s algorithms must already be robust to these behaviors, because they are typically indistinguishable from delays induced by higher-than-average network latency or virtual machine and operating system scheduling issues.

Your Operating System matters

Remember, MongoDB relies on host operating system capabilities for reading the wall clock time, and for synchronizing events with wall clock time. As such, you should ensure that the operating system running under MongoDB is itself prepared for leap seconds.  The most widely documented database problems during the June 2012 leap second were actually caused by a livelock bug in the Linux kernel futex synchronization primitive.  The DataStax developer blog has a brief summary of the cause of the June 2012 issue in Cassandra, which correctly assigns responsibility to a since-resolved issue in the Linux kernel.

If you use Red Hat Enterprise Linux, they have a nice knowledge base article that covers the topic of leap second preparedness for RHEL. If you’re running on Windows, Microsoft  has a very brief knowledge base article on the subject of leap seconds.

If you’re interested in learning more about the operational best practices of MongoDB, download our guide:

LEARN ABOUT OPS BEST PRACTICES

About the Author - Andy

Andy Schwerin is the Director of Distributed Systems Engineering at MongoDB in New York.

Meteor provides a complete open source platform for building web and mobile apps in pure JavaScript. The Meteor team chose MongoDB as its datastore for its performance, scalability, and rich features for JSON. Meteor apps run using JavaScript via Node.JS on the server and JavaScript in the phone’s browser. You write both the client and the server sides of your application in JavaScript using the Meteor.JS framework.

What’s so great about Meteor? Here’s a few things that are particularly interesting to MongoDB developers:

• Productivity Through Simplicity. Through reactivity and intelligent implication, Meteor requires much less code than other frameworks to get the same tasks done.
• One Language. You write the client and server portions of your application in the same JavaScript language using the same framework. You can even place both in one file if you like.
• Database Everywhere. The same methods access your MongoDB database on the server or on the phone.
• Data on the Wire. Meteor does not send HTML over the network. The server sends data and the client renders it.
• Latency Compensation. On the client, Meteor prefetches data from a local database to make it look like server method calls return instantly.
• Full Stack Reactivity. All layers update themselves as data changes without any additional code.
• Open Ecosystem. Open Source Meteor integrates with other open source tools and frameworks

The Database Everywhere and Latency Compensation concepts are especially interesting. Meteor maintains a local copy of data by implementing its own miniMongo database in phone memory. All of the phone-to-server data communication and synchronization is provided by Meteor. miniMongo is a JavaScript implementation of the MongoDB API.

With the automatic syncing provided by Meteor, you don’t need to have one set of developers working in one framework for the phone, and another set of developers working in another framework for the server. Now you can have full stack developers use one language and one framework. The phone code and the server code reside in the same JavaScript file. You build and deploy with a single command.

To show you the amazing simplicity of Meteor, let me show you the implementation of a barebones app that does this on a phone:

This HTML presents the page:

Meteor Blaze is a powerful library for creating live-updating user interfaces. The Blaze UI responds to messages from objects inside the <template> object. Blaze replaces {{counter}} with the value of the counter value in the JavaScript section. Intelligent “reactivity” will update the counter variable as it changes.

Now for the JavaScript. Note how the client and server code reside in a single file:

Blaze replaces frameworks such as Angular, Backbone, Ember, React or Knockout. Blaze updates HTML templates automatically without directives, model classes, data dependency declarations, or rendering functions. Blaze infers the data dependency of arbitrary JavaScript, sets up callbacks to detect changes in the template’s data sources, recompute values, and changes the DOM in place with the update. Meteor calls this feature “reactivity.” You just write simple HTML with variable names enclosed in double brackets as we did with {{counter}} in the HTML above.

Blaze lets you create rich phone user interfaces as in these screens from the Verso app built with Meteor and MongoDB:

The same codebase that powers the phone app can power a website for a PC or Mac in a browser. Employees can use their phones to run an app while managers use their desktops for reporting. For example, the same codebase that creates the phone interfaces above presents these web pages in a browser running on a PC or Mac desktop computer:

You can install and use Meteor for free. That link has excellent step by step tutorial showing you how to create a simple “todo” app that uses MongoDB for the datastore. At the end of the tutorial you can install a more elaborate sample todo app or a more complex customer engagement app that shows off native phone hardware functionality and social features. Meteor even provides a free server sandbox to deploy and test your apps.

If you’re interested in learning more about the architecture of MongoDB, download our guide:

DOWNLOAD ARCHITECTURE GUIDE

MongoDB 3.0 introduces compression with the WiredTiger storage engine. In this post we will take a look at the different options, and show some examples of how the feature works. As always, YMMV, so we encourage you to test your own data and your own application.

Why compression?

Everyone knows storage is cheap, right?

But chances are you’re adding data faster than storage prices are declining, so your net spend on storage is rising. Your internal costs might also incorporate management and other factors, so they may be significantly higher than commodity market prices. In other words, it still pays to look for ways to reduce your storage needs.

Size is one factor, and there are others. Disk I/O latency is dominated by seek time on rotational storage. By decreasing the size of the data, fewer disk seeks will be necessary to retrieve a given quantity of data, and disk I/O throughput will improve. In terms of RAM, some compressed formats can be used without decompressing the data in memory. In these cases more data can fit in RAM, which improves performance.

Storage properties of MongoDB

There are two important features related to storage that affect how space is used in MongoDB: BSON and dynamic schema.

MongoDB stores data in BSON, a binary encoding of JSON-like documents (BSON supports additional data types, such as dates, different types of numbers, binary). BSON is efficient to encode and decode, and it is easily traversable. However, BSON does not compress data, and it is possible its representation of data is actually larger than the JSON equivalent.

One of the things users love about MongoDB’s document data model is dynamic schema. In most databases, the schema is described and maintained centrally in a catalog or system tables. Column names are stored once for all rows. This approach is efficient in terms of space, but it requires all data to be structured according to the schema. In MongoDB there is currently no central catalog: each document is self-describing. New fields can be added to a document without affecting other documents, and without registering the fields in a central catalog.

The tradeoff is that with greater flexibility comes greater use of space. Field names are defined in every document. It is a best practice to use shorter field names when possible. However, it is also important not to take this too far – single letter field names or codes can obscure the field names, making it more difficult to use the data.

Fortunately, traditional schema is not the only way to be space efficient. Compression is very effective for repeating values like field names, as well as much of the data stored in documents.

There is no Universal Compression

Compression is all around us: images (JPEG, GIF), audio (mp3), video (MPEG), and most web servers compress web pages before sending to your browser using gzip. Compression algorithms have been around for decades, and there are competitions that award innovation.

Compression libraries rely on CPU and RAM to compress and decompress data, and each makes different tradeoffs in terms of compression rate, speed, and resource utilization. For example, one measure of today’s best compression library for text can compress 1GB of Wikipedia data to 124MB compared to 323MB for gzip, but it takes about almost 3,000 times longer and 30,000 times more memory to do so. Depending on your data and your application, one library may be much more effective for your needs than others.

MongoDB 3.0 introduces WiredTiger, a new storage engine that supports compression. WiredTiger manages disk I/O using pages. Each page contains many BSON documents. As pages are written to disk they are compressed by default, and when they are read into the cache from disk they are decompressed.

One of the basic concepts of compression is that repeating values – exact values as well as patterns – can be stored once in compressed form, reducing the total amount of space. Larger units of data tend to compress more effectively as there tend to be more repeating values. By compressing at the page level – commonly called block compression – WiredTiger can more efficiently compress data.

WiredTiger supports multiple compression libraries. You can decide which option is best for you at the collection level. This is an important option – your access patterns and your data could be quite different across collections. For example, if you’re using GridFS to store large documents such as images and videos, MongoDB automatically breaks the large files into many smaller “chunks” and reassembles them when requested. The implementation of GridFS maintains two collections: fs.files, which contains the metadata for the large files and their associated chunks, and fs.chunks, which contains the large data broken into 255KB chunks. With images and videos, compression will probably be beneficial for the fs.files collection, but the data contained in fs.chunks is probably already compressed, and so it may make sense to disable compression for this collection.

Compression options in MongoDB 3.0

In MongoDB 3.0, WiredTiger provides three compression options for collections:

• No compression
• Snappy (enabled by default) – very good compression, efficient use of resources
• zlib (similar to gzip) – excellent compression, but more resource intensive

There are two compression options for indexes:

• No compression
• Prefix (enabled by default) – good compression, efficient use of resources

You may wonder why the compression options for indexes are different than those for collections. Prefix compression is fairly simple – the “prefix” of values is deduplicated from the data set. This is especially effective for some data sets, like those with low cardinality (eg, country), or those with repeating values, like phone numbers, social security codes, and geo-coordinates. It is especially effective for compound indexes, where the first field is repeated with all the unique values of second field. Prefix indexes also provide one very important advantage over Snappy or zlib – queries operate directly on the compressed indexes, including covering queries.

When compressed collection data is accessed from disk, it is decompressed in cache. With prefix compression, indexes can remain compressed in RAM. We tend to see very good compression with indexes using prefix compression, which means that in most cases you can fit more of your indexes in RAM without sacrificing performance for reads, and with very modest impact to writes. The compression rate will vary significantly depending on the cardinality of your data and whether you use compound indexes.

Some things to keep in mind that apply to all the compression options in MongoDB 3.0:

• Random data does not compress well
• Binary data does not compress well (it may already be compressed)
• Text compresses especially well
• Field names compress well in documents (the additional benefits of short field names are modest)

Compression is enabled by default for collections and indexes in the WiredTiger storage engine. To explicitly set the compression for the replica at startup, specify the appropriate options in the YAML config file. use the command line option --wiredTigerCollectionBlockCompressor. Because WiredTiger is not the default storage engine in MongoDB 3.0, you’ll also need to specify the --storageEngine option to use WiredTiger and take advantage of these compression features.

To specify compression for specific collections, you’ll need to override the defaults by passing the appropriate options in the db.createCollection() command. For example, to create a collection called email using the zlib compression library:

How to measure compression

The best way to measure compression is to separately load the data with and without compression enabled, then compare the two sizes. The db.stats() command returns many different storage statistics, but the two that matter for this comparison are storageSize and indexSize. Values are returned in bytes, but you can convert to MB by passing in 1024*1024:

This is the method we used for the comparisons provided below.

Testing compression on different data sets

Let’s take a look at some different data sets to see how some of the compression options perform. We have four databases:

Enron
This is the infamous Enron email corpus. It includes about a half million emails. There’s a great deal of text in the email body fields, and some of the metadata has low cardinality, which means that they’re both likely to compress well. Here’s an example (the email body is truncated):

Here’s how the different options performed with the Enron database:

Flights
The US Federal Aviation Administration (FAA) provides data about on-time performance of airlines. Each flight is represented as a document. Many of the fields have low cardinality, so we express this data set to compress well:

Here’s how the different options performed with the Flights database:

MongoDB Config Database
This is the metadata MongoDB stores in the config database for sharded clusters. The manual describes the various collections in that database. Here’s an example from the chunks collection, which stores a document for each chunk in the cluster:

Here’s how the different options performed with the config database:

TPC-H
TPC-H is a classic benchmark used for testing relational analytical DBMS. The schema has been modified to use MongoDB’s document model. Here’s an example from the orders table with only the first of many line items displayed for this order:

Here’s how the different options performed with the TPC-H database:

Twitter
This is a database of 200K tweets. Here’s a simulated tweet introducing our Java 3.0 driver:

Here’s how the different options performed with the Twitter database:

Comparing compression rates

The varying sizes of these databases make them difficult to compare side by side in terms of absolute size. We can take a closer look at the benefits by comparing the storage savings provided by each option. To do this, we compare the size of each database using Snappy and zlib to the uncompressed size in WiredTiger. As above, we’re adding the value of storageSize and indexSize.

Another way some people describe the benefits of compression is in terms of the ratio of the uncompressed size to the compressed size. Here’s how Snappy and zlib perform across the five databases.

How to test your own data

There are two simple ways for you to test compression with your data in MongoDB 3.0.

If you’ve already upgraded to MongoDB 3.0, you can simply add a new secondary to your replica set with the option to use the WiredTiger storage engine specified at startup. While you’re at it, make this replica hidden with 0 votes so that it won’t affect your deployment. This new replica set member will perform an initial sync with one of your existing secondaries. After the initial sync is complete, you can remove the WiredTiger replica from your replica set then connect to that standalone to compare the size of your databases as described above. For each compression option you want to test, you can repeat this process.

Another option is to take a mongodump of your data and use that to restore it into a standalone MongoDB 3.0 instance. By default your collections will use the Snappy compression option, but you can specify different options by first creating the collections with the appropriate setting before running mongorestore, or by starting mongod with different compression options. This approach has the advantage of being able to dump/restore only specific databases, collections, or subsets of collections to perform your testing.

For examples of syntax for setting compression options, see the section “How to use compression.”

A note on capped collections

Capped collections are implemented very differently in the MMAP storage engines as compared to WiredTiger (and RocksDB). In MMAP space is allocated for the capped collection at creation time, whereas for WiredTiger and RocksDB space is only allocated as data is added to the capped collection. If you have many empty or mostly-empty capped collections, comparisons between the different storage engines may be somewhat misleading for this reason.

If you’re interested in learning more about the performance best practices of MongoDB, read our guide:

READ PERFORMANCE BEST PRACTICE

About the Author - Asya

Asya is Lead Product Manager at MongoDB. She joined MongoDB as one of the company's first Solutions Architects. Prior to MongoDB, Asya spent seven years in similar positions at Coverity, a leading development testing company. Before that she spent twelve years working with databases as a developer, DBA, data architect and data warehousing specialist.

It is no longer sufficient for organizations to deliver run-of-the mill business process applications. Mobile, social, web and sensor-enabled applications are not just potential differentiators — in many cases, they are now essential for remaining relevant. Trying to force-fit data models designed decades ago to support these new types of applications inhibits agility and drives up cost and complexity. Semi-structured and unstructured data does not lend itself to be stored and processed in the rigid row and column format imposed by relational databases, and cannot be fully harnessed for real time analytics if stored as opaque BLOBS in simple Key-Value databases.

As a result, more developers are turning to document databases such as MongoDB that allow data to be represented as rich structures, providing a viable alternative to the standard, normalized, relational model. MongoDB has a number of unique features such as atomic updates, indexed array keys, and a powerful query framework that can significantly influence schema design. While many developers have internalized the rules for designing schemas for relational databases, these same rules don't apply to document databases. This blog post series will demonstrate how to take advantage of documents to build modern applications.

From 2-Dimensional Tables to Rich Document Data Models

The most fundamental difference between relational databases and MongoDB is the way in which the data is modeled. Before getting into the details, lets compare the terminology used in the relational and document model domains:

Table 1: Translating between relational and document data models

Schema design requires a change in perspective for data architects, developers and DBAs:

• From the legacy relational data model that flattens data into rigid, 2-dimensional, tabular structures of rows and columns...
• To a rich and dynamic document data model with embedded sub-documents and arrays.

MongoDB stores JSON documents in a binary representation called BSON (Binary JSON). BSON encoding extends the popular JSON representation to include additional data types such as int, long, and floating point.

With sub-documents and arrays, JSON documents also align with the structure of objects in modern programming languages. This makes it easy for developers to map the data used in the application to its associated document in the database. By contrast, trying to map the object representation of the data to the tabular representation of an RDBMS can slow down development. Adding Object Relational Mappers (ORMs) can create additional complexity by reducing the flexibility to evolve schemas and to optimize queries to meet new application requirements.

So how do I JOIN my Data?

The first concern from those coming from a relational background is the absence of JOINs in non-relational databases. As demonstrated below, the document model makes JOINs redundant in many cases.

In Figure 1, the RDBMS uses the Pers_ID field to JOIN the “Person” table with the “Car” table to enable the application to report each car’s owner. Using the document model, embedded sub-documents and arrays effectively pre-JOIN data by aggregating related fields within a single data structure. Rows and columns that were traditionally normalized and distributed across separate tables can now be stored together in a single document, eliminating the need to JOIN separate tables when the application has to retrieve complete records.

Modeling the same data in MongoDB enables us to create a schema in which we embed an array of sub-documents for each car directly within the Person document.

In this simple example, the relational model consists of only two tables (in reality most applications will need tens, hundreds or even thousands of tables.) This approach does not reflect the way architects think about data, nor the way in which developers write applications. The document model enables data to be represented in a much more natural and intuitive way.

The choice of whether to embed related data, or instead to create a reference between separate documents is something we consider in the second part of this blog series.

To further illustrate the differences between the relational and document models, consider the example of a blogging platform in Figure 3. In this example, the application relies on the RDBMS to join five separate tables in order to build the blog entry. With MongoDB, all of the blog data is aggregated within a single document, linked with a single reference to a user document that contains both blog and comment authors.

In addition to making it more natural to represent data at the database level, the document model also provides performance and scalability advantages:

• An aggregated document can be accessed with a single call to the database, rather than having to JOIN multiple tables to respond to a query. The MongoDB document is physically stored as a single object, requiring only a single read from memory or disk. On the other hand, RDBMS JOINs require multiple reads from multiple physical locations.
• As documents are self-contained, distributing the database across multiple nodes (a process called sharding) becomes simpler and makes it possible to achieve massive horizontal scalability on commodity hardware. The DBA no longer needs to worry about the performance penalty of executing cross-node JOINs (should they even be possible in the existing RDBMS) to collect data from different tables.

So now we’ve introduced some of the concepts, in part 2, we will start to put documents into action by discussing schema design. We will cover how to manage related data with embedding and referencing, we’ll touch on indexing and the MongoDB transaction model.

To learn more, take a look at the Thinking in Documents webinar.

To look at specific considerations in moving from relational databases, download the guide below.

DOWNLOAD THE RDBMS MIGRATION GUIDE

(Topics included: Security, HA (journaling, replication) and durability, backup)

Welcome to our multi-part series on operations best practices. This last post in the series covers what you need to know to prepare for a secure and highly available MongoDB deployment. Download the MongoDB Operations Best Practices guide to learn more.

Blog Series:

1. Laying the Groundwork for MongoDB High Performance
2. Managing MongoDB
3. Deploying MongoDB with Confidence

Securing your MongoDB deployment

Running a secure application is an important aspect of any MongoDB deployment. While there is no way to completely eliminate risk, there are best practices to follow for securing your MongoDB deployment.

We recommend following a Defense in Depth approach [see the NSA definition] which is to layer your environment to prevent the potential of any one single point of failure that would allow unauthorized access to data stored in a MongoDB database.

To reduce your security risk, consider the following measures:

• Running MongoDB in a trusted environment
• Limiting access to the database
• Following a system of least privileges
• Instituting a secure development lifecycle
• Following deployment best practices

MongoDB Enterprise offers advanced security features with Kerberos and LDAP authentication, Red Hat Identity Management Certification, and auditing. In addition to MongoDB’s already comprehensive security framework – which includes Role-Based Access Control, PKI certificates, SSL and Field-Level Redaction – MongoDB Enterprise includes advanced features such as user rights management, auditing, encryption, and administrative controls. Download the MongoDB Security Architecture white paper for additional details on these security features.

Reference our Security Checklist documentation for an extensive checklist of security measures to protect your MongoDB installation.

Deploying for high availability

Making sure your data is durable and your database is highly available in MongoDB is essential for any high performing application. MongoDB supports continuous operation through replication providing redundancy and self-healing recovery. With MongoDB, you should use replica sets to maintain copies of your data in order to prevent database downtime. Replica failover is fully automated in MongoDB so you don’t need to manually intervene in case of failure.

There is much to know about MongoDB replication features and you can find out more in our Replication documentation. A few helpful considerations when planning your replication architecture include:

• You can configure the number of replica nodes in a replica set. Having a large number of replica nodes gives you increased protection against downtime in the event of multiple machine failures.
• You should deploy replica sets across multiple data centers to avoid downtime in the case of regional disaster (such as a fire, flood, hurricane).
• Make sure that your replica set members will always be able to elect a primary replica set member. To do this, run an odd number of replica set members. Not each has to be a data-bearing node. You can for example configure a lightweight arbiter process which is able to vote in elections.
• When you have geographically distributed members, you should know where the majority of replica set members will be located in case of any network partitions. Try to make sure that the set can elect a primary among the members in the primary data center.
• Consider including a hidden member in the replica set. They are useful for live backups or running applications that need to be isolated from regular operational workloads – for example, analytics.

Administrators of MongoDB can specify the level of availability when issuing writes to the database, also known as write concerns. Reference the Write Concern for Replica Sets documentation for more detail.

You have several options when configuring write concerns on a per connection, per database, per collection or per operation basis.

• Write acknowledged is the default global write concern.
• Replica safe. MongoDB supports writing to a specific number of replicas or to a majority. Ensuring writes propagate to additional replicas is an advantage of distributed databases, providing you with a richer set of durability guarantees than a conventional relational database .
• Journal safe. The mongod process will confirm the write operation to the application only after it has flushed the operation to the journal on the primary.
• Data center awareness. You can create sophisticated policies in terms of how writes are to be written to data centers. Our Data Center Awareness documentation has additional information on this topic.

To ensure strong consistency, reading from the primary replica set member is the default but you have many options for configuring reads. You could configure reads to be routed to secondary replica set members in the event that the primary is unavailable, thereby maintaining continuous query operations for your applications. You could direct reads to only be serviced by nodes that are geographically closest to the client – thus improving customer experience by eliminating lengthy network hops You can learn more about this topic by referencing our Read Preferences documentation.

Journaling is an important part of your disaster recovery strategy. MongoDB supports write-ahead journaling of operations to help with fast crash recovery in the case of a failure. In the event of an outage, journal entries are automatically recovered. We encourage you to read through the detailed documentation on Journaling to learn more.

Developing a sound backup and recovery strategy

You hope it never happens but disaster can strike your data center in the form of a fire, flood or human error. Administrators of MongoDB can restore operations without data loss by having a comprehensive and tested backup and recovery strategy in place.

You can backup a MongoDB system with Ops Manager, available with Enterprise Advanced, or with the cloud-hosted MongoDB Management System (MMS). Both offer point-in-time backup of replica sets and cluster-wide snapshots of sharded clusters. This allows you to restore your system to the precise moment you need.

There are also other options to backup your MongoDB system: File system backups or using the mongodump tool packaged with MongoDB. File system backups allow you to create a consistent snapshot of the file system. The documentation on Backup and Restore with Filesystem Snapshots offers more details on this method. mongodump performs a live backup of the data in MongoDB and can be used to dump an entire database, collection or result of a query. You can then use the mongorestore utility to restore the data to a new or existing database in case of a disaster.

There are many resources to help you get started on adopting operations best practices. Read the past blog posts in this series, including Part 1 on high performance and Part 2 on managing MongoDB. Have a look at the plethora of resources on our website: from the webinar archives to conference presentations. And if you’re interested in taking classes to learn more, sign up for our free operations and advanced operations online courses on MongoDB University.

And if you haven’t already, download our best practices guide learn more about operations.

DOWNLOAD OPS BEST PRACTICE

<< Read Part 2

About the Author - Pam

Pam is a product marketer with over a decade of experience leading marketing programs for B2B technology products. Before MongoDB, she worked at DoubleClick, the ad serving platform company, and then at Google where she worked on marketing display advertising products for over 5 years. Pam earned her BA in History from Barnard College and an MBA from Duke University.

Does anyone care about the skills you have? If you know Hadoop, MongoDB or Flume then the answer is a definite yes. And people will pay a lot of money for those skills.

This week Experis launched its Tech Cities Job Watch Q1 report. The research reviewed the technology employment landscape across the ten biggest cities in the UK from January to March. It was a good quarter too - hiring demand is up 9% and average technology salaries are now as high as £52,982 (approx $81,000USD) in London.

Big data was far and away the most lucrative technology sector with average salaries exceeding £60,000 (approx $91,000 USD) in some cities, such as Glasgow. Helpfully the report also revealed what specific skills were most desired by employers. The top three were Hadoop, MongoDB and Apache Flume.

It’s no secret that data-related skills are in high demand. Many of the findings in Tech Cities Job Watch will be unlikely to shock readers. However, the point here is just how profitable it has become. It’s also never been easier to gain these skills - there are numerous free online courses that could give you a foot in the door to a big data career.

Spending Intentions

On the other side of the Atlantic, a study came out this week which looked at the other side of the equation. The report, by Enterprise Technology Research, surveyed 685 enterprise CIOs to find out how they will be spending their budgets.

The top five buying intentions were: Docker, Hortonworks, Airwatch (VMware), MongoDB and Cloudera. Spot any trends here? Just in case you missed it: a lot of money is being spent on big data.

So, if you’re not earning as much as you deserve then your course is clear. Develop your big data skills and consider moving to the UK. A good salary makes it a lot easier to put up with the milky tea, bad weather and royal family.

Interested in more MongoDB? Sign up for our free MongoDB University courses:

MONGODB UNIVERSITY

About the Author - Jack

Jack is the EMEA communications manager for MongoDB.

Product catalog data management is a complex problem for retailers today. After years of relying on multiple monolithic, vendor-provided systems, retailers are now reconsidering their options and looking to the future.

In today’s vendor-provided systems, product data must frequently be moved back and forth using ETL processes to ensure all systems are operating on the same data set. This approach is slow, error prone, and expensive in terms of development and management. In response, retailers are now making data services available individually as part of a centralized service-oriented architecture (SOA).

This is a pattern that we commonly see at MongoDB, so much so that we’ve begun to define some best practices and reference architecture specifically targeted at the retail space. As part of that effort, today we’ll be taking a look at implementing a catalog service using MongoDB as the first of a three part series on retail architecture.

Why MongoDB?

Many different database types are able to fulfill our product catalog use case, so why choose MongoDB?

• Document flexibility: Each MongoDB document can store data represented as rich JSON structures. This makes MongoDB ideal for storing just about anything, including very large catalogs with thousands of variants per item.

• Dynamic schema: JSON structures within each document can be altered at any time, allowing for increased agility and easy restructuring of data when needs change. In MongoDB, these multiple schemas can be stored within a single collection and can use shared indexes, allowing for efficient searching of both old and new formats simultaneously.

• Expressive query language: The ability to perform queries across many document attributes simplifies many tasks. This can also improve application performance by lowering the required number of database requests.

• Indexing: Powerful secondary, compound and geo-indexing options are available in MongoDB right out of the box, quickly enabling features like sorting and location-based queries.

• Data consistency: By default, all reads and writes are sent to the primary member of a MongoDB replica set. This ensures strong consistency, an important feature for retailers, who may have many customers making requests against the same item inventory.

• Geo-distributed replicas: Network latency due to geographic distance between a data source and the client can be problematic, particularly for a catalog service which would be expected to sustain a large number of low-latency reads. MongoDB replica sets can be geo-distributed, so that they are close to users for fast access, mitigating the need for CDNs in many cases.

These are just a few of the characteristics of MongoDB that make it a great option for retailers. Next, we’ll take a look at some of the specifics of how we put some of these to use in our retail reference architecture to support a number of features, including:

• Searching for items and item variants
• Retrieving per store pricing for items
• Enabling catalog browsing with faceted search

Item Data Model

The first thing we need to consider is the data model for our items. In the following examples we are showing only the most important information about each item, such as category, brand and description:

This type of simple data model allows us to easily query for items based on the most demanded criteria. For example, using db.collection.findOne, which will return a single document that satisfies a query:

• Get item by ID
  db.definition.findOne({_id:”301671”})

• Get items for a set of product IDs
  db.definition.findOne({_id:{$in:[”301671”,”452318”]}})

• Get items by category prefix
  db.definition.findOne({category:/^Shoes\/Women/})

Notice how the second and third queries used the $in operator and a regular expression, respectively. When performed on properly indexed documents, MongoDB is able to provide high throughput and low latency for these types of queries.

Variant Data Model

Another important consideration for our the product catalog is item variants, such as available sizes, colors, and styles. Our item data model above only captures a small amount of the data about each catalog item. So what about all of the available item variations we may need to retrieve, such as size and color?

One option is to store an item and all its variants together in a single document. This approach has the advantage of being able to retrieve an item and all variants in a single query. However, it is not the best approach in all cases. It is an important best practice to avoid unbounded document growth. If the number of variants and their associated data is small, it may make sense to store them in the item document.

Another option is to create a separate variant data model that can be referenced relative to the primary item:

This data model allows us to do fast lookups of specific item variants by their SKU number:

db.variation.find({_id:”93284847362823”})

As well as all variants for a specific item by querying on the itemId attribute:

db.variation.find({itemId:”30671”}).sort({_id:1})

In this way, we maintain fast queries on both our primary item for displaying in our catalog, as well as every variant for when the user requests a more specific product view. We also ensure a predictable size for the item and variant documents.

Per Store Pricing

Another consideration when defining the reference architecture for our product catalog is pricing. We’ve now seen a few ways that the data model for our items can be structured to quickly retrieve items directly or based on specific attributes. Prices can vary by many factors, like store location. We need a way to quickly retrieve the specific price of any given item or item variant. This can be very problematic for large retailers, since a catalog with a million items and one thousand stores means we must query across a collection of a billion documents to find the price of any given item.

We could, of course, store the price for each variant as a nested document within the item document, but a better solution is to again take advantage of how quickly MongoDB is able to query on _id. For example, if each item in our catalog is referenced by an itemId, while each variant is referenced by a SKU number, we can set the _id of each document to be a concatenation of the itemId or SKU and the storeId associated with that price variant. Using this model, the _id for the pair of pumps and its red variant described above would look something like this:

• Item: 30671_store23
• Variant: 93284847362823_store23

This approach also provides a lot of flexibility for handling pricing, as it allows us to price items at the item or the variant level. We can then query for all prices or just the price for a particular location:

• All prices: db.prices.find({_id:/^30671/})
• Store price: db.prices.find({_id:/^30671_store23/})

We could even add other combinations, such as pricing per store group, and get all possible prices for an item with a single query by using the $in operator:

Browse and Search Products

The biggest challenge for our product catalog is to enable browsing with faceted search. While many users will want to search our product catalog for a specific item or criteria they are looking for, many others will want to browse, then narrow the returned results by any number of attributes. So given the need to create a page like this:

We have many challenges:

• Response time: As the user browses, each page of results should return in milliseconds.
• Multiple attributes: As the user selects different facets—e.g. brand, size, color—new queries must be run on multiple document attributes.
• Variant-level attributes: Some user-selected attributes will be queried at the item level, such as brand, while others will be at the variant level, such as size.
• Multiple variants: Thousands of variants can exist for each item, but we only want to display each item once, so results must be de-duplicated.
• Sorting: The user needs to be allowed to sort on multiple attributes, like price and size, and that sorting operation must perform efficiently.
• Pagination: Only a small number of results should be returned per page, which requires deterministic ordering.

Many retailers may want to use a dedicated search engine as the basis of these features. MongoDB provides an open source connector project, which allows the use of Apache Solr and Elasticsearch with MongoDB. For our reference architecture, however, we wanted to implement faceted search entirely within MongoDB.

To accomplish this, we create another collection that stores what we will call summary documents. These documents contain all of the information we need to do fast lookups of items in our catalog based on various search facets.

Note that in this data model we are defining attributes and secondary attributes. While a user may want to be able to search on many different attributes of an item or item variant, there is only a core set that are most frequently used. For example, given a pair of shoes, it may be more common for a user to filter their search based on available size than filtering by heel height. By using both the attr and sattr attributes in our data model, we are able to make all of these item attributes available to search, but incur only the expense of indexing the most used attributes by indexing only attr.

Using this data model, we would create compound indices on the following combinations:

• department + attr + category + _id
• department + vars.attr + category + _id
• department + category + _id
• department + price + _id
• department + rating + _id

In these indices, we always start with department, and we assume users will chose the department to refine their search results. For a catalog without departments, we could have just as easily begun with another common facet like category or type. We can then perform the queries needed for faceted search and quickly return the results to the page:

• Get summary from itemId
  db.variation.find({_id:”30671”})

• Get summary of specific item variant
  db.variation.find({vars.sku:”93284847362823”},{“vars.$”:1})

• Get summaries for all items by department
  db.variation.find({department:”Shoes”})

• Get summaries with a mix of parameters
  db.variation.find({ “department”:”Shoes”,
        “vars.attr”: {“color”:”red”},
        “category”: “^/Shoes/Women”})
  

Recap

We’ve looked at some best practices for modeling and indexing data for a product catalog that supports a variety of application features, including item and item variant lookup, store pricing, and catalog browsing using faceted search. Using these approaches as a starting point can help you find the best design for your own implementation.

Learn more

To discover how you can re-imagine the retail experience with MongoDB, read our white paper. In this paper, you'll learn about the new retail challenges and how MongoDB addresses them.

To find out how MongoDB’s consulting team can get your app off the ground faster, explore our Rapid Start engagement.

LAUNCH YOUR APP FASTER

In March, Facebook announced the ability to recognize different types of actions in videos. This is one of many recent examples of powerful and interesting innovations in the area of Artificial Intelligence.

Being able to draw context out of text, images and rich media types will allow Facebook to be more effective in content curation. In other words, animal lovers may start seeing a lot more videos of kittens playing pianos in their feeds. This technology should help Facebook better target their audience, suggest friends and sell more ads.

But the possibilities go far beyond kitten videos.

In addition to delivering more personalized content, AI can lengthen our lives, make our businesses more efficient, and protect our citizens.

We can already see the huge impact of predictive analytics in metropolitan services. For example, the City of Chicago relies on their WindyGrid service to collect and make sense of the millions of pieces of information gathered daily from Chicago’s 15 most crucial departments, including police, transportation, and fire. It’s an ever-changing view of what makes the city tick. Roadwork updates, trash pickup delays, 911 health emergencies, 311 complaints about noise, public tweets about the minutia of the city’s workings, bus locations along their route, traffic light patterns, and much more. WindyGrid analyzes trends across multiple data sources to make predictions about what will happen next.

Now imagine what the City of Chicago could do with AI technology similar to Facebook’s. We move beyond understanding trends to being able to develop solutions that automatically understand and respond to specific events as they’re happening in real-time.

Chicago health officials could know if an elderly citizen is experiencing a health emergency when they are no longer able to call for help. Emergency responders could be notified when an infant is in need health attention before he first cries out. Firemen could be deployed when the first ash ignites, and an intelligent system could recognize the severity of the fire to recommend an appropriate response. Video surveillance could recognize a burglary as it is happening and send alerts to the property authorities. The possibilities are endless.

Building a practical system of this kind hasn’t been easy. Natural language processing, machine learning and reasoning require the processing of high volume, variant data which had overwhelmed traditional data stores. New technologies and new databases allow for associations, patterns, and vectors to be recognized. MongoDB has made the real-world deployment of massive, integrated machine learning systems a practical reality.

If you're interested in learning more about how the City of Chicago leveraged real-time analytics for their WindyGrid service, read the customer case study, or come to MongoDB World this June and hear for yourself!

Norberto Leite spoke at Spring.io last week. In this blog post, he will discuss his talk: Spring + MongoDB: Two Leaves of the Same Tree.

Spring Framework and MongoDB

At MongoDB, we believe that integration is key. It is important to integrate with a wide variety of languages and ecosystems, and it is important to expose all the features and functionality that make MongoDB great. We want you to be able to build all kinds of applications on all sorts of different environments using the tools that are best suited for your purpose. As part of this effort of enabling and boosting developer productivity, we support a large variety of programming languages through our official and community drivers and enabling existing frameworks and applications stacks to integrate correctly with MongoDB.

Spring Framework, and in particular, Spring Data, is a very good example of how one can consolidate the development experience using familiar or well understood tools to build their applications

Spring Projects and MongoDB

Spring is one of the most prominent frameworks used across Java enterprise projects. Many applications across a variety of businesses, environments, and stacks rely on Spring projects for many of the integrations and general implementation of functionality. Some Spring projects are widely used like Spring Batch which offers a generic approach to batch processing, Spring Boot where we can automatize a large set of application processes so developers can focus on the business logic and differentiated algorithms of their apps and services. Spring Data offers applications a very powerful ODM to support not only application level abstraction of the persistence layer but also an integrated approach to handle all the data manipulation and common impedance mismatches that application logic provokes.

This presentation discusses a set of features that make this integration "gel" well: Spring Data abstraction layer The way that Spring Data covers Object Mapping Optimizations that Spring Data enables to make the most out of MongoDB Batch processing and indexing will also be covered, with particular emphasis around the method overriding and query optimization.

Use your tools wisely

There are significant benefits of using ODMs, especially for large complex projects:

• Assures integration with existing components
• Abstraction layers allow architects to delay decisions and avoid pre-optimizations
• Common patterns that are recurrent across different data stores

But also bear in mind that many of the existing ORMs/ODMs do not have a "Document Oriented Database" first policy but have been evolving to adjust to today’s database industry revolution. Many of the implementations are based on an architecture that is oriented to relational technology, and they make significant tradeoffs to accommodate several different systems.

Spring Data is one of the most popular and best-designed ORM technologies out there. MongoDB is committed to making sure the integration between these technologies is great.

View Norberto's presentation here.