"Scaling is the process of decoupling load from latency." -- Kiril Savino
In his talk "Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger", Kiril Savino charts the course of GameChanger's increasingly sophisticated use of MongoDB as they grew in popularity and faced increasing scale requirements.
GameChanger has taken amateur sports teams (e.g. Little League) into the digital era. Their mobile app and website provide scorekeeping, stats and team management for the coaching staff, and live game updates, stats and recap stories for families and fans.
At the heart of it all is the mobile scorekeeping app, which allows coaching staff to track the plays in a game with a few simple taps on their mobile device. This data is the input to a pipeline that leads to output like live game stats on the web, persistent player stats, and real-time email/SMS notifications to fans, so when Jimmy (Kiril’s go-to generic little league player) hits a double, Jimmy’s Mom can see the play happen wherever she is.
With thousands of games being scored across the USA every day, you can imagine this creates a lot of data. About 12TB of data since 2009, in fact, and today GameChanger is handling 120,000 operations per second. Achieving this scale did not come without a few growing pains.
In 2009, GameChanger was headed to market with a MySQL backend. Their mobile and web apps were backed by Django (they still are), with its built-in ORM (not anymore!). For many teams, the promise of ORM is that it insulates developers from having to concern themselves with the details of relational schema, by turning object models into normalized, relational tables, and by handling the composition of SQL statements behind the scenes to load and store data.
Kiril refers to ORM as "the leakiest of leaky abstractions". He says the trouble with ORM is at any real scale, it performs terribly until you do the optimization work that it was supposed to keep you from having to do in the first place. In GameChanger's most extreme case, they found an API call that took 16,000 queries and 30 seconds to satisfy... and even after optimizing, that only got down to 600 queries and 3 seconds. Beyond the inefficiencies of a naive ORM implementation, the relational data model itself was a problem.
This is what led GameChanger to migrate their data layer to MongoDB, when 1.2 was released. They reasoned that a document oriented database simply suited their use case better. They were getting JSON data from mobile apps, storing it in a data layer, and sending JSON back out to web apps, so it didn't make sense for them to have a phase in between where the data was decomposed into a relational schema. Nor would it make sense for them to create their own JSON storage layer.
Their first implementation in MongoDB was what Kiril calls their naive model. When a play was marked in their mobile app, their iOS app would emit a JSON document that, via the API, would make its way into a MongoDB collection. When someone would check the score of a game via the web, queries against a few collections would be done, as well as a few calculations (symbolized in the below diagrams as “+ / - *”), before the view could be served to the user.
Using this model, GameChanger ran into their first ceiling: system latency grew quickly with read load. Past a certain number of visitors to their site, they would slow to a crawl. Or go offline. Their solution to this problem was based on the knowledge that their system served many more reads (people viewing team info and game scores on the web) than writes (game info being input via the mobile apps). In such an environment, doing calculations and aggregations when servicing reads is wasteful -- the system winds up repeating work when nothing has changed. So GameChanger's second model did calculation and aggregation when new data was input. When an event in a game was sent to the API, after it was recorded, the system would also run calculations and update other documents, for example a stats document for the game the new play just affected.
Using this new model, the GameChanger website was able to handle much more load, but they knew they had only moved the problem to the write side of their pipeline, when the mobile app posts play data. While this threshold was farther away, what was really required was a way to decouple the aggregation work from both writing and reading.
With the third version of their infrastructure, they accomplished decoupling of the aggregation work from writes and reads with the use of an asynchronous work queue. Now, when plays are recorded, a single insert to the event collection is done, and an entry in a work queue is made (GameChanger uses Redis) to update stats based on that play. When stats pages are viewed on the web, a single read is made to the collection that contains the current stats for the game being viewed. Meanwhile, separate worker processes are taking events from the queue and running the calculations to update the stats collections. Queue processing is handled independently, and can be throttled to prioritize the from reads and writes of users. User inserts and finds are done as quickly as can be, because they access a single document, and do no aggregation work.
The benefit of this architecture is that it preserves the ability to let mobile apps continue to write play data even under high load. Under normal load, the work queue is emptied as quickly as new entries are made, so stats pages ordinarily display up-to-date information. Under crushing load, the worst case scenario is that the work queue backs up and browsers serve slightly stale stats data. When the load eases, the backlog gets cleared and all the stats are once again up to date. And that is how you want a site to degrade under crushing load -- gracefully.
GameChanger's story is a great example for teams building apps on MongoDB, and there was a lot more to Kiril's talk, so go check it out, and benefit from their hard-earned wisdom.
Planning for Scale? MongoDB is Here to Help.
Whether you're running apps for a personal project, the next startup or a large enterprise, MongoDB is here to help you scale. We can make a MongoDB expert available for an introductory consult to discuss your application requirements. Get to know our team and take advantage of our expertise with MongoDB.