I had the chance to meet with John Cox, Senior Technology Director at MachineShop, who running their Internet of Services platform on MongoDB. MachineShop is one of many startups who are using MongoDB to power the Internet of Things and are changing the way developers and organizations engage data to garner insights and connect to their environments.
Tell us a little bit about your company. What are you trying to accomplish? How do you see yourself growing in the next few years? MachineShop is an on-demand middleware service that simplifies the way organizations build applications, integrate systems and share data within an enterprise and its ecosystem. MachineShop is uniquely architected to connect with Internet enabled devices, systems and databases and offers an API-oriented approach to aggregating and managing services that engage, enrich, expose and manage data and their underlying sources. We offer Developers and Organizations access to rich tools, reports and analytics about their services and applications through the MachineShop Services Exchange – a customizable web-based portal that offers hundreds of discrete APIs and services based on the unique roles and permissions of users.
What problem were you trying to solve? ve that problem?
When aggregating disparate data sources to be processed by central business logic and served up through a standard RESTful API, we needed a database solution that can accommodate multi-structured data and gives us high-throughput. We also need something that’s easy to scale out as we add customers and ramp up data inputs exponentially. MongoDB has it all in spades. The fact that it’s super easy to spit everything out to our API in JSON is a [very nice] bonus.
Was this a new project or did you migrate from a different database? What was it like to learn MongoDB.
Earlier iterations of MachineShop used a relational database, but the current product was build from the ground up on MongoDB. There was still a small learning curve for the team jumping into MongoDB. It was tiny, though. The prototype for the current product was built entirely in Ruby (Sinatra/Rails). The fact that we used the Mongoid ODM made the transition really easy to understand as a developer. There were a few things we had to get smart on quickly on system admin, but honestly it was fairly trivial. (Thank you!)
Did you consider other alternatives, like a relational database or non-relational database?
We considered a few alternatives. It became clear very quickly that we wanted to go with a NoSQL solution. Once we crossed that bridge, MongoDB was just an obvious choice. The barrier to entry was low – both in dollars and technical resources. There are a ton of folks working with it that made finding resources online and building relationships in the local community really easy. It’s really fun to work with great, new technology that’s constantly moving forward. It’s also nice to not be on an island trying to figure it out.
Please describe your MongoDB deployment
Right now we’re a pretty small footprint – 3 replica sets and that’s it. It’s fine for the moment. The plan is to move very soon to many shards across a lot of small instances. The idea is that striping the data buys us speed and it’s easy to scale out. We run Ubuntu on AWS for everything. We’re currently using MongoDB 2.4.6 in production.
Are you using any tools to monitor, manage and backup your MongoDB deployment? If so what?
We’re using MMS primarily for monitoring. We also use MongoLab for hosting our production database. They have some pretty good value-add service offerings that we use. We also monitor indirectly through our apps using Scout.
Are you integrating MongoDB with other data analytics, BI or visualization tools like Hadoop? If so can you share any details
We have a proof of concept in place with Hadoop for analytics as well as Storm for real-time processing and aggregation. In production we do fairly basic on-the-fly aggregations and MapReduce jobs with data from devices as well as API request metering. The ultimate goal is to make sure that it’s easy to bolt on common BI tools to allow customers to slice and dice however they like.
How are you measuring the impact of MongoDB on your business?
We’ve never measured anything like cost savings directly. With MongoDB we picked a direction and just started running. Using MongoDB never felt like cost us on any of the metrics you listed. It’s pretty much been smooth as silk. Had we not used MongoDB, I could definitely see where it would cost us in terms of engineering solutions to problems that we never encountered.
What advice would you give someone who is considering using MongoDB for their next project
Fear not! Dive in. When engineering solutions we need to make sure we’re using the right tool for any job that we do. MongoDB happens to be a great tool that can be the right one in a LOT of situations. It lets you move fast and treat your data as just data. It’s freaking fast, too. You don’t have to make so many decisions up front. You can experiment and move pieces around as needed.
A couple of things I would recommend specifically: *Make sure you have sufficient memory to store your working set (frequently accessed data and indexes). It’s just better. (Google “mongodb working set”) *If you’re using some abstraction of data access, pay close attention to performance on aggregation. We ended up sidestepping some of the abstraction to gain performance in this area.
MongoDB's dynamic schema and object-oriented structure make it a great fit for the Internet of Things. See how companies like Enernoc and Bosch are building a more connected world with MongoDB.