Skunkworks 2022: A Week of Building for MongoDB Engineers

MongoDB’s 2022 internal Skunkworks hackathon wrapped up in July, and it was a bustling time of hacking, building, and developing. For MongoDB engineers, Skunkworks is a week of no meetings and no interviews—engineers can have fun and work alongside co-workers from different teams to build side projects, proof of concepts, or anything else imaginable.

Many companies do one- or two-day hackathons, maybe once per year. We do one-week hackathons about every eight to ten months. Why? Hackathons are an important part of our engineering culture and embody our values of “Think Big, Go Far” and “Build Together.” We find that dedicating an entire week to the hackathon leads to significant innovation.

With more than 115 projects submitted, this year’s Star Trek-themed event was one of the best and largest hackathons yet. Building from some of the newest MongoDB 6.0 features, let’s look at a few winning projects and the people behind them.

Charts Slack Integration

Team: James Wang, Ryan Nguyen, Andrew McMenemy, and Muthukrishnan Krishnamurthy

We work on the Charts team and genuinely love our product; plus, it’s always fun taking complete ownership of it for a week. Reporting is a big ticket item for any data visualization tool. It’s something we’ve always had in the back of our minds, and users seem really keen on it. Variations of this request have been asked on our User Voice page since 2019, along with a request for a Slack integration with Charts since 2020.

This feature is built on top of our Embedded Charts SDK. We updated the URL used to fetch an embedded Chart/Dashboard to take on a new query parameter for screen grabbing. This new URL will trigger an AWS Lambda instance we wrote for the project, which will make use of our embedding SDK to embed and take a screenshot of the chart.

Skunkworks is our favorite time of the year. Learning software through projects is so much easier said than done, especially once you start a full-time career in software. MongoDB’s hackathon improves us as engineers, provides a break from the standard work week, and allows us to work on whatever we want, whether it’s a solo project or helping to make someone's dream a reality. It inspires us to always be looking for the next best idea, along with it being a great mental health week.

MongoDB Carbon Footprint Calculation

Team: Nellie Spektor, Maya Raman, Cathy Wang, Rohan Chhaya, and Tiffany Feng

Our team was inspired by a previous Skunkworks hackathon project focused on sustainability within MongoDB Atlas. We decided to do a deep dive on carbon footprint measurement within MongoDB. We began by investigating the carbon efficiency of various parts of MongoDB, from drivers to Atlas.

For testing Atlas, we first set up multiple clusters, each varying in either location, cluster size, and sharding status. Then, using an atlas-co2 calculator script that a MongoDB Developer Advocate made, we were able to test how much carbon each cluster was emitting. Finally, we were able to display our findings about different cluster sizes and cloud providers and their carbon footprints.

While investigating drivers, we tested 100 insert/find/update/delete operations on 7 of our drivers and calculated the time taken and the wattage used, which was used to calculate carbon footprint and rank the drivers in terms of efficiency.

The biggest takeaway for us was the sheer difference in carbon emissions that a simple choice can make. For example, using the Rust driver instead of Java uses 144 times less electricity and therefore emissions. Simply shifting your cluster from one cloud provider in the Virginia region to a different cloud provider in the Iowa region saves over 3kg of carbon a week.

MongoDB is collaborating with a third-party vendor to get more robust carbon emission calculations, while tangentially understanding how we can provide a more efficient, sustainable product. Efforts like these help us to better refine our overall corporate emissions calculations.

The Skunkworks hackathon is a great initiative that shows engineers how much MongoDB values creativity and personal growth. It's amazing that we are actually encouraged to put aside our regular work and try out anything we’d like. While some people work on personal development projects, other people take the opportunity to tackle some tech debt or explore new features without the constraints of the normal product development process. Furthermore, it allows us to integrate our outside interests into our work, which makes us even more passionate and motivated.

Atlas Static Site Search

Team: Ben Perlmutter, Joon Young Lee, Shibi Balamurugan, Marcus Eagan, and Nick Larew

Our project was inspired by Algolia DocSearch. They’ve done a great job making it super easy to add search to a website. We wanted to make something similar with the MongoDB Atlas developer data platform, and we knew Atlas had the tools we needed to make this possible. It was just a question of writing some code to connect these services and creating a streamlined developer experience.

We called the project Atlas Static Site Search, and it has the following distinct components:

• A website scraper that pulls site data and adds it to MongoDB built using Atlas Triggers.

• A search index built with Atlas Search using the site data.

• An Atlas Function that queries Atlas Search.

• A React component that you can add to a website that uses the Realm Web SDK to call the Atlas Function that performs search.

• A CLI that you can use to set up the whole backend (site scraper, search index, and search query function) with one command.

The biggest challenge was getting all the different cloud services to work together well. Since it is a one week hackathon, we didn’t have time to write proper integration tests to validate that things were working as expected before deploying them. There was a lot of deploying code, praying it would work, it not working, and hotfixing.

The Skunkworks hackathon is one of our favorite parts of working at MongoDB. Whether you are an intern or an experienced engineer, you’re given the freedom to work on a passion project or learn something new. It speaks to the respect that MongoDB leadership has for the engineering and product teams. A lot of great ideas and innovative products have come out of hackathons in the past. We’ll see what happens with Atlas Static Site Search!

Simulating Common Customer Workloads

Team: Xiaochen Wu, Kyle Suarez, and Nishith Atreya

Our team recognized how replicating customer workloads has a myriad of benefits and can directly or indirectly help build a better testing environment, empower our support team, and identify potential improvement opportunities in our own product portfolio.

We tried to replicate two different workloads—one transactional and one in-app analytical. After identifying these workloads, we brainstormed how to use MongoDB features and products to support them. Then, we identified important characteristics of each workload and began replicating them using available datasets. Following this, we monitored how each workload performed in the MongoDB platform and collected insights and recommendations for our internal teams.

One of the biggest challenges was trying to figure out the important characteristics of each workload we were trying to replicate. For example, it was more difficult than expected to think of the most commonly used queries, search, and recommendation patterns that would appropriately represent the transactional customer workload.

After the completion of our project, we created a recommendation for MongoDB to build a workload suite consisting of workloads that cover a variety of customer industries. This would allow our engineering organization to test major upgrades, perform product research, and identify improvement opportunities in our platform.

$semanticSearch Aggregation Stage

Team: Thomas Rueckstiess and Steve Liu

At MongoDB Labs, we're always exploring how new technologies can be integrated with MongoDB. We read a paper that was published by a few researchers from Meta describing a novel architecture for semantic search and thought this could be a cool week-long project. We made four key changes:

1. Built a web service that was the API interface to the model

2. Introduced an aggregation pipeline called $semanticSearch that communicated with the API

3. Deployed the web service on AWS

4. Built a Star Trek themed front end using React

The Skunkworks hackathon helps us explore the creativity of MongoDB engineers. Every project submitted looked well polished and innovative. We walked away inspired by the talent that's evident in the business. Hackathons provide a creative outlet for engineers away from the day-to-day tickets and helps build a meritocratic culture where any project can receive recognition and reward.