Solving the Stitch Challenge: Rules & Webhooks

The week four challenge in Eliot's MongoDB World Weekly Challenges was to build a basic API for the same Airbnb data set we loaded in week two. Back in week two, we used MongoDB Charts to get interesting insights. In week four, we are using MongoDB Stitch's Rules and Webhooks features.

Before we dive into the solution, let's quickly remind ourselves of some of the challenge details.

Your API needed to provide three incoming webhooks:

• search-listings to search Airbnb listings using Stitch Rules and Filters to focus on relevant documents and fields.
• add-swagpref to capture and validate swag preferences.
• get-swagprefs to return all swag preferences sorted by submission date (descending).

The challenge also included a basic test suite to help you check the correctness of your API, but the test suite did not necessarily cover all of the requirements.

One of the interesting aspects of judging this challenge was seeing the different approaches taken to implementing webhook functions. There is often more than one way to achieve the desired outcome using different JavaScript functions or MongoDB functionality. Using a test suite allowed us to verify the expected outcome was achieved despite varying implementations.

If you'd like to follow along with this challenge solution, the prerequisite set up steps are:

1. Launch a free MongoDB Atlas M0 instance.
2. Load the Atlas sample data sets into your cluster.
3. Create a new Stitch application.

Challenge Task 1: Create a user

Collection rules and filters allow you to configure data visibility via the Stitch admin UI. When you create a webhook, you can choose one of three ways to configure the execution permissions. We will be configuring our webhooks to run as a specific application user so we will be able to apply rules and filters.

To create a user you need to:

1. Enable the Email/Password Authentication provider. Once you’ve enabled this, you are ready to add one or more users.
2. Create an Email/Password User. Our search-listings webhook function will execute using the permissions of this user account in order to take advantage of Stitch rules and filters.

If you have successfully created a user you should see this listed as a normal user with the Username/Password Provider type:

Challenge Task 2: Create a Stitch Rule

Stitch Rules provide a Permissions UI for field-level access control (read or write) based on user roles. In this solution we're only going to use the default user role and provide read-only access to a specific set of fields:

Managing field access via the Stitch UI provides central control of field visibility based on user roles. Webhook functions that execute queries using rules will only have access to fields in this collection with Read permissions.

Important notes:

• Use the “+Add field” button to add fields required for the challenge.
• Read permissions should be checked; Write permissions should be unchecked.
• Make sure the "All Additional Fields" checkbox is unchecked so that any fields that aren't in the rule will be hidden.

Challenge Tasks 3: Create a query filter

Stitch Rules can also apply a query filter to limit relevant results. Filters are written using the MongoDB query language and can be applied based on specific conditions of the request, such as the user role.

We want a filter that applies to all queries, so the Apply When criteria for this filter should be set to {"%%true": true}.

Your filter criteria based on the challenge should look like:

{
  "price": {
    "$lt": {
      "$numberInt": "300"
    }
  },
  "number_of_reviews": {
    "$gt": {
       "$numberInt": "50"
    }
  },
  "review_scores.review_scores_rating": {
    "$gt": {
      "$numberInt": "95"
    }
  }
}

If you want an easier way to build and test query filters, the Atlas Data Explorer can be a handy shortcut.

Challenge Task 4: Create the search-listings webhook function

With a rule in place that should provide the correct read-only permissions and query filter, you should be ready to create your first webhook.

A webhook is an API endpoint to exchange data with your Stitch application. You can use the HTTP service to create Incoming Webhooks in order to provide your own API endpoints.

Before you can create any webhooks, you need to Create an HTTP Service Interface. For this challenge the service name should be offsite. The service name will form part of the path that external services will use to call our webhooks:

Your webhook configuration screen should look similar to:

There are some important details required for the webhook to work as expected:

• The webhook name should match the API specification: search-listings.
• The “Respond with Result” option should be enabled. This indicates that the HTTP response of the webhook will contain the result of the function executed by this webhook.
• The “Run Webhook As” option should be User Id and the “Execute as a specific user” should select the user created in Task 1. A webhook function that runs as System would have direct access to collection data and bypass the rules and filters that we have created.

Tip: The Settings tab includes the full Webhook URL if you want to test this webhook function from a browser or command line. For example, using the curl command line tool you can call your webhook using something similar to:

curl "https://webhooks.mongodb-stitch.com/api/client/v2.0/app/<APP_ID>/service/offsite/incoming_webhook/search-listings".

Your Stitch AppID will already be included if you copy the URL from the Webhook Settings tab. You can also find your AppID near the top left of your stitch application page.

We mentioned the basic test suite earlier. Now is a great time to put that test suite to work. The README for the test suite includes more details on how to install - the only prerequisite is Node.js.

You can try running the tests against your Stitch application using your own AppID:

npm start -- --app "<APP_ID>"

It is expected that most of the tests will fail because you haven’t finished configuring your webhooks:

 *** Eliot's Weekly MongoDB World Challenge Week 4 - Stitch Star ***
    Webhook 1: search-listings
      ✓ should return HTTP status code 200 (3485ms)
      1) should return an array of 5 listings without any query parameters (3186ms)
      2) should only include the fields configured in the Stitch rule (2156ms)
      3) should only return listings matching the Stitch filter for the collection (3583ms)
      4) should return listings matching query parameters (1587ms)
    Webhook 2: add-swagpref
      5) should be able to insert a document and return status code 201 (1074ms)
      6) should return an error when inserting invalid swag_type value (1194ms)
    Webhook 3: get-swagprefs
      7) should return an array sorted in descending order by date (1340ms)

1 passing (18s)
7 failing  

You can now work on implementing your webhook function and re-run the tests periodically to check on progress. To save on time, you can limit the test run to the search-listings webhook using:

npm start -- --app "<APP_ID>" -g 'search-listings'

A webhook function that meets the challenge requirements might look like:

exports = function(payload, response) {
  const mongodb = context.services.get("mongodb-atlas");
  const collection = mongodb.db("sample_airbnb").collection("listingsAndReviews");

  let cursor = collection.find(payload.query).limit(5).sort({ "review_scores.review_scores_rating": -1 }).toArray();
  cursor.then(function(result){
    response.setStatusCode(200);
    response.setHeader(
      "Content-Type",
      "application/json"
    );
    response.setBody(EJSON.stringify(result));
  });

};

Challenge Task 5: Create an employees.preference collection with schema validation

The next task is to create a new database and collection to store employee swag preferences. Head over to the Rules view, select Add Database/Collection from the “...” menu, and add a database called employees with a collection called preferences:

Users should have read/write access to the preferences collection, but we'd like document inserts and updates to be validated according to a document schema check. Stitch schema validation uses the same JSON Schema specification built into the core MongoDB server.

The JSON Schema described in the challenge should look like:

{
  "required": [
   "firstname",
    "lastname"
  ],
  "properties": {
    "_id": {
      "bsonType": "objectId"
    },
   "firstname": {
      "type": "string"
    },
    "lastname": {
      "type": "string"
    },
    "swag_type": {
      "type": "string",
      "enum": [
        "jacket",
        "t-shirt",
        "hoodie",
        "vest"
      ]
    }
  }
}

Challenge Task 6: Create the add-swagpref webhook function

The configuration for the add-swagpref webhook is almost identical to the search-listings webhook, but the "POST" HTTP Method should be selected, with request validation set to "Require Secret…" and a required Secret of submission2:

The HTTP POST request method is designed to accept larger amounts of data such as form submissions. A POST request function can retrieve parameters from the request query string (for example the submission secret), but the submitted content is expected to be in the body of the request.

The add-swagpref webhook function will need to parse values from the payload.body and insert a new document:

exports = async function(payload, response) {
  const body = payload.body ? payload.body.text() : payload.body

  console.log("Request Body:", body);
  console.log("Payload: ", JSON.stringify(payload));
  const data = EJSON.parse(body); 
  console.log("Data:", data);

  if(data) {
    const mdb = context.services.get('mongodb-atlas');
    const requests = mdb.db("employees").collection("preferences")
    const { insertedId } = await requests.insertOne(data);
    response.setStatusCode(201)
    response.setBody(`Successfully saved with _id: ${insertedId}.`);
  } else {
    response.setStatusCode(400)
    response.setBody(`Could not insert in the webhook request body.`);
  }
  return { msg: "finished!" };
}

Challenge Task 7: Insert a sample document

Test that your add-swagpref webhook response and schema validation works as expected by trying to insert some sample documents.

For example, using curl:

curl -X POST -i -H "Content-Type: application/json" \
  --data '{"firstname":"Bobby", "lastname":"Tables", "swag_type":"jacket", "swag_size": "m"}' \
"https://webhooks.mongodb-stitch.com/api/client/v2.0/app/<APP_ID>/service/offsite/incoming_webhook/add-swagpref?secret=submission2"

Challenge Task 8: Create the get-swagprefs webhook function

Implementation of the get-swagprefs webhook function is very similar to search-listings, but the results are returned sorted in descending order by submission date without a limit.

As the hint in the challenge suggests, the default primary key for each document (_id) is an ObjectId value that includes a timestamp as the leading component. The embedded timestamp is a Unix time value (seconds since the epoch) so if multiple entries are created within the same second the submission order won’t be precise.

For more deterministic ordering by submission date it would be better to add an explicit creation date field when inserting new documents. However, sorting by _id is sufficient for the challenge:

exports = function(payload, response) {
  const mongodb = context.services.get("mongodb-atlas");
  const collection = mongodb.db("employees").collection("preferences");

  console.log(JSON.stringify(payload.query));

  let cursor = collection.find(payload.query).sort({"_id": -1}).toArray();
  cursor.then(function(result){
    response.setStatusCode(200);
    response.setHeader(
      "Content-Type",
      "application/json"
    );
    response.setBody(JSON.stringify(result));
  });

};

Wrapping Up

There were some excellent submissions that covered 100% of our test cases, and a few creative suggestions that went beyond what was asked.

We noticed a few common errors across submissions for this challenge:

• Not ensuring the provided test suite was passing.
• Only implementing functionality to pass the provided tests, rather than meeting the specification.
• Adding unnecessary projection or query manipulation in webhook functions.
• Making assumptions about details that weren't in the spec (for example, upserting swag preferences rather than inserting).

We hope you enjoyed learning about Stitch Rules & Webhooks are ready to take on the next challenge in this series: JAM Session Challenge. Join in and level up your MongoDB skills!