Introduction
This is the third in a series of blog posts examining the technologies that are driving the development of modern web and mobile applications.
Part 1: Introducing The MEAN Stack (and the young MERN upstart) introduced the technologies making up the MEAN (MongoDB, Express, Angular, Node.js) and MERN (MongoDB, Express, React, Node.js) Stacks, why you might want to use them, and how to combine them to build your web application (or your native mobile or desktop app).
The remainder of the series is focused on working through the end to end steps of building a real (albeit simple) application. – MongoPop. Part 2: Using MongoDB With Node.js created an environment where we could work with a MongoDB database from Node.js; it also created a simplified interface to the MongoDB Node.js Driver.
This post builds on from those first posts by using Express to build a REST API so that a remote client can work with MongoDB. You will be missing a lot of context if you have skipped those posts – it's recommended to follow through those first.
The REST API
A Representational State Transfer (REST) interface provides a set of operations that can be invoked by a remote client (which could be another service) over a network, using the HTTP protocol. The client will typically provide parameters such as a string to search for or the name of a resource to be deleted.
Many services provide a REST API so that clients (their own and those of 3rd parties) and other services can use the service in a well defined, loosely coupled manner. As an example, the Google Places API can be used to search for information about a specific location:
Breaking down the URI used in that curl
request:
- No method is specified and so the
curl
command defaults to a HTTPGET
. maps.googleapis.com
is the address of the Google APIs service./maps/api/place/details/json
is the route path to the specific operation that's being requested.placeid=ChIJKxSwWSZgAUgR0tWM0zAkZBc
is a parameter (passed to the function bound to this route path), identifying which place we want to read the data for.key=AIzaSyC53qhhXAmPOsxc34WManoorp7SVN_Qezo
is a parameter indicating the Google API key, verifying that it's a registered application making the request (Google will also cap, or bill for, the number of requests made using this key).
There's a convention as to which HTTP method should be used for which types of operation:
- GET: Fetches data
- POST: Adds new data
- PUT: Updates data
- DELETE: Removes data
Mongopop's REST API breaks this convention and uses POST
for some read requests (as it's simpler passing arguments than with GET
).
These are the REST operations that will be implemented in Express for Mongopop:
Route Path | HTTP Method | Parameters | Response | Purpose |
---|---|---|---|---|
/pop/ |
GET | { "AppName": "MongoPop", "Version": 1.0 } |
Returns the version of the API. | |
/pop/ip |
GET | {"ip": string} |
Fetches the IP Address of the server running the Mongopop backend. | |
/pop/config |
GET | { mongodb: { defaultDatabase: string, defaultCollection: string, defaultUri: string }, mockarooUrl: string } |
Fetches client-side defaults from the back-end config file. | |
/pop/addDocs |
POST | { MongoDBURI: string, collectionName: string, dataSource: string, numberDocs: number, unique: boolean } |
{ success: boolean, count: number, error: string } |
Add numberDocs batches of documents, using documents fetched from dataSource |
/pop/sampleDocs |
POST | { MongoDBURI: string, collectionName: string, numberDocs: number } |
{ success: boolean, |
Read a sample of the documents from a collection. |
/pop/countDocs |
POST | { MongoDBURI: string, collectionName: string } |
{ success: boolean, |
Counts the number of documents in the collection. |
/pop/updateDocs |
POST | { MongoDBURI: string, collectionName: string, matchPattern: Object, dataChange: Object, threads: number } |
{ success: boolean, count: number, error: string } |
Apply an update to all documents in a collection which match a given pattern |
Express
Express is the web application framework that runs your back-end application (JavaScript) code. Express runs as a module within the Node.js environment.
Express can handle the routing of requests to the right functions within your application (or to different apps running in the same environment).
You can run the app's full business logic within Express and even use an optional view engine to generate the final HTML to be rendered by the user's browser. At the other extreme, Express can be used to simply provide a REST API – giving the front-end app access to the resources it needs e.g., the database.
The Mongopop application uses Express to perform two functions:
- Send the front-end application code to the remote client when the user browses to our app
- Provide a REST API that the front-end can access using HTTP network calls, in order to access the database
Downloading, running, and using the application
The application's Express code is included as part of the Mongopop package installed in Part 2: Using MongoDB With Node.js.
What are all of these files?
A reminder of the files described in Part 2:
package.json
: Instructs the Node.js package manager (npm
) on what it needs to do; including which dependency packages should be installednode_modues
: Directory wherenpm
will install packagesnode_modues/mongodb
: The MongoDB driver for Node.jsnode_modues/mongodb-core
: Low-level MongoDB driver library; available for framework developers (application developers should avoid using it directly)javascripts/db.js
: A JavaScript module we've created for use by our Node.js apps (in this series, it will be Express) to access MongoDB; this module in turn uses the MongoDB Node.js driver.
Other files and directories that are relevant to our Express application:
config.js
: Contains the application–specific configuration optionsbin/www
: The script that starts an Express application; this is invoked by thenpm start
script within thepackage.json
file. Starts the HTTP server, pointing it to theapp
module inapp.js
app.js
: Defines the main application module (app
). Configures:- That the application will be run by Express
- Which routes there will be & where they are located in the file system (
routes
directory) - What view engine to use (Jade in this case)
- Where to find the /views/ to be used by the view engine (
views
directory) - What middleware to use (e.g. to parse the JSON received in requests)
- Where the static files (which can be read by the remote client) are located (
public
directory) - Error handler for queries sent to an undefined route
views
: Directory containing the templates that will be used by the Jade view engine to create the HTML for any pages generated by the Express application (for this application, this is just the error page that's used in cases such as mistyped routes ("404 Page not found"))routes
: Directory containing one JavaScript file for each Express routeroutes/pop.js
: Contains the Express application for the/pop
route; this is the implementation of the Mongopop REST API. This defines methods for all of the supported route paths.
public
: Contains all of the static files that must be accessible by a remote client (e.g., our Angular to React apps). This is not used for the REST API and so can be ignored until Parts 4 and 5.
The rest of the files and directories can be ignored for now – they will be covered in later posts in this series.
Architecture
The new REST API (implemented in routes/pop.js
) uses the javascripts/db.js
database layer implemented in Part 2 to access the MongoDB database via the MongoDB Node.js Driver. As we don't yet have either the Angular or React clients, we will user the curl
command-line tool to manually test the REST API.
Code highlights
config.js
The config
module can be imported by other parts of the application so that your preferences can be taken into account.
expressPort
is used by bin/www
to decide what port the web server should listen on; change this if that port is already in use.
client
contains defaults to be used by the client (Angular or React). It's important to create your own schema at Mockaroo.com and replace client.mockarooUrl
with your custom URL (the one included here will fail if used too often).
bin/www
This is mostly boiler-plate code to start Express with your application. This code ensures that it is our application, app.js
, that is run by the Express server:
This code uses the expressPort
from config.js
as the port for the server to listen on; it will be overruled if the user sets the PORT
environment variable:
app.js
This file defines the app
module ; much of the contents are boilerplate (and covered by comments in the code) but we look here at a few of the lines that are particular to this application.
Make this an Express application:
Define where the views (templates used by the Jade view engine to generate the HTML code) and static files (files that must be accessible by a remote client) are located:
Create the /pop
route and associate it with the file containing its code (routes/pop.js
):
routes/pop.js
This file implements each of the operations provided by the Mongopop REST API. Because of the the /pop
route defined in app.js
Express will direct any URL of the form http://<mongopop-server>:3000/pop/X
here. Within this file a route handler is created in order direct incoming requests to http://<mongopop-server>:3000/pop/X
to the appropriate function:
As the /pop
route is only intended for the REST API, end users shouldn't be browsing here but we create a top level handler for the GET
method in case they do:
This is the first time that we see how to send a response to a request; res.json(testObject);
converts testObject
into a JSON document and sends it back to the requesting client as part of the response message.
The simplest useful route path is for the GET
method on /pop/ip
which sends a response containing the IP address of the back-end server. This is useful to the Mongopop client as it means the user can see it and add it to the MongoDB Atlas whitelist. The code to determine and store publicIP
is left out here but can be found in the full source file for pop.js
.
We've seen that it's possible to test GET
methods from a browser's address bar; that isn't possible for POST
methods and so it's useful to be able to test using the curl
command-line command:
The GET
method for /pop/config
is just as simple – responding with the client-specific configuration data:
The results of the request are still very simple but the output from curl
is already starting to get messy; piping it through python -mjson.tool
makes it easier to read:
The simplest operation that actually accesses the database is the POST
method for the /pop/countDocs
route path:
database
is an instance of the object prototype defined in javascripts/db
(see The Modern Application Stack – Part 2: Using MongoDB With Node.js) and so all this method needs to do is use that object to:
- Connect to the database (using the address of the MongoDB server provided in the request body). The results from the promise returned by
database.connect
is passed to the function(s) in the first.then
clause. Refer back to Part 2: Using MongoDB With Node.js if you need a recap on using promises. - The function in the
.then
clause handles the case where thedatabase.connect
promise is resolved (success). This function requests a count of the documents – the database connection information is now stored within thedatabase
object and so only the collection name needs to be passed. The promise returned bydatabase.countDocuments
is passed to the next.then
clause. Note that there is no second (error) function provided, and so if the promise fromdatabase.connect
is rejected, then that failure passes through to the next.then
clause in the chain. - The second
.then
clause has two functions:- The first is invoked if and when the promise is resolved (success) and it returns a success response (which is automatically converted into a resolved promise that it passed to the final
.then
clause in the chain).count
is the value returned when the promise from the call todatabase.countDocuments
was resolved. - The second function handles the failure case (could be from either
database.connect
ordatabase.countDocuments
) by returning an error response object (which is converted to a resolved promise).
- The first is invoked if and when the promise is resolved (success) and it returns a success response (which is automatically converted into a resolved promise that it passed to the final
- The final
.then
clause closes the database connection and then sends the HTTP response back to the client; the response is built by converting theresultObject
(which could represent success or failure) to a JSON string.
Once more, curl
can be used from the command-line to test this operation; as this is a POST
request, the --data
option is used to pass the JSON document to be included in the request:
curl
can also be used to test the error paths. Cause the database connection to fail by using the wrong port number in the MongoDB URI:
Cause the count to fail by using the name of a non-existent collection:
The POST
method for the pop/sampleDocs
route path works in a very similar way:
Testing this new operation:
The POST
method for pop/updateDocs
is a little more complex as the caller can request multiple update operations be performed. The simplest way to process multiple asynchronous, promise-returning function calls in parallel is to build an array of the tasks and pass it to the Promise.all
method which returns a promise that either resolves after all of the tasks have succeeded or is rejected if any of the tasks fail:
Testing with curl
:
The final method uses example data from a service such as Mockaroo to populate a MongoDB collection. A helper function is created that makes the call to that external service:
That function is then used in the POST
method for /pop/addDocs
:
This method is longer than the previous ones – mostly because there are two paths:
- In the first path, the client has requested that a fresh set of 1,000 example documents be used for each pass at adding a batch of documents. This path is much slower and will eat through your Mockaroo quota much faster.
- In the second path, just one batch of 1,000 example documents is fetched from Mockaroo and then those same documents are repeatedly added. This path is faster but it results in duplicate documents (apart from a MongoDB-created
_id
field). This path cannot be used if the_id
is part of the example documents generated by Mockaroo.
So far, we've used the Chrome browser and the curl
command-line tool to test the REST API. A third approach is to use the Postman Chrome app:
Debugging Tips
One way to debug a Node.js application is to liberally sprinkle console.log
messages throughout your code but that takes extra effort and bloats your code base. Every time you want to understand something new, you must add extra logging to your code and then restart your application.
Developers working with browser-side JavaScript benefit from the excellent tools built into modern browsers – for example, Google's Chrome Developer Tools which let you:
- Browse code (e.g. HTML and JavaScript)
- Add breakpoints
- View & alter contents of variables
- View and modify css styles
- View network messages
- Access the console (view output and issue commands)
- Check security details
- Audit memory use, CPU, etc.
You open the Chrome DevTools within the Chrome browser using "View/Developer/Developer Tools".
Fortunately, you can use the node-debug
command of node-inspector
to get a very similar experience for Node.js back-end applications. To install node-inspector
:
node-inspector
can be used to debug the Mongopop Express application by starting it with node-debug
via the express-debug
script in package.json
:
To run the Mongopop REST API with node-debug
, kill the Express app if it's already running and then execute:
Note that this automatically adds a breakpoint at the start of the app and so you will need to skip over that to run the application.
Depending on your version of Node.js, you may see this error:
If you do, apply this patch to /usr/local/lib/node_modules/node-inspector/lib/InjectorClient.js
.
Summary & what's next in the series
Part 1: Introducing The MEAN Stack provided an overview of the technologies that are used by modern application developers – in particular, the MERN and MEAN stacks. Part 2: Using MongoDB With Node.js set up Node.js and the MongoDB Driver and then used them to build a new Node.js module to provide a simplified interface to the database.
This post built upon the first two of the series by stepping through how to implement a REST API using Express. We also looked at three different ways to test this API and how to debug Node.js applications. This REST API is required by both the Angular (Part 4) and React (Part 5) web app clients, as well as by the alternative UIs explored in Part 6.
The next part of this series implements the Angular client that makes use of the REST API – at the end of that post, you will understand the end-to-end steps required to implement an application using the MEAN stack.
Continue to follow this blog series to step through building the remaining stages of the MongoPop application:
- Part 1: Introducing The MEAN Stack (and the young MERN upstart)
- Part 2: Using MongoDB With Node.js
- Part 3: Building a REST API with Express.js
- Part 4: Building a Client UI Using Angular 2 (formerly AngularJS) & TypeScript
- Part 5: Using ReactJS, ES6 & JSX to Build a UI (the rise of MERN)
- Part 6: Browsers Aren't the Only UI – Mobile Apps, Amazon Alexa, Cloud Services...
If you're interested in learning everything you need to know to get started building a MongoDB-based app you can sign up for one of our free online MongoDB University courses.
Node.js Developers today!