Automating Adobe AEM on MongoDB Atlas and Microsoft Azure

Introduction

Given the overwhelming trend towards automation in the technology industry we thought it would be a great idea to showcase how you can use MongoDB to do some awesome automation with two of MongoDB's premiere partners, Adobe and Microsoft.

Adobe Experience Manager (AEM) is a leading digital experience management solution. Enterprises like Manulife, Nissan, and Philips rely on AEM to deliver rapidly adaptable experiences across their customer journey. With MongoDB Atlas behind AEM, customers get a comprehensive solution integrated with a highly available, globally scalable, and enterprise-level database. Running all this on Microsoft's Azure cloud infrastructure ensures everything will run smoothly, meet your SLA's, and not break your budget. All this together makes delivering great digital experiences that much easier and allows you and your team to focus on your core missions.

This blog post will run through a straight-forward end-to-end automated deployment of an Adobe AEM Author node using MongoDB Atlas all running on Microsoft Azure for a development or testing environment. Production-level deployments require additional steps, such as nodes for additional AEM components (publish, etc) and high-availability for the AEM nodes for example. We hope you can leverage this example in your own automated workflows.

AEM released version 6.4 in March 2018. One of the exciting new features is ability to support cloud-based deployments for MongoDB. For customers looking to move to the cloud, MongoDB is easy to deploy and even easier when leveraging MongoDB Atlas for a full DBaaS - thus minimizing operational overhead, costs, and additional risk of maintaining your own databases. Atlas is our cloud-hosted MongoDB service engineered and run by the same team that builds the database.

This blog post presents a step-by-step example which automates the deployment of your own standalone AEM Author node along with a MongoDB Atlas replica set in Microsoft Azure. We'll use "development-level" cloud services to keep the costs low, but the steps for production-grade deployments are the same. Simply select beefier servers for your database and AEM instance. And, guess what? We'll actually be able to migrate to more substantial resources without requiring any downtime! How is this possible? MongoDB Atlas supports live cluster migration and AEM supports the ability to host multiple Author nodes.

Getting Ready - Cloud Services Access

Step 0 - 5 minutes

To start let's ensure we have accounts on all the services we need to use. There are 3:

Adobe

Use the Adobe Licensing Website to access the AEM software.

Microsoft Azure

Microsoft Azure will host our AEM server in the cloud.

MongoDB Atlas

For deploying the backing MongoDB replica set. Open MongoDB Atlas and click the green "Get started free" button. You'll actually need to configure your Atlas account payment settings as we'll be deploying a MongoDB cluster that is not free.

If you haven’t already, please use the links above and follow the directions to create your accounts. If you already have accounts for these services, then you’re all set.

Provisioning - Spinning up Cloud Resources

Step 1 - MongoDB Atlas API Access - 5 minutes

We'll start by provisioning our database for AEM to store its content. The goal is to configure a 3 node MongoDB replica set using the M20 Cluster Tier.

Sign into MongoDB Atlas to start managing multiple cloud-based MongoDB deployments. Atlas uses the concepts of Organizations and Projects to help manage these deployments. Each organization can have multiple projects. You can also create teams with various levels of access to each project. The automated deployment presented below requires a set of credentials for API access. Please follow the instructions for configuring API access in the MongoDB Atlas documentation. This is the only manual Atlas step required.

If this is your first visit to Atlas, you'll need to create an organization and a project within that organization. For these, you'll just need to name them something. I used AEM-ON-AZURE and aem-on-azure-db. Note your project name, you'll need to update the deployment script with that name in order to associate the AEM MongoDB database with that group.

Step 2 - Get automation tools and scripts

For this demonstration, we'll be showing some of the great automation capabilities of Azure (MongoDB Atlas also has a rich API along with some open source automation tooling.) To get started, download and install the Azure CLI 2.0: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest Mac users with brew can just run:

brew update && brew install azure-cli

A script which automates launching an AEM Author node can be found in the aem-on-atlas-and-azure Github repository.

git clone https://github.com/jasonmimick/aem-on-atlas-and-azure
cd aem-on-atlas-and-azure

The spin-up-demo-aem-azure.sh script will:

1. Provision a MongoDB Atlas replica-set.
2. Create an Azure resource group.
3. Launch an Ubuntu VM.
4. Create a public IP and firewall rule for the AEM Author HTTP service.
5. Upload your AEM jar file and license.properties.
6. Start the AEM Author node.

We've leveraged an excellent blog our partner Adobe released on running a minimal setup AEM6 on MongoDB for the commands to launch AEM. We're also doing a non-interactive launch which allows us to set the admin password without user intervention (https://helpx.adobe.com/in/experience-manager/6-3/sites/administering/using/security-configure-admin-password.html).

Using your favorite editor, update the variables in the spin-up script.

It's a best practice to use the same region for both the Atlas and Azure components, the script contains 2 variables which define the region (fixed values between Atlas and Azure for "US East" are slightly different formats).

Step 3 - Run spin-up script

You'll just need to login to your Azure account and run the script.

az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AW84CG7T5 to 
authenticate.
./spin-up-demo-aem-azure.sh aem-demo      # DEMO_NAME="aem-demo"


The very last step in the spin-up script starts the AEM author node. Once this is running we can ssh into the new Azure VM and do some inspection. A quick grep in the AEM logs does indeed validate that AEM is connecting to Atlas…

$# Note - put your DEMO_NAME for the 'resourceGroup'
$ssh aem@$(az network public-ip list --query="[?resourceGroup == 'my-aem-demo-61h'] | [].ipAddress" --output tsv)
aem@my-aem-demo-61h-aem-vm:~$grep mongodb ./crx-quickstart/logs/error.log
26.06.2018 16:11:40.133 *INFO* [cluster-ClusterId{value='5b3265bb19c9cf6b565256a1', description='MongoConnection for Oak DocumentMK'}-srv2-shard-00-00-n1oxl.mongodb.net:27017] org.mongodb.driver.connection Opened connection [connectionId{localValue:3, serverValue:46593}] to srv2-shard-00-00-n1oxl.mongodb.net:27017
26.06.2018 16:11:40.137 *INFO* [cluster-ClusterId{value='5b3265bb19c9cf6b565256a1', description='MongoConnection for Oak DocumentMK'}-srv2-shard-00-00-n1oxl.mongodb.net:27017] org.mongodb.driver.cluster Monitor thread successfully connected to server with description ServerDescription{address=srv2-shard-00-00-n1oxl.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 5]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3391937, setName='srv2-shard-0', canonicalAddress=srv2-shard-00-00-n1oxl.mongodb.net:27017, hosts=[srv2-shard-00-00-n1oxl.mongodb.net:27017, srv2-shard-00-01-n1oxl.mongodb.net:27017, srv2-shard-00-02-n1oxl.mongodb.net:27017], passives=[], arbiters=[], primary='srv2-shard-00-01-n1oxl.mongodb.net:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, lastWriteDate=Tue Jun 26 16:11:37 UTC 2018, lastUpdateTimeNanos=5447761739832831}
26.06.2018 16:11:40.169 *INFO* [FelixStartLevel] org.mongodb.driver.connection Opened connection [connectionId{localValue:4, serverValue:96728}] to srv2-shard-00-01-n1oxl.mongodb.net:27017


Checking the Atlas database with the mongo shell shows that our new AEM Author node is indeed able to connect and write data. Note the use of the mongodb+srv format connection string. This feature allows MongoDB replica set nodes to be discovered through DNS SRV records.

$mongo "mongodb+srv://srv2-n1oxl.mongodb.net/test" --username aem
MongoDB shell version v3.6.3
Enter password:
connecting to: mongodb+srv://srv2-n1oxl.mongodb.net/test
2018-06-28T14:58:40.036-0400 I NETWORK  [thread1] Starting new replica set monitor for srv2-shard-0/srv2-shard-00-02-n1oxl.mongodb.net.:27017,srv2-shard-00-00-n1oxl.mongodb.net.:27017,srv2-shard-00-01-n1oxl.mongodb.net.:27017
2018-06-28T14:58:40.327-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-02-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-02-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.333-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-00-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-00-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.690-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-01-n1oxl.mongodb.net.:27017 (1 connections now open to srv2-shard-00-01-n1oxl.mongodb.net.:27017 with a 5 second timeout)
2018-06-28T14:58:40.690-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-01-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-01-n1oxl.mongodb.net:27017 with a 5 second timeout)
2018-06-28T14:58:40.765-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] changing hosts to srv2-shard-0/srv2-shard-00-00-n1oxl.mongodb.net:27017,srv2-shard-00-01-n1oxl.mongodb.net:27017,srv2-shard-00-02-n1oxl.mongodb.net:27017 from srv2-shard-0/srv2-shard-00-00-n1oxl.mongodb.net.:27017,srv2-shard-00-01-n1oxl.mongodb.net.:27017,srv2-shard-00-02-n1oxl.mongodb.net.:27017
2018-06-28T14:58:42.015-0400 I NETWORK  [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to srv2-shard-00-00-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-00-n1oxl.mongodb.net:27017 with a 5 second timeout)
2018-06-28T14:58:42.015-0400 I NETWORK  [thread1] Successfully connected to srv2-shard-00-02-n1oxl.mongodb.net:27017 (1 connections now open to srv2-shard-00-02-n1oxl.mongodb.net:27017 with a 5 second timeout)
MongoDB server version: 3.6.5
MongoDB Enterprise srv2-shard-0:PRIMARY> show dbs
admin       0.000GB
aem-author  0.320GB
config      0.000GB
local       0.936GB
MongoDB Enterprise srv2-shard-0:PRIMARY> use aem-author
switched to db aem-author
MongoDB Enterprise srv2-shard-0:PRIMARY> show tables
blobs
clusterNodes
journal
nodes
settings
MongoDB Enterprise srv2-shard-0:PRIMARY>


Wrap up

The aem-on-atlas-and-azure repository also contains a script to clean things up. The tear-down-aem-on-atlas-and-azure.sh bash script will attempt to read the latest configuration from the log file created by the spin-up script and then delete the corresponding Atlas and Azure resources.

This post reviewed an automated procedure for provisioning a complete AEM Author node hosted in the cloud. We hope you've found this informational and are able to apply these techniques in your own AEM deployments. An additional benefit to these kinds of deployments is that they are automatically integrated with the Atlas and Azure built in monitoring and alerting systems. Are you currently or planning to provision AEM in the cloud within your organization? If so, head over to our MongoDB community Slack, join the #aem channel, and let us know!