IData - PeerIslands Data Sync Accelerator

Today’s enterprises are in the midst of digital transformation, but they’re hampered by monolithic, on-prem legacy applications that don’t have the speed, agility and responsiveness required for digital applications. To make the transition, enterprises are migrating to the cloud. MongoDB has partnered with PeerIslands to develop 1Data, a reference architecture and solution accelerator that helps users with their cloud modernization.

This post details the challenges enterprises face with legacy systems and walks through how working with 1Data helps organizations expedite cloud adoption.

Modernization Trends

As legacy systems become unwieldy, enterprises are breaking them down into microservices and adopting cloud native application development. Monolith-to-microservices migration is complex, but provides value across multiple dimensions. These include:

• Development velocity
• Scalability
• Cost-of-change reduction
• Ability to build multiple microservice databases concurrently

One common approach for teams adopting and building out microservices is to use domain driven design to break down the overall business domain into bounded contexts first. They also often use the Strangler Fig pattern to reduce the overall risk, migrate incrementally, and then decommission the monolith once all required functionality is migrated. While most teams find this approach works well for the application code, it’s particularly challenging to break down monolithic databases into databases that meet the specific needs of each microservice. There are several factors to consider during transition:

1. Duration. How long will the transition to microservices take?
2. Data synchronization. How much and what types of data need to be synchronized between monolith and microservice databases?
3. Data translation in a heterogeneous schema environment. How do the same data elements are processed and stored differently?
4. Synchronization cadence. How much data needs syncing, and how often (real-time, nightly, etc.)? 5.Data anti-corruption layer. How do you ensure new data doesn’t corrupt the old data and ensures transaction data integrity?

Simplifying Migration to the Cloud

Created by PeerIslands and MongoDB, 1Data helps enterprises address the challenges detailed above.

Migrate and synchronize your data with confidence with 1Data

1. Schema migration tool. Convert legacy DB schema and related components automatically to your target MongoDB instance. Use the GUI-based data mapper to track errors.
2. Real-time data sync pipeline. Sync data between monolith and microservice databases nearly in real time with enterprise grade components.
3. Conditional data sync. Define how to slice the data you’re planning to sync.
4. Data cleansing. Data translation as it’s moved.
5. DSLs for data transformation. Apply domain-specific business rules for the MongoDB documents you want to create from your various aggregated source system tables. This layer also acts as an anti-corruption layer.
6. Data auditing. Independently verify data sync between your source and target systems.
7. Go beyond the database. Synchronize data from APIs, Webhooks & Events.
8. Bidirectional data sync. Replicate key microservice database updates back to the monolithic database as needed.

Get Started with Real Time Data Synchronization

With the initial version of 1Data, PeerIslands addresses the core functionality of real time data sync between source and target systems.

Here’s a view of the logical architecture:

Source System. The source system can be a relational database like Oracle, where we’ll rely on CDC, or other sources like Events, API, or Webhooks.

Data Capture & Streaming. This stage captures the required data from the source system and converts them into data streams using either off-the-shelf DB connectors or custom connectors, depending on the source type. 1Data implements data sharding and throttling, which enable data synchronization at scale, in this phase.

Data Transformation. This is the core of the accelerator, when we convert the source data streams into target MongoDB document schemas. We use LISP-based Domain Specific Language to enable simple, rule-based data transformation, including user-defined rules.

Data Sink & Streaming. Captures the data streams that need to be updated into the MongoDB database through stream consumers. The actual update into the target DB is done through sink connectors.

Target system. The target system is the MDB database used by the microservices.

Auditing. Most data that gets migrated is enterprise-critical; 1Data audits the entire data synchronization process for missed data and incorrect updates.

Two way sync. The logical architecture can be utilized to enable data synchronization from the MongoDB database back to the source database.

We used MongoDB, Confluent Kafka and Debezium to implement this initial version of 1Data:

The technical architecture is cloud agnostic and can be deployed on-prem as well. We’ll be customizing it for key cloud platforms as well as fleshing out specific architectures to adopt for common data sync scenarios.

Conclusion

The 1Data solution accelerator lends itself to multiple use cases, from single view to legacy modernization.