Unlocking the Value of Enterprise Data with Solace PubSub+ and deltamap

Mathew Hobbis

Solace Chief Arhitect FSI

In today’s data-rich world, effectively managing and optimising organisational data is crucial for success. Proper data sharing provides valuable insights, drives strategic decisions, and creates a competitive advantage. However, uncontrolled data sharing can lead to legal, ethical, and security issues. Organisations must navigate a complex landscape of regulations and best practices to share data responsibly.

One significant challenge is that organisations often lack a clear understanding of their current data landscape. Questions like where data is located, who owns it, how to prove its integrity and provenance, and who can access it are common. Effective data sharing and control are complicated by these uncertainties.

In this blog, we’ll explore how organisations can use the Solace PubSub+ platform and deltamap to address these challenges and unlock the full value of their enterprise data.

Solace PubSub+: Realising Effective Event-Driven Architecture (EDA)

Event-Driven Architecture (EDA) allows real-time data sharing between data generators and consumers, enabling organisations to be agile and responsive. Solace PubSub+ is an industry-leading platform for EDA, providing efficient transport for events through an Event Mesh that intelligently routes information. It supports hybrid, multi-cloud deployments, and application portability without painful reconfiguration.

Key Features of Solace PubSub+:

1.       Event Brokers and Event Mesh:

• The PubSub+ Event Brokers provide efficient and effective transport for events throughout the organisation.

• These brokers can be connected to form an Event Mesh that intelligently routes information from the source to the sink(s) by automatically learning and building a subscription routing table.

• Deployment options include traditional data centers, private cloud, public cloud, or any hybrid mix, enabling flexible and scalable solutions.

2.       Integration and APIs:

• A comprehensive set of Open APIs, Protocols, Micro-Integrations, and iPaaS connectors provide flexible on-ramps and off-ramps for the Event Mesh, ensuring all end systems can be integrated smoothly.

3.       Event Portal:

• The Event Portal offers a catalogue and design-time view of schemas, events, event APIs, and applications/microservices interacting over the Event Mesh.

• This portal facilitates unambiguous communication between application teams, promoting collaboration and event reuse.

4.       Low-Code Application Development:

• The Event Portal allows exporting application components into AsyncAPI definitions, which can be used with open-source code generation tools for low-code application development.

• A typical application lifecycle includes discovering events, designing applications, generating code, integrating with change request processes, deploying applications, and auditing runtime performance.

  
  

deltamap: Enhancing Data Management, Lineage, and Security

deltamap complements PubSub+ by providing deep insights, control, and governance over data assets. It tracks data movement, integrity, and usage in real-time, creating an accurate snapshot of the data landscape. deltamap is built on Cilium, an open-source technology providing networking, security, and observability for cloud-native environments. Additionally, deltamap leverages reinforcement learning to continually improve its understanding and management of data flows.

One of the standout features of deltamap is its largely zero-configuration setup, making it easy to deploy and start using with minimal manual configuration. Furthermore, deltamap provides both historical and current views of data, allowing organisations to analyze trends over time and maintain a comprehensive understanding of their data landscape.

deltamap supports over 100 use cases, including data sovereignty, GDPR compliance, BCBS regulations, DCAM data quality management, and observing schema evolution, making it a versatile tool for a wide range of data management needs.

Key Features of deltamap:

1.       Real-Time Data Tracking:

• deltamap provides automated data tracking, enabling organisations to visualise how data moves from its source through various stages of processing to its destination.

2.       Data Lineage and Governance:

• It offers a clear and intuitive view of data flows and lineage, crucial for understanding, analyzing, and optimising data movement to align with business objectives.

3.       Hashing and Indexing:

• deltamap uses hashing and indexing to store references to data flows without creating additional copies, maintaining a current view of the data landscape.

• It also has a historical view of your data landscape which it uses to predict a future view.

4.       Anomaly Detection:

• The platform detects anomalies in data flows, such as unexpected schema changes, unusual latencies, or data residency violations, and generates alerts for quick resolution.

5.       Built on Cilium:

• By using Cilium, deltamap benefits from advanced networking and security features designed for cloud-native environments. Cilium's eBPF-based data plane provides high-performance observability and security for data in motion.

6.       Reinforcement Learning:

• deltamap employs reinforcement learning algorithms to learn and adapt to the organisation's data flows, enhancing its ability to detect anomalies, optimise data usage, and maintain effective governance with minimal human intervention.

7.       Zero-Configuration Setup:

• deltamap features a largely zero-configuration setup, enabling quick deployment and use without extensive manual configuration. This reduces the complexity and time required to start benefiting from its capabilities.

8.       Historical and Current Views:

• deltamap provides both historical and current views of data, allowing organisations to analyze trends over time and maintain a comprehensive understanding of their data landscape.

Combining PubSub+ and deltamap

Using deltamap with PubSub+ provides real-time validation of your EDA goals. deltamap learns data flows and can compare them to design intents, flagging anomalies and compliance issues as they arise. This ensures that data governance remains effective even as requirements change.

For example, deltamap can detect unexpected schema changes, track data paths for regulatory compliance, and enforce geofencing rules to prevent data residency violations. This real-time monitoring helps organisations minimise exposure to potential regulatory issues.

Conclusion

Solace PubSub+ and deltamap together offer a comprehensive solution for unlocking the value of enterprise data. By enabling controlled, governed, and monitored data sharing, these platforms enhance operational control, support regulatory compliance, and improve decision-making. Our partnership with Solace makes accessing and indexing data in motion even more accessible, supporting countless use cases and potentially impacting ESG initiatives.