Data Mesh Design Pattern
A data mesh is an architectural and organizational paradigm to enable the management and access of big, complex and disparate data. It's particularly relevant in organizations where data is often spread across various systems and teams or where organizations need to share data in a b2b setting. The key aspects of a data mesh include:
Domain-oriented Data Ownership: Rather than centralizing data management, a data mesh advocates for distributing the responsibility. Each domain (like marketing, sales, operations) within an organization owns and manages its own data, which includes responsibility for data quality, governance, and sharing.
Data as a Product: In a data mesh, data is treated as a product, meaning it's well-documented, high-quality, and easy to use. This approach ensures that data is accessible and valuable to those who need it internal or external the domain.
Self-serve Data Infrastructure: To support domain teams, a data mesh relies on a secured, self-serve infrastructure at the domain level. Domains are capable of authorizing their participants as well as allocating compute environments for products.
Federated Computation & Governance: Both compute and user authorization is shared across the framework into domains, ensuring that data usage complies with laws, regulations, and company policies, while still empowering domain teams to operate independently. Queries and compute can be orchestrated through a federated framework.
In a Data Mesh, the logical separator between major groups of participants is called a Domain, and participants play either a Provider or a Consumer role. That is they are providing data to for other participants to consume or visa versa. Domains are owned by Providers and Providers are in the business of delivering data products to Consumers by configuring entry points on the existing Mesh platform infrastructure. Domains employ various types of token contracts to facilitate a market for procuring policies. Consumers obtain policies to operate over these data resources by interacting with Mesh smart contracts in a Domain. For best results, use the Mesh SDK or official Mesh Dapp (coming soon).
This is a diagram showing the abstraction we are attempting to provide for developers building on IDM. Circles represent various token standards such as Fungible, Non Fungible, and Semi Fungible. In the Instruxi Data Mesh, developers create instances of smart contracts to issue tokens of various types. The data mesh infrastructure is used to build applications on top of. Keep in mind that token use cases can vary greatly. For example, a Service Account token may be used to represent an individual's identity while a Credit may be used to represent the fractional ownership of an asset compared to its whole. Group Policies make great governance backends for data sharing.
For additional information [see]
Updated 6 months ago