Data Mesh Nedir?

Giriş

Açıklaması şöyle

Data mesh is a highly decentralized data architecture in which independent teams are responsible for managing data within their domains. Each domain or department, such as finance or supply chain, becomes a building block of an ecosystem of data products called mesh.

Tarihçesi

Açıklaması şöyle

The concept of data mesh was created by Zhamak Dehghani, director of next tech incubation at the software consultancy ThoughtWorks. Dehghani attempted to solve problems caused by centralized data infrastructure. She observed how many of her clients that centralized all of their data in one platform found it hard to manage a huge variety of data sources. A centralized setup also forced teams to change the whole data pipeline when responding to new needs. Teams struggled with solving the influx of data requests from other departments, which suffocated innovation, agility, and learning.

Challenges of Data Mesh

Açıklaması şöyle

Decentralized data architecture also leads to several challenges, such as:

- Duplication of Data: Repurposing and replicating data from the source domain to serve other domain’s needs may lead to the duplication of data and higher data management costs.

- Neglected Quality: The existence of multiple data products and pipelines may lead to the neglect of quality principles and huge technical debt.

- Change Management Efforts: Deploying data mesh architecture and decentralized data operations will involve a lot of change management efforts.

- Choosing Future-Proof Technologies: Teams will have to carefully decide on which technologies to use to standardize them across the company and ensure they can tackle future challenges.

Data Mesh vs. Data Fabric

Açıklaması şöyle

The data mesh concept may appear similar to data fabric as both architectures provide access to data across various platforms. But there are several differences between the two.

For one, data fabric brings data to a unified location, while with data mesh, data sets are stored across multiple domains. Also, data fabric is tech-centric because it primarily focuses on technologies, such as purpose-built APIs, and how they can be efficiently used to collect and distribute data. Data mesh, however, goes a step further. It not only requires teams to build data products by copying data into relevant data sets but also introduces organizational changes, including the decentralization of data ownership.

There are various interpretations of how data mesh compares to data fabric. And two companies may introduce different tech solutions for data mesh or data fabric depending on their data size and type, security protocols, employee skillsets, and financial resources.