What are the key steps to consider when modeling data for microservices architecture?

SQL Databases: Applications with structured and relational data. For applications where data integrity and consistency are crucial, and where relationships between different data entities need to be maintained New Sql - Aimed at addressing scalability issues while retaining ACID properties . When you need the scalability of NoSQL databases but require strong consistency and ACID compliance NoSQL Databases: For unstructured or semistructured data, and horizontal scalability is essential. Types : Document Databases (MongoDB): flexible and evolving schemas. Wide Column Stores (Cassandra): large amounts of data with high write and read throughput. InMemory Databases (Redis): low-latency and fast data access are critical.

Yes we can choose different data storage strategy for different services in micro services. But maintaining consistency amongst the different part of application also has to be thought of. Depending on choice of data we may have different way to manage consistency. Just to provide an example : 2 Phase commit cannot be used when a logica operation spans multiple services, thereby multiple data stores. And NoSQL do not support them either. ACID property is not guaranteed here, design now is for availability thereby we are losing Isolation. When concurrent request are fired we need certain locking or some mechanism to manage that. These are just few of the possible outcome of what data storage is choose

In microservices architecture, each service is independent, allowing for flexibility and scalability. However, this independence creates a challenge in data modeling, as different microservices may have varying views of the same data. The key trade-off involves maintaining data consistency while preserving service autonomy. To address this, strategies like data contracts are used. Data contracts define how data is shared and ensure consistent data models across services. This approach balances the autonomy of microservices with the need for unified data views, but it can increase complexity in system design and coordination.

Data contracts, which specify data format, structure, and semantics, ensure seamless communication among microservices or external systems. No matter which standard is most used, it plays a vital role in maintaining data consistency and compatibility across diverse services and platforms. This is a step that should be taken proactively, so you can not only protect the system against data loss or corruption but also foster a robust ecosystem for scalable microservices architecture.

One part of problem that we solve with data layer is storing data related to business context say for example order creation in restaurant app. But that's not all, We need to publish events to be consumed by other services may order fulfilment service, delivery service etc.. At Event publisher end - problems are how to raise relevant events without missing any important data. At consumer end, we have different issues like what if consumer crashed before fully consuming the event. For such cases - we will need atomic transactions - one more data store apart from data store for storing business entities like order. The second one will be used to managed events, Again there are more to this..more details can be added on subject.

Measuring latency of system as whole need to be done. This could open many possibilities. Recently we encountered a problem where we realised data volume is the issue. But just increasing CPU or Memory was not the solution but archival of older records solved the issue.