How do you ensure optimal data architecture?

Whether it is a big data or a small data, you are using state of art technology stack or just an excel. Important is to know your data and organize it well so that it is secure, reliable and you can get this quickly when you need it. This is what a data architect does, sounds so simple and often ignored. And post mortem report of many failed application/not able to survive long enough would say "data model of the application was not good enough or nobody could find datamodel/data dictionary for analysis - it may not be there at all" It is never too late, get a data architect for your application, you will see the differnece and it is worth.

I've been hired many times to fix a failing Data Warehouse or Lake. Most were built by engineering and "foisted" on the business creating lack of trust. Keep it simple. The worst stack was Airflow + Python - with way over engineered code. The best I've used was Fivetran + dBt + Snowflake / Snowpark. This stack was fully Integrated and with the SDLC and Github. No Kubernetes or Docker needed, very low maintenance and we had Data Analysts coding their own dBt SQL ELT pipelines. Data Scientists coded their Python ELT pipelines. Full orchestration and monitoring and management and full data observability built in. Our Data Catalog was the icing on the cake. Everyone knew what data we had and which were usable for their business use case

The most important thing for me is to understand the data and the business. In many cases I have seen that data teams concentrate more on technology than understanding the data. This will lead to either over engineered or under engineered data architecture. The first questions we need to answer are: 1.) What data do we have? 2.) What are the insights that business needs from the data? 3.) What insights can we bring out for them from the data? Once we have answered these questions we can look at technologies and in many cases just a traditional SQL Database will suffice. No everyone needs Spark/Hadoop. The architecture will keep evolving and sometimes starting from the basic architecture is a good thing.

Data Architecture is not a one time project that an organization needs to do. It's an investment that an organization will do over a longer period supported by a clear strategy and vision. The major aspects to be considered are: > What kind of data your organization is dealing with like Structured, Unstructured, Semi Structured etc. > How are you are planning for the pipelines for data operations like Extracting, Cleaning and Transforming the data? > Whether you want to invest in Cloud based datawarehousing or inhouse? > How you are going to deploy your AI and ML modeling within that data? > Strong Planning for Data Democratization, Governance and providing Data Insights to the stake holders!

This works on the assumption that data architecture exclusively operates in the data "consumer" space I.e. warehousing, reporting, MI, BI, analytics and AI space. This also works on the assumption that data architecture starts at an enterprise architecture level. In reality data architecture should be a cornerstone of all business systems at all levels. Data Architecture should start with an organisation's operating/transactional systems. Data is never going to be consumed to the pervasiveness and standard required to realise value if it is not produced correctly at the point of capture. The world has moved on from the monolithic systems where data architecture was a by-product of the system and the business is the thing that adapts.

Data Architecture is an influential way of taking Data Strategy to next Data maturity level. On broader level, Data Mesh, Data Fabric, DWH, Lakehouse, Information Mart, DV2 are some of Data Architecture styles. There are several steps of managing data i.e. Integration, Storage, Processing, availability, Orchestration, Data Catalog, Data Linage, Reporting and Visualization layer, Data Security, Data Governance, Data Modelling etc... Each step has multiple options with pros and cons. Choosing right option, as per user case, shapes Data Architecture. In my view, there is no 'right or wrong', Data Architecture. Its fluid & end customer should get Value, on/before time. Its ongoing process to improve Data Architecture.

Again assuming that the only goal of data is use as a product in itself at the macro/enterprise level. Data architecture at the Micro level can deliver more value upfront by automating front line processes, reducing manual entry and providing customers direct access to their data frees up human time to improve quality if products and services whilst increasing capacity for more volume in tandem. This can release real measurable value to a balance sheet long before any macro use cases like a data platform with lakes will. Microservice and event driven architectures require consistent, interoperable data architecture across operational data, messaging, integration and centralised/long term data services.

This statement holds true. But not just data systems. to ensure its consistency, quality, and interoperability across all systems.

Based on my experience in industry model development and used extensively the same another important aspect is scalability. More room in the design for data governance by logging by enabling various audit logs.

Data governance needs to be treated as a program and not a project. There would be multiple projects driving it - Governance frameworks, Tools, Deployments, Catalog, Lineage etc The success depends on adopting it at enterprise level. Different teams/sources might have different flavors of data domain but unless the knowledge is democratized between consumers/producers it’s hard to bring in a change

Good data governance costs. Often one of the main reasons for lack of this kind of governance is that data governance is just about cascading policies and approvals from afar. Good data governance isn't about telling people and projects what they can and can't do because of data risks it's also about funding the skills and technologies to implement, operate and maintain good data governance.

Data governance plays a major role in the modern data world. The nature of the data is unorganized, so good governance is the key to building valuable data assets. The governance starts from the design level.

Don't forget data architecture isn't just a capital investment. Often this area gets overlooked in the ongoing operational costs to maintain a valuable data ecosysyem.