Designing Cloud Data Platforms for AI

Part 8 of the Info2K Data Engineering for AI Series

Artificial Intelligence initiatives often begin with a model, an experiment, or a promising proof of concept.

A team may successfully demonstrate a chatbot, Retrieval-Augmented Generation system, predictive model, or AI assistant using a limited collection of data.

The difficult question comes next:

How can the organisation operate that solution securely, reliably, and at scale?

Moving from an AI experiment to a production capability requires much more than additional computing power. It requires a data platform that can continuously deliver trusted, current, governed, and appropriately protected information.

Cloud services provide many of the necessary building blocks. However, purchasing cloud services does not automatically create a successful data platform.

A platform must bring those services together through a coherent architecture, clear ownership, engineering standards, governance controls, operational processes, and measurable service expectations.

This article is Part 8 of the Info2K Data Engineering for AI series. It explores the capabilities organisations should consider when designing modern cloud data platforms for analytics, machine learning, RAG, AI agents, and other enterprise AI workloads.

Connecting Responsible AI to Platform Architecture

In Part 7, we discussed how responsible AI begins with responsible data.

Principles such as privacy, transparency, lineage, security, accountability, and human oversight are essential. But principles alone are not sufficient.

They must be supported by technical and operational capabilities.

For example:

• Access policies must be enforced by identity and authorisation controls.

• Data quality expectations must be implemented as measurable tests.

• Lineage must be captured across pipelines and transformations.

• Sensitive information must be classified and protected.

• AI inputs and outputs must be monitored.

• Changes must be versioned and auditable.

• Incidents must have owners and response procedures.

A well-designed cloud data platform turns responsible-data principles into repeatable engineering practices.

A Cloud Data Platform Is More Than a Collection of Services

It is easy to describe a platform by listing technologies:

• Object storage

• Data warehouses

• Lakehouses

• Streaming services

• Orchestration tools

• Data catalogues

• Vector databases

• Machine learning platforms

• Business intelligence tools

These technologies are important, but a list of products is not an architecture.

A useful platform should explain:

• How data enters the environment

• Where raw and curated information is stored

• How data is validated and transformed

• How identities and permissions are enforced

• How consumers discover trusted data

• How changes are deployed safely

• How failures are detected and resolved

• How costs are monitored and controlled

• How AI applications receive current, relevant context

• Who owns each capability and dataset

Current official cloud guidance similarly emphasises evolving architectures, purpose-built services, scalable storage, unified governance, security, monitoring, and selecting components according to workload requirements rather than assuming that one technology suits every use case. AWS Prescriptive Guidance and the Azure Architecture Center provide vendor-specific implementations of these broader principles.

The platform is therefore both:

1. A technical architecture, and

2. An operating model for delivering trusted data products.

Begin with Outcomes, Not Technologies

A platform design should begin by understanding the workloads it must support.

Different use cases have different requirements.

A monthly financial report may prioritise consistency, auditability, and controlled transformation.

A fraud-detection system may need low-latency event processing.

A healthcare AI assistant may require current clinical information, terminology context, patient-level permissions, and evidence linked to every generated summary.

A RAG application may require document extraction, chunking, metadata enrichment, permission-aware retrieval, and frequent index updates.

Before selecting technologies, organisations should ask:

• Who will use the platform?

• What decisions will it support?

• How current must the information be?

• What happens if the data is incomplete or delayed?

• Which information is sensitive?

• What evidence must be retained?

• What availability and recovery expectations apply?

• How will success be measured?

• Who is accountable for the data and its use?

Technology selection becomes much clearer once these requirements are understood.

The Logical Layers of a Modern Data Platform

Cloud data platforms can be implemented in many ways, but a useful logical architecture often includes the following layers.

1. Source Systems

Data may originate from:

• Operational databases

• Enterprise applications

• APIs

• Software-as-a-Service platforms

• Medical and scientific systems

• Event streams

• Files and documents

• Images and audio

• Internet of Things devices

• External data providers

Understanding the source is essential.

A platform should capture information about ownership, structure, update frequency, sensitivity, reliability, and permitted use before that data is supplied to AI.

2. Ingestion

The ingestion layer brings information into the platform.

Common patterns include:

• Scheduled batch loads

• Application programming interfaces

• File transfers

• Change Data Capture

• Event streaming

• Message queues

• Document ingestion

• Secure data sharing

The correct pattern depends on latency, volume, reliability, source limitations, and cost.

Not every workload requires real-time processing. A carefully designed daily load can be more appropriate than streaming when the business requirement does not justify additional complexity.

3. Landing and Raw Storage

Many platforms retain an initial representation of received data.

This can support:

• Traceability

• Reprocessing

• Troubleshooting

• Historical investigation

• Recovery from transformation errors

• Comparison with the original source

Raw storage does not mean uncontrolled storage.

Retention, encryption, classification, access restrictions, and deletion obligations still apply. Some sensitive information may require masking, isolation, or exclusion before it reaches a general-purpose landing environment.

4. Transformation and Quality

Data is then standardised, validated, combined, and enriched.

This layer may perform:

• Schema validation

• Type and format standardisation

• Duplicate detection

• Identifier resolution

• Business-rule validation

• Terminology mapping

• Data modelling

• Aggregation

• Metadata enrichment

• Quality measurement

A successful pipeline should not merely report that it completed.

It should provide evidence that the resulting data is appropriate for its intended use.

5. Trusted Data Products

Curated information can be published as managed data products for specific consumers.

A data product should have more than a table name. It should include:

• A defined purpose

• An accountable owner

• Documented meaning

• Quality expectations

• Access policies

• Refresh expectations

• Versioning

• Support arrangements

• Known limitations

This helps teams consume trusted information without repeatedly reverse-engineering source systems.

6. Serving and Retrieval

Different consumers may need different serving technologies.

Examples include:

• Data warehouses for reporting

• Lakehouse tables for analytics and machine learning

• Operational data stores

• Search indexes

• Vector indexes

• Feature stores

• Graph databases

• APIs

• Streaming outputs

The goal is not to force every use case into one storage technology. It is to provide governed access through the most suitable interface while preserving consistent definitions and controls.

7. AI and Consumption

The platform may ultimately support:

• Dashboards and reports

• Machine learning

• Generative AI

• RAG applications

• AI agents

• Decision-support systems

• Data science

• Operational applications

• Secure external data products

AI is therefore a consumer of the data platform, not a separate environment with permission to bypass its governance.

Cross-Cutting Platform Capabilities

Some capabilities should apply across every layer rather than being implemented as isolated additions.

These include:

• Identity and access management

• Encryption and key management

• Metadata and catalogue services

• Data classification

• Lineage

• Quality monitoring

• Orchestration

• Observability

• Audit logging

• Cost management

• Backup and recovery

• Infrastructure automation

• Policy enforcement

• Software delivery controls

Cloud guidance treats monitoring, auditing, compliance, sensitive-data protection, backup, disaster recovery, and high availability as core management and governance concerns, not optional finishing work. Microsoft’s management and governance architecture guidance provides one example of this cross-cutting approach.

Warehouse, Data Lake, or Lakehouse?

These terms are frequently presented as competing choices, but the correct architecture depends on the organisation’s workloads.

Data Warehouse

A data warehouse is often effective for:

• Structured analytics

• Business reporting

• Governed dimensional models

• Predictable queries

• Controlled performance

• Established business metrics

Data Lake

A data lake can support:

• Large volumes of varied data

• Structured and unstructured information

• Data science experimentation

• Historical storage

• Flexible processing

• Document and media collections

Without strong governance and management, however, a data lake can become difficult to understand or trust.

Lakehouse

A lakehouse aims to combine flexible, scalable storage with capabilities traditionally associated with warehouses, such as transactional consistency, structured tables, governance, and query performance.

In practice, an organisation may use several of these patterns together.

The most important questions are not which label is most fashionable, but:

• Can consumers find and understand the data?

• Can quality be measured?

• Are permissions consistently enforced?

• Can transformations be traced?

• Can workloads operate reliably?

• Can the platform evolve without unnecessary duplication?

Batch and Streaming Should Serve the Requirement

Modern platforms often support both batch and streaming workloads.

Batch processing is appropriate when data can be collected and processed at defined intervals.

Streaming becomes useful when the delay between an event and a response materially affects the outcome.

Potential streaming use cases include:

• Fraud detection

• Operational monitoring

• Device telemetry

• Security events

• Time-sensitive clinical alerts

• Live customer interactions

Streaming introduces additional design considerations:

• Event ordering

• Late-arriving data

• Duplicate events

• Replay

• State management

• Failure recovery

• Event-time versus processing-time logic

Real-time architecture should therefore be adopted because the use case requires it, not simply because the technology is available.

AI Introduces New Data Platform Requirements

Traditional analytics platforms primarily prepare structured data for queries, reports, and models.

Generative AI expands the platform’s responsibilities.

An AI-ready cloud platform may need to manage:

• Documents and unstructured data

• Parsing and extraction

• Chunking strategies

• Embeddings

• Vector indexes

• Prompt and configuration versions

• Permission-aware retrieval

• Model access

• Evaluation datasets

• User feedback

• AI interaction logs

• Safety and policy controls

RAG systems should be able to update or remove knowledge without retraining the underlying language model. This can improve currency and governance, but it also requires secure access to source data, consistent permission filtering, and controlled index management. AWS security guidance for generative AI data access discusses these concerns in a production context.

If a source document is corrected, expired, reclassified, or deleted, the retrieval environment should reflect that change.

Otherwise, the AI system may continue to use information that the organisation no longer considers valid or accessible.

Security Must Follow the Data

A secure platform does not rely only on a protected network boundary.

Security should follow data through ingestion, storage, transformation, retrieval, and consumption.

Important controls may include:

• Strong identity management

• Least-privilege access

• Role- or attribute-based authorisation

• Encryption in transit and at rest

• Data masking

• Tokenisation

• Secrets management

• Network isolation

• Audit logging

• Sensitive-data discovery

• Retention and deletion policies

• Environment separation

AI applications must retrieve information according to the permissions of the user and the purpose of the request.

A model should not gain unrestricted access merely because it is operating inside the organisation.

Platform-level guardrails help different AI applications inherit a consistent baseline of security and governance. This is also the approach recommended in AWS guidance for enterprise generative AI platforms.

Observability: Knowing Whether the Platform Can Be Trusted

A pipeline can run successfully while producing unusable data.

Operational monitoring should therefore cover more than infrastructure availability.

Useful signals include:

• Data freshness

• Completeness

• Volume

• Validity

• Duplicate rates

• Schema changes

• Distribution changes

• Failed quality rules

• Processing latency

• Retrieval relevance

• Model and application behaviour

• Cost anomalies

Teams should also define what happens when an expectation is not met.

For example:

• Should publishing be stopped?

• Should consumers receive a warning?

• Is a previous valid version retained?

• Who investigates the failure?

• How quickly must it be resolved?

• Which downstream systems are affected?

A dashboard without ownership or response procedures provides visibility, but not necessarily operational control.

Reproducibility and AI Operations

Production AI systems may change across several dimensions:

• Source data

• Transformations

• Training data

• Models

• Prompts

• Retrieval configurations

• Embedding models

• Index contents

• Evaluation criteria

• Application code

When an output changes, teams need to understand which component caused the change.

This requires coordinated DataOps, MLOps, and, where appropriate, GenAIOps practices.

Versioning, automated testing, controlled deployment, evaluation, and rollback should apply across the system rather than only to application code.

Feedback loops are also valuable. User feedback can help identify retrieval failures, unsafe outputs, missing information, and changing operational needs. However, feedback itself must be validated, governed, and protected before being used for improvement. AWS operational guidance similarly treats structured feedback pipelines and human oversight as production capabilities.

Cost Is an Architectural Requirement

Cloud platforms make it easy to provision resources. They can also make it easy to create uncontrolled cost.

AI workloads may introduce substantial demand through:

• Large-scale data processing

• Repeated document ingestion

• Embedding generation

• Vector search

• Model training

• Model inference

• Storage duplication

• High-frequency streaming

• Extensive logging

Cost management should be built into platform design through:

• Workload isolation

• Budget and usage alerts

• Storage lifecycle policies

• Appropriate compute sizing

• Autoscaling controls

• Query optimisation

• Caching

• Data retention rules

• Chargeback or showback

• Regular architecture reviews

The lowest-cost service is not automatically the most cost-effective choice. Reliability, operational effort, scalability, integration, latency, and maintainability also matter.

Enable Self-Service Without Losing Control

A central platform team cannot manually implement every pipeline, dataset, and AI use case.

Organisations therefore need an appropriate form of self-service.

Self-service should not mean unrestricted access or inconsistent implementation.

A stronger approach is to provide approved paths, sometimes called paved roads, for common activities such as:

• Creating a new pipeline

• Publishing a data product

• Requesting access

• Registering metadata

• Deploying an AI workload

• Monitoring quality

• Managing secrets

• Recording lineage

Reusable templates and automated controls allow teams to move faster while remaining inside agreed architectural and governance boundaries.

The platform team provides shared capabilities and standards. Domain teams contribute business meaning, ownership, and use-case expertise.

A Healthcare Example

Consider a cloud platform supporting an AI assistant that summarises clinical information.

The platform may need to:

1. Ingest data from clinical systems, documents, laboratory feeds, and medication services.

2. Resolve identities carefully.

3. standardise terminology and timestamps.

4. Distinguish current, historical, suspected, and negated information.

5. Apply patient- and user-level access controls.

6. Track the source and version of each record.

7. Detect missing or delayed feeds.

8. Provide evidence supporting the generated summary.

9. Record how the system was used.

10. Support clinical review and escalation.

The AI model is only one component.

The reliability of the overall solution depends on the platform’s ability to preserve meaning, enforce permissions, monitor data, and support human judgement.

In healthcare, cloud scale is useful, but scale without context, governance, and safety is not success.

A Practical Architecture Checklist

Before describing a cloud data platform as ready for AI, ask:

Purpose and Ownership

• Are the priority use cases clearly defined?

• Does every critical data product have an owner?

• Are platform and domain responsibilities understood?

Data

• Are important sources identified and documented?

• Are quality expectations measurable?

• Are meaning, currency, and limitations recorded?

• Can data be corrected or removed throughout the system?

Architecture

• Are batch, streaming, analytics, search, and AI patterns used intentionally?

• Can the architecture evolve?

• Is unnecessary duplication controlled?

• Are availability and recovery requirements defined?

Governance and Security

• Is sensitive data classified?

• Is access based on identity, role, purpose, and policy?

• Are lineage and audit records available?

• Are retention, privacy, and sovereignty requirements implemented?

Operations

• Are freshness, quality, failures, and costs monitored?

• Are incidents assigned to accountable teams?

• Can changes be tested, deployed, and rolled back safely?

• Are AI inputs, retrieval behaviour, and outputs evaluated?

Consumption

• Can users discover trusted data?

• Do published products include meaning and quality expectations?

• Can AI applications obtain current context without bypassing governance?

• Can outputs be connected to supporting evidence?

If these questions cannot be answered, the organisation may have cloud technology, but not yet a dependable data platform for AI.

The Central Message

Cloud services are ingredients.

A production-grade cloud data platform is the coordinated capability that turns those ingredients into trusted outcomes.

It connects:

• Data ingestion

• Storage

• Transformation

• Quality

• Metadata

• Governance

• Security

• Retrieval

• AI

• Monitoring

• Human accountability

The objective is not to create the most complicated architecture.

It is to create the simplest architecture that can reliably satisfy the organisation’s use cases, risks, responsibilities, and future direction.

Successful AI requires more than access to a powerful model.

It requires a platform that can continuously deliver the right data, with the right context and controls, to the right consumer.

Continue the Info2K Data Engineering for AI Series

This article is Part 8 of the Info2K Data Engineering for AI series.

The series includes:

1. The Hidden Data Engineering Behind Successful AI Projects

2. What Does AI-Ready Data Actually Mean?

3. ETL vs ELT for Modern AI Systems

4. Building Reliable RAG Pipelines

5. Data Quality for AI

6. Healthcare NLP: Why Clinical Text Is Different

7. Responsible AI Starts with Responsible Data

8. Designing Cloud Data Platforms for AI

9. Modern Data Engineering as the Foundation of Enterprise AI

In Part 9, we will bring the series together and explore how modern Data Engineering is evolving from pipeline delivery into a strategic foundation for enterprise AI.

About Info2K

Info2K : Information to Knowledge

Info2K shares practical knowledge and provides support across Data Engineering, AI Engineering, Cloud Data Platforms, Digital Health, Healthcare NLP, and responsible technology delivery.

If your organisation is designing a modern data platform, preparing data for AI, improving governance, or moving an AI initiative towards production, please get in touch to discuss how Info2K can support your project.

Website: https://info2k.co.nz

LinkedIn: https://www.linkedin.com/company/info2k