Data Quality for AI: Why Clean Data Is Not Enough

Part 5 of the Info2K Data Engineering for AI Series

Artificial Intelligence systems depend on data, but having data available does not necessarily mean that it is reliable, appropriate, or safe to use.

In Part 2 of this series, we defined AI-ready data as information that is reliable, contextual, governed, accessible and suitable for a particular AI use case. Data quality is one essential part of that broader readiness. In this article, we examine that component in greater depth: how quality is measured, how acceptable thresholds are determined, how failures are controlled and how trust is maintained after deployment.

Organisations often describe data as “clean” when:

• required fields are populated;

• formats are valid;

• duplicate records have been removed;

• the schema is correct;

• the pipeline completed successfully.

These are important technical checks. However, they do not fully answer the question that matters most:

Is this data trustworthy enough for the specific AI use case?

A dataset can pass every structural validation rule and still represent the wrong person, contain outdated information, omit an important population, lose essential context, or use definitions that differ across systems.

For AI, data quality must go beyond cleaning. It must consider meaning, context, intended use, risk, ownership, monitoring, and what happens when expectations are not met.

This article explores how organisations can move from basic data cleaning towards measurable, continuously managed, fit-for-purpose data quality.

Clean Data Is Not Necessarily Fit-for-Purpose Data

The phrase “clean data” can create a false sense of confidence.

Consider a date field that follows the required YYYY-MM-DD format. Technically, the value is valid. But it may represent the wrong event, use the wrong timezone, or refer to information that is no longer current.

A customer identifier may contain the correct number of characters but be associated with the wrong customer.

A medical diagnosis may be correctly extracted from a clinical note but refer to:

• a historical condition;

• a suspected condition;

• a family member;

• or a condition explicitly ruled out.

The values may look clean while the meaning is wrong.

This is why data quality should be evaluated against the intended use.

Instead of asking only:

Is the data clean?

We should ask:

• Is it accurate enough for this decision?

• Is it sufficiently complete?

• Is it current enough?

• Does it represent the correct population?

• Has its context been preserved?

• Can we trace where it came from?

• Are users authorised to access it?

• What happens if it is wrong?

These questions connect data quality to practical outcomes.

The Main Dimensions of Data Quality

Data quality is multi-dimensional. No single score can fully represent whether data is trustworthy.

Common dimensions include:

Accuracy

Does the data correctly represent the real-world entity, event, or condition?

An address may be structurally valid but no longer belong to the customer. A laboratory value may be correctly stored but associated with the wrong patient.

Completeness

Are the required values present?

Completeness does not mean that every field must always contain a value. It means that the information necessary for the intended use is available.

Validity and Conformity

Does the data follow expected formats, ranges, types, and permitted values?

Examples include valid dates, recognised country codes, acceptable measurement ranges, and correctly structured identifiers.

Consistency

Does the same information have compatible definitions and values across systems?

For example, different platforms may use different definitions for an “active customer,” “current medication,” or “completed transaction.”

Uniqueness

Are duplicate entities or events being represented unintentionally?

Duplicate customer, patient, document, or transaction records can distort analytics and cause AI systems to retrieve conflicting information.

Timeliness and Freshness

Is the information current enough for the intended decision?

A dataset refreshed daily may be suitable for monthly reporting but unsuitable for real-time fraud detection or time-sensitive healthcare workflows.

Microsoft Purview operationalises several of these dimensions through configurable rules, scheduled scans, scores, monitoring, and quality reporting. However, the dimensions that matter, and their acceptable thresholds must still be selected in relation to the use case. Microsoft Purview data-quality documentation

AI Introduces Additional Quality Considerations

Traditional data-quality dimensions remain important, but AI systems introduce additional concerns.

Context

Does the system understand what the data means, where it came from, when it applies, and how it should be interpreted?

Context may include:

• source and owner;

• effective and expiry dates;

• region or department;

• business definition;

• intended audience;

• security classification;

• historical versus current status.

Without this context, AI can produce convincing responses based on information that is technically correct but inappropriate for the current situation.

Representativeness

Does the dataset adequately reflect the population and operational conditions where the AI system will be used?

A dataset may be internally clean while underrepresenting important customer groups, clinical conditions, geographical regions, or unusual events.

Provenance and Lineage

Can the organisation trace the information from its original source through transformations and into the AI system?

Lineage helps teams understand:

• where a defect originated;

• which downstream systems are affected;

• which transformation changed the information;

• who owns the source;

• whether an AI output used the correct version.

Evidence Quality

For Retrieval-Augmented Generation systems, quality also includes the authority, relevance, currency, permissions, and completeness of retrieved evidence.

A perfectly functioning vector search can still retrieve an expired policy, an unauthorised document, or a passage that lacks the context required for a safe answer.

Technical Validation Can Hide Business Failure

Data pipelines usually perform technical validation:

• schema checks;

• type validation;

• null checks;

• format validation;

• range checks;

• duplicate detection.

These controls are necessary, but business validation is equally important.

Business validation asks questions such as:

• Does the transaction balance according to accounting rules?

• Does the identifier belong to the correct person?

• Is this status valid at this stage of the process?

• Is a recorded medication still active?

• Is a policy applicable to this country?

• Does a negative laboratory result supersede an earlier result?

Technical rules determine whether data is structurally acceptable.

Business rules determine whether it makes sense.

AI-ready data requires both.

“Good Enough” Depends on Risk and Purpose

There is no responsible universal threshold for data quality.

A completeness score of 95% may sound impressive, but its meaning depends on what is missing.

If optional product descriptions are missing, the impact may be limited.

If patient identifiers, transaction values, or medication status fields are missing, even a small failure rate may be unacceptable.

Microsoft describes a data-quality threshold as the point that answers:

How good is good enough?

Its guidance also notes that applying one fixed threshold to every field can be misleading, particularly for critical or regulated information. Microsoft Purview threshold guidance

For each important rule, organisations should consider:

• the business or human impact of failure;

• acceptable error tolerance;

• required freshness;

• regulatory obligations;

• evidence required for verification;

• the accountable owner;

• the action triggered when the threshold is missed.

The same data element may require different thresholds when used for exploratory reporting, operational predictions, automated actions, or high-consequence decisions.

Data Quality Must Be Continuous

Data quality is sometimes treated as a cleanup activity performed before a project begins.

In production, quality changes continuously.

• Source systems evolve.

• Schemas change.

• Business definitions are revised.

• New records arrive.

• Documents are replaced.

• Permissions change.

• Populations and behaviours shift.

• Upstream defects propagate downstream.

Data quality should therefore operate as a continuous cycle:

Define
  ↓
Profile
  ↓
Validate
  ↓
Apply thresholds
  ↓
Publish
  ↓
Monitor
  ↓
Remediate
  ↓
Improve

The cycle begins by defining the use case, critical data elements, quality dimensions, thresholds, owners, and failure responses.

Profiling then helps teams understand distributions, missing values, patterns, relationships, and anomalies in the actual data.

Validation applies technical and business rules.

Monitoring identifies changes and emerging defects after publication.

Remediation should address both the immediate issue and its upstream cause.

Quality Gates Turn Failures into Controlled Outcomes

Not every quality defect should have the same response.

Depending on severity and use, a pipeline may:

• record a quality metric;

• generate a warning;

• quarantine affected records;

• prevent a data product from being published;

• stop the pipeline;

• notify an owner;

• disable an affected AI capability.

For example:

• A missing optional description may generate a metric.

• An invalid country code may be quarantined.

• A duplicated payment may require investigation.

• A broken patient identifier may stop publication entirely.

Databricks pipeline expectations illustrate this approach by allowing quality constraints to monitor violations, drop invalid records, or fail a pipeline update. Databricks pipeline-expectations documentation

The important point is not the product used. It is that failure behaviour should be designed deliberately.

A quality rule without a defined response identifies a problem but does not control its impact.

Quality Across Structured and Unstructured Data

Modern AI systems consume many forms of information.

Structured data

Structured datasets require schema validation, referential integrity, business rules, reconciliation, freshness checks, and distribution monitoring.

Linked data

Records from multiple systems require reliable entity resolution.

An AI system must know whether two records describe the same customer, patient, organisation, product, or event.

Documents and unstructured text

Documents require quality controls around:

• parsing;

• reading order;

• headings and sections;

• table interpretation;

• document versions;

• metadata;

• duplication;

• language;

• temporal context;

• access permissions.

RAG and retrieval data

Retrieval pipelines require evaluation of:

• authority;

• currency;

• relevance;

• ranking;

• permission enforcement;

• context preservation;

• evidence coverage;

• citation accuracy.

Quality must follow the complete information path from the source to the final decision, not only individual tables.

Healthcare Shows Why Context Changes Quality

Healthcare provides a clear example of why technically valid data may still be unsafe.

A laboratory value may use the wrong measurement unit.

A medication may be correctly recorded but already discontinued.

A diagnosis may be historical, suspected, or negated.

A clinical abbreviation may have multiple possible meanings.

Two individually valid records may belong to different patients.

Before healthcare information is used by AI, several conditions should be addressed:

• correct patient identity;

• current medication and diagnosis status;

• valid units and reference ranges;

• preservation of negation and uncertainty;

• distinction between historical and current information;

• terminology interpretation;

• privacy and authorised access;

• traceable supporting evidence.

This does not mean AI cannot assist healthcare professionals.

It means that assistance must be supported by strong data engineering, domain context, transparent evidence, and human oversight.

AI should assist people, not blindly replace their judgement and accountability.

Metrics Without Action Are Decoration

Quality scores and dashboards can improve visibility, but visibility alone does not restore trust.

An operational quality system needs:

• meaningful dimensions;

• use-case thresholds;

• severity levels;

• named owners;

• alerts and escalation;

• lineage for investigation;

• remediation workflows;

• communication with affected consumers;

• evidence that the issue has been resolved.

Microsoft Purview, for example, supports quality scores, trend monitoring, and alerts when data falls below configured expectations. Microsoft Purview data-quality scores

Technology can detect and communicate the signal. Organisations must still determine who owns the response and what authority they have to fix the underlying issue.

A metric without a threshold is difficult to interpret.

A threshold without an owner is easy to ignore.

An owner without a remediation process cannot restore trust.

A Practical Data Quality for AI Checklist

Before using a dataset or knowledge source in an AI system, ask:

1. What decision or user outcome will this information support?

2. Which data elements are critical to that outcome?

3. Which quality dimensions matter most?

4. What thresholds are acceptable for this use case?

5. Which technical and business rules are required?

6. Is the information sufficiently current?

7. Has its meaning and context been preserved?

8. Does it represent the intended operational population?

9. Can it be traced to its source?

10. Are access and privacy requirements enforced?

11. What happens when a quality rule fails?

12. Who owns investigation and remediation?

13. How will quality be monitored after deployment?

14. Can affected AI outputs or data products be identified?

15. Is human review available when consequences are significant?

These questions turn data quality from a general ambition into an operational capability.

The Central Takeaway

Data quality for AI is not simply the absence of nulls, duplicates, or formatting problems.

It is:

Fitness for purpose, measured continuously and connected to action.

Reliable AI requires technical checks and business validation. It requires context, lineage, appropriate thresholds, monitoring, ownership, and remediation.

The objective is not to achieve a perfect global score.

The objective is to make explicit, governed decisions about whether information is trustworthy enough for a specific use, and to stop, limit, or clearly qualify its use when it is not.

Because reliable AI does not begin with the model.

It begins with data that people can understand, evaluate, and trust.

Continue the Info2K Data Engineering for AI Series

This article is Part 5 of the Info2K Data Engineering for AI series.

Previously:

• Part 1: The Hidden Data Engineering Behind Successful AI Projects

• Part 2: What Does AI-Ready Data Actually Mean?

• Part 3: ETL vs ELT for Modern AI Systems

• Part 4: Building Reliable RAG Pipelines

Coming next:

• Part 6: Healthcare NLP: Why Clinical Text Is Different

• Part 7: Responsible AI Starts with Responsible Data

• Part 8: Designing Cloud Data Platforms for AI

• Part 9: Modern Data Engineering as the Foundation of Enterprise AI

Follow Info2K as we continue exploring how information becomes reliable, governed, contextual, and usable knowledge.

About Info2K

Info2K : Information to Knowledge

Info2K shares practical insights across Data Engineering, AI Engineering, Cloud Data Platforms, Digital Health, and responsible technology.

Our focus is on connecting technical engineering with context, governance, reliability, and real-world outcomes.

Website: https://info2k.co.nzLinkedIn: https://www.linkedin.com/company/info2k