Responsible AI Starts with Responsible Data

Part 7 of the Info2K Data Engineering for AI Series

Artificial Intelligence is becoming part of everyday organisational systems.

AI can help people search for information, summarise documents, classify content, identify patterns, generate recommendations, automate repetitive work and support decision-making.

The potential is significant.

However, organisations should not evaluate an AI system only by asking:

• Does it work?

• Is it accurate?

• Is it fast?

• Can it reduce costs?

• Can it automate this process?

They must also ask:

• Should it be used for this purpose?

• Who could benefit?

• Who could be harmed?

• Is the underlying data appropriate?

• Can people understand its role and limitations?

• Who remains accountable for the outcome?

• What happens when the system is wrong?

• Can it be safely corrected, overridden or withdrawn?

These are Responsible AI questions.

Responsible AI is sometimes presented as a set of ethical principles added after a system has already been built.

In practice, responsibility must begin much earlier.

It starts with the purpose, the data, the architecture, the people affected and the decisions the system will support.

Connecting Healthcare NLP with Responsible AI

In Part 6: Healthcare NLP: Why Clinical Text Is Different, we explored how clinical meaning depends on negation, uncertainty, time, experiencer, terminology, provenance and supporting evidence.

That discussion illustrates a broader Responsible AI principle:

A technically functioning AI system can still produce an irresponsible outcome when context, risk and human consequences are ignored.

For example, a Healthcare NLP system may correctly identify a clinical concept while incorrectly representing it as:

• current rather than historical;

• confirmed rather than suspected;

• present rather than negated;

• belonging to the patient rather than a family member.

The model may have completed its technical task, but the result may still be unsuitable for the intended decision.

Responsible AI therefore requires more than model performance.

It requires the complete system, including data, context, workflow, governance, monitoring and human oversight, to be designed around its intended use and potential consequences.

Responsible AI Is a System Property

An AI model is only one component of an AI system.

The complete system may include:

Purpose and intended use
          ↓
Data sources and pipelines
          ↓
Model or AI service
          ↓
Rules, retrieval and application logic
          ↓
User interface and workflow
          ↓
People making or receiving decisions
          ↓
Monitoring, feedback and governance

Responsibility cannot be assigned only to the model.

An accurate model can still be used irresponsibly if:

• the input data is inappropriate;

• users misunderstand what the output means;

• sensitive information is exposed;

• the system is applied outside its evaluated purpose;

• automation encourages people to stop checking;

• no one owns failures;

• affected people cannot question or challenge an outcome.

Responsible AI must therefore be evaluated as a socio-technical system: technology operating within an organisation, workflow and human context.

Start with a Clearly Defined Purpose

Before choosing a model, organisations should define what the AI system is intended to achieve.

A useful purpose statement should explain:

• the problem being addressed;

• the intended users;

• the people affected;

• the decision or workflow being supported;

• the expected benefit;

• the system’s boundaries;

• what the system must not do;

• the consequences of failure.

“Using AI to improve productivity” is too broad.

A more useful definition might be:

Assist authorised support staff by retrieving approved policy information and presenting source evidence for verification.

This statement identifies the user, the task, the information boundary and the role of human verification.

A clearly defined purpose makes it possible to determine:

• which data is appropriate;

• which risks matter;

• which performance measures are relevant;

• what level of human oversight is needed;

• how the system should be monitored.

Without a defined purpose, almost any result can appear successful.

Responsible AI Begins with Responsible Data

AI systems learn from, retrieve or act on data.

That data reflects historical decisions, organisational processes, collection methods, missing populations, language, definitions and human assumptions.

Even technically clean data may be unsuitable if it is:

• collected for a different purpose;

• unrepresentative of the intended population;

• missing important context;

• outdated;

• incorrectly labelled;

• poorly governed;

• used without appropriate permission;

• affected by historical inequality;

• unable to support the claims made from it.

Responsible data practices should consider:

Provenance

Where did the data come from, and how was it collected?

Purpose and permission

Is the proposed AI use compatible with the reason the information was collected and the applicable legal, contractual and ethical requirements?

Quality

Is the information accurate, complete, current, consistent and suitable for this use?

Representativeness

Does the data reflect the population and operating conditions where the AI system will be used?

Context

Are definitions, time, source, ownership and limitations preserved?

Lineage

Can data be traced through transformations into the model, retrieval index or final output?

Access

Who is authorised to use the data and derived information?

Retention

How long should source data, prompts, outputs, logs and derived datasets be retained?

Responsible AI cannot be created on top of uncontrolled data foundations.

Fairness Is More Than Removing a Sensitive Field

A common assumption is that fairness can be achieved by deleting fields such as age, ethnicity, gender or disability status.

The issue is more complex.

Other variables may act as proxies. Historical outcomes may reflect unequal access, inconsistent treatment or previous organisational decisions. Missing data may affect some groups more than others.

Fairness should be examined in relation to:

• who receives the benefit;

• who carries the risk;

• who is represented in the data;

• which errors affect different groups;

• how outputs influence real decisions;

• whether people can seek review or correction.

An overall accuracy score may hide meaningful differences.

For example, a system may perform well on average while producing substantially more false negatives for a smaller population.

The appropriate fairness analysis depends on the use case. It may involve:

• subgroup performance;

• error-rate comparisons;

• data-coverage analysis;

• accessibility;

• qualitative review;

• participation from affected communities;

• examination of downstream outcomes.

Fairness is not only a mathematical property of the model. It is also a question of how the system distributes benefits, errors and opportunities.

Transparency Should Be Useful to the Audience

Transparency does not mean publishing every technical detail to everyone.

Different audiences need different information.

End users may need to know:

• that AI is being used;

• what the system is designed to do;

• what information it uses;

• what its limitations are;

• whether the output requires verification;

• how to report a problem.

Technical teams may need:

• training or retrieval-data provenance;

• model and prompt versions;

• evaluation results;

• known limitations;

• security controls;

• monitoring metrics;

• change history.

Governance and assurance teams may need:

• intended-use documentation;

• risk assessments;

• approvals;

• accountable owners;

• audit records;

• incident and remediation history.

People affected by decisions may need:

• a meaningful explanation;

• information about their rights;

• a way to request review;

• a way to correct inaccurate information.

Transparency is valuable when it enables informed use, effective oversight and meaningful challenge.

Explainability Must Match the Decision

Explainability is frequently treated as a universal technical requirement.

The necessary explanation depends on the system and audience.

A data scientist debugging a model needs different information from a customer receiving a decision or a clinician reviewing a recommendation.

Useful explanation may include:

• the factors that most influenced an output;

• the source documents used;

• the relevant evidence;

• the assumptions applied;

• uncertainty or confidence;

• known limitations;

• alternative possibilities;

• why human review is required.

For a RAG system, source evidence may be more useful than a generic explanation of model internals.

For a decision-support system, showing the relevant data and uncertainty may be more important than presenting a simplified feature-importance chart.

An explanation should help someone understand, verify or challenge the outcome. It should not merely create the appearance of transparency.

Human Oversight Must Be Designed

“Human in the loop” is not sufficient by itself.

A person cannot provide meaningful oversight if they:

• do not understand the system;

• cannot inspect the evidence;

• have too little time;

• feel pressured to accept the recommendation;

• cannot override the result;

• do not know how to report an error;

• are not accountable for the final decision.

Effective human oversight requires:

• a clearly defined human role;

• appropriate expertise;

• enough information to review the output;

• visibility of uncertainty and evidence;

• authority to reject or override;

• escalation pathways;

• training;

• monitoring for automation bias;

• clear accountability.

In healthcare, the World Health Organization states that humans should remain in control of healthcare systems and medical decisions. Its guidance also emphasises safety, transparency, accountability, inclusiveness and sustainability. WHO ethics and governance guidance for AI in health

AI should assist people, not quietly transfer responsibility to a system that cannot be held accountable.

Privacy and Security Are Responsible AI Requirements

AI can create new paths through which sensitive information is collected, copied, inferred, exposed or retained.

Privacy risks may arise through:

• training data;

• prompts;

• uploaded documents;

• model outputs;

• vector databases;

• application logs;

• monitoring systems;

• human review;

• third-party services.

Security risks may include:

• unauthorised access;

• prompt injection;

• malicious or poisoned data;

• insecure integrations;

• data leakage;

• model or credential theft;

• manipulation of outputs.

Privacy and security should be considered during architecture design.

Important questions include:

• Is this information necessary?

• Where is it processed?

• Is it sent to another provider?

• Is it used for model training?

• Who can access prompts and outputs?

• What is logged?

• How long is it retained?

• Can sensitive information be reconstructed or inferred?

• What happens after an incident?

Responsible AI requires both data protection and system security throughout the lifecycle.

Accountability Must Be Explicit

When an AI system fails, responsibility should not disappear among teams, vendors and automated components.

Organisations should identify accountable owners for:

• the business purpose;

• the data;

• the model or service;

• the application;

• security and privacy;

• evaluation;

• deployment approval;

• operational monitoring;

• incident response;

• user communication.

Accountability does not mean that one individual performs every task.

It means that responsibilities, decisions and escalation paths are documented and understood.

Useful governance artefacts may include:

• intended-use statements;

• risk assessments;

• data documentation;

• model or system cards;

• evaluation reports;

• approval records;

• change logs;

• incident registers;

• monitoring dashboards;

• retirement plans.

Governance should make responsible action repeatable, not simply create paperwork.

Risk Should Determine the Level of Control

Not every AI application creates the same level of risk.

An internal tool that suggests alternative wording for a low-impact document is different from a system influencing employment, credit, insurance, healthcare or access to public services.

A risk-based approach considers:

• severity of potential harm;

• likelihood of failure;

• number of people affected;

• reversibility;

• sensitivity of the information;

• degree of automation;

• vulnerability of affected people;

• ability to detect and correct errors;

• legal and regulatory obligations.

Higher-risk applications may require:

• stronger evidence before deployment;

• independent review;

• more representative evaluation;

• stricter access controls;

• enhanced logging;

• meaningful human oversight;

• staged release;

• continuous monitoring;

• formal incident response;

• safe fallback or shutdown mechanisms.

The OECD AI Principles state that AI systems should remain robust, secure and safe throughout their lifecycle, with risks continually assessed and managed. They also recommend mechanisms to override, repair or safely decommission systems where needed. OECD robustness, security and safety principle

Responsibility should be proportional to the possible consequences.

Evaluation Must Go Beyond Model Accuracy

Responsible AI evaluation should examine the complete system.

Depending on the use case, this may include:

Technical performance

• accuracy;

• precision and recall;

• robustness;

• calibration;

• latency;

• reliability.

Data and subgroup performance

• representativeness;

• missingness;

• subgroup errors;

• distribution shifts;

• label quality.

Generative AI behaviour

• groundedness;

• unsupported claims;

• citation accuracy;

• instruction following;

• unsafe output;

• information leakage.

Human interaction

• whether users understand the system;

• whether they verify outputs;

• automation bias;

• accessibility;

• usability;

• ability to challenge a result.

Operational performance

• monitoring coverage;

• incident frequency;

• escalation effectiveness;

• upstream and downstream failures;

• cost and sustainability.

Real-world outcomes

• whether the system creates the intended benefit;

• whether harm or inequality increases;

• whether people develop inappropriate confidence;

• whether the workflow changes in unexpected ways.

NIST describes trustworthy AI characteristics that include validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. NIST AI Risk Management Framework FAQs

No single metric represents all of these characteristics.

Responsible AI Is a Continuous Lifecycle

An AI system does not remain responsible merely because it passed an initial review.

After deployment:

• input data changes;

• user behaviour changes;

• organisational processes change;

• external knowledge changes;

• models and vendors release updates;

• new risks emerge;

• people discover unexpected uses;

• performance may differ from testing.

Responsible AI should therefore operate as a lifecycle:

Govern
  ↓
Map the context and risks
  ↓
Measure performance and impact
  ↓
Manage identified risks
  ↓
Deploy with controls
  ↓
Monitor real-world use
  ↓
Respond and improve
  ↓
Retire safely when appropriate

The NIST AI Risk Management Framework organises its core around Govern, Map, Measure and Manage. It treats governance as cross-cutting and risk management as continuous throughout the AI lifecycle. NIST AI RMF Core

Monitoring should cover more than technical uptime.

It may need to track:

• input and output quality;

• drift;

• subgroup performance;

• unsupported answers;

• user overrides;

• complaints;

• incidents;

• privacy or security events;

• unintended uses;

• emerging impacts.

Monitoring is valuable only when thresholds, owners and response processes are defined.

Generative AI Creates Additional Responsibilities

Generative AI can create fluent and plausible output even when the information is unsupported or incorrect.

Additional risks include:

• hallucinated information;

• false or mismatched citations;

• sensitive-data disclosure;

• harmful content;

• prompt injection;

• unapproved actions by AI agents;

• misleading anthropomorphic behaviour;

• uncontrolled use of generated material;

• uncertain ownership and provenance.

Controls may include:

• approved information sources;

• permission-aware retrieval;

• prompt and output filtering;

• groundedness evaluation;

• source evidence;

• restricted tools and actions;

• human approval for consequential actions;

• logging and monitoring;

• clearly communicated limitations;

• safe refusal behaviour.

The NIST Generative AI Profile supplements the broader AI RMF with risks and actions relevant to generative systems. NIST Generative AI Profile

The objective is not to remove every possible risk. It is to identify, prioritise, control and monitor risk in relation to the actual use.

A Practical Responsible AI Checklist

Before deploying an AI system, ask:

Purpose and people

1. What problem is the system intended to solve?

2. Who will use it?

3. Who will be affected by it?

4. What must it not be used for?

5. What happens if it is wrong?

Data

6. Where does the data come from?

7. Is its use permitted and appropriate?

8. Is it representative of the intended context?

9. Are quality, provenance and lineage understood?

10. Are privacy, access and retention controlled?

System

11. Has the complete workflow been evaluated, not only the model?

12. Are limitations and uncertainty visible?

13. Can outputs be traced to evidence?

14. Is the system secure against foreseeable misuse?

15. Can it fail safely?

People and accountability

16. Is human oversight meaningful?

17. Can people override or challenge the output?

18. Are accountable owners named?

19. Are users trained to understand the system?

20. Can affected people request correction or review?

Operations

21. What will be monitored after deployment?

22. What thresholds trigger action?

23. Who investigates incidents?

24. How are changes reviewed and approved?

25. Can the system be suspended or retired safely?

A checklist cannot guarantee responsible outcomes, but it can turn broad principles into concrete engineering and governance questions.

Responsible AI Enables Sustainable Innovation

Responsible AI is sometimes described as a barrier that slows innovation.

Poorly governed AI can create much greater delays.

An untested or misunderstood system can produce:

• harm to individuals;

• loss of trust;

• privacy or security incidents;

• costly rework;

• regulatory exposure;

• operational disruption;

• rejection by users.

Responsible engineering creates the conditions for AI to be adopted with greater confidence.

It helps organisations understand where AI can provide value, which controls are needed and when a proposed use should be redesigned or stopped.

The goal is not innovation at any cost.

It is innovation that remains useful, safe and trustworthy when it moves from demonstration into real life.

The Central Takeaway

Responsible AI does not begin with a principles document produced shortly before deployment.

It begins with:

• a justified purpose;

• appropriate and governed data;

• an understanding of affected people;

• risk-based engineering;

• meaningful transparency;

• effective human oversight;

• explicit accountability;

• continuous evaluation and monitoring.

Responsibility is not a feature of the model alone.

It is a property of the complete system and the organisation operating it.

AI should help people make better use of information, not hide uncertainty, weaken accountability or automate harm.

Continue the Info2K Data Engineering for AI Series

This article is Part 7 of the Info2K Data Engineering for AI series.

Previously:

• Part 1: The Hidden Data Engineering Behind Successful AI Projects

• Part 2: What Does AI-Ready Data Actually Mean?

• Part 3: ETL vs ELT for Modern AI Systems

• Part 4: Building Reliable RAG Pipelines

• Part 5: Data Quality for AI: Why Clean Data Is Not Enough

• Part 6: Healthcare NLP: Why Clinical Text Is Different

Coming next:

• Part 8: Designing Cloud Data Platforms for AI

• Part 9: Modern Data Engineering as the Foundation of Enterprise AI

Follow Info2K as we continue exploring how information becomes reliable, governed, contextual and usable knowledge.

About Info2K

Info2K : Information to Knowledge

Info2K shares practical insights across Data Engineering, AI Engineering, Responsible AI, Healthcare NLP, Cloud Data Platforms and Digital Health.

Our focus is on connecting technology, data, governance, domain knowledge and human outcomes.

If your organisation needs support exploring an AI opportunity, evaluating risk, reviewing an architecture, improving data foundations or developing a responsible AI solution, please get in touch.

Website: https://info2k.co.nz

LinkedIn: https://www.linkedin.com/company/info2k