Imagine that an AI research system has spent the night evaluating thousands of possible experiments.
By morning, it has narrowed them to one.
The proposed experiment has the highest expected scientific payoff. The simulation suggests a strong chance of success. The AI has already generated the protocol, checked the equipment schedule, estimated the cost, and prepared the analysis code.
There is only one problem.
The experiment carries a small but serious risk.
Should it proceed?
At this point, more computation does not necessarily solve the problem.
The AI can refine the probability.
It can model possible outcomes.
It can search the literature.
It can calculate expected costs and benefits.
But someone still has to answer a different kind of question:
What ought we to do?
That question marks an important boundary in AI for Science.
Prefer to watch? Continue with the video below, or keep reading.
Knowing What Will Happen Is Not the Same as Knowing What Should Happen
Science is exceptionally good at questions of the form:
What is true?
What is likely to happen?
What causes this effect?
Which intervention will produce this outcome?
Ethics asks a different class of questions:
Which outcome should we prefer?
How much risk is acceptable?
Who may bear that risk?
What obligations do we have to people who did not choose to participate?
Are there things we should not do even if they are technically possible?
No amount of predictive accuracy automatically answers those questions.
Suppose an AI can estimate two research strategies perfectly.
Strategy A produces scientific progress more quickly but exposes a small group to greater risk.
Strategy B is slower but distributes risk differently.
Which should we choose?
The probabilities may be facts.
The choice among them depends on values.
AI Can Reason About Values Without Owning Them
This does not mean AI is useless in ethical reasoning.
Quite the opposite.
An AI system can help identify stakeholders, reveal hidden tradeoffs, test a decision against different ethical principles, search relevant policies, and point out inconsistencies in human reasoning.
It may even reason more carefully than a tired person in some situations.
But there is a crucial distinction between reasoning about values and being the source of those values.
An AI system learns from human-created data and from objectives, rules, feedback, and constraints provided during its development and deployment.
If it says that one value should outweigh another, we should ask:
Where did that ordering come from?
Who chose it?
Whose values were represented?
What happens when legitimate values conflict?
There is no neutral optimization function for humanity.
Every objective leaves something out.
Scientific Judgment Has More Than One Layer
As AI becomes more capable, it helps to separate several kinds of judgment that are often mixed together.
Epistemic judgment
This is judgment about knowledge.
Is the evidence strong enough?
Is the result reproducible?
Is the model extrapolating beyond its training?
Does an anomaly indicate a discovery or an instrument failure?
AI may become extremely good at parts of this.
But humans still need ways to challenge the model, especially when the evidence is incomplete or the stakes are high.
Practical judgment
This is judgment about action.
Should we spend another month on this experiment?
Should we deploy this method outside the laboratory?
Is the expected gain worth the cost?
Should the system proceed automatically, or should it stop for review?
These questions mix evidence with priorities.
Moral judgment
This is the deepest layer.
Even if an action is technically effective and economically efficient, is it acceptable?
Does it respect human dignity and rights?
Does it impose serious risk on people who did not consent?
Could the knowledge be used in ways that create disproportionate harm?
What do we owe future generations?
These are not merely technical questions.
They concern the purposes for which science exists.
Human Judgment Is Not the Same as Human Intuition
Saying that humans must remain responsible does not mean that human instinct is always wise.
Humans are biased.
We overlook evidence.
We protect our own interests.
We make emotional decisions.
Groups can reinforce one another's errors.
History gives us many examples of scientific and technological decisions that were made by humans and were still deeply wrong.
So “keep a human in the loop” is not, by itself, a safety solution.
A distracted person clicking Approve on every recommendation is not meaningful oversight. Human-factors research calls one version of this automation bias: people may over-trust an automated recommendation even when contrary evidence is available.
There is also a danger in assigning responsibility without real control. Researcher Madeleine Clare Elish introduced the phrase moral crumple zone for situations in which a human absorbs blame for the failure of a highly automated system even though that person had limited practical control over what the system did.
Meaningful human oversight therefore requires authority, time, information, and the ability to refuse—not simply a person positioned near the end of the workflow.
Human judgment has value when it is supported by reason:
- relevant evidence is available,
- alternatives are considered,
- uncertainties are stated,
- affected people are taken seriously,
- conflicts of interest are visible,
- and the decision can be questioned.
The goal is not human control for its own sake.
It is accountable judgment.
The Human Role May Move Upward
As AI becomes better at routine scientific work, human involvement may shift.
A scientist may no longer need to inspect every line of analysis or manually select every experimental parameter.
That would defeat much of the purpose of automation.
Instead, human responsibility may move upward in the hierarchy.
AI may decide:
Which of these 10,000 parameter settings should we test next?
A human may decide:
Is this class of experiment acceptable at all?
AI may decide:
Which candidate molecule best satisfies the stated objective?
A human may decide:
Should this objective be pursued, and under what constraints?
AI may decide:
What action maximizes the specified reward?
Humans must decide:
Is that the reward we actually want to maximize?
This division is not fixed.
As systems become more reliable, some decisions can be delegated.
But high-consequence decisions require a different standard from routine ones.
The Goal Is Part of the Experiment
We often picture scientific reasoning as beginning after the question has already been chosen.
But choosing the question is itself a decision.
Why study one disease rather than another?
Why optimize a technology for speed rather than accessibility?
Why build a system that maximizes output rather than resilience?
Why pursue a scientific capability at all?
AI can help compare options, but the objective function does not descend from nature.
Someone chooses it.
This means that human values enter the AI4S loop before the first experiment is run:
\[ \boxed{\text{Values and Goals}} \rightarrow \text{Question} \rightarrow \text{Hypothesis} \rightarrow \text{Experiment} \rightarrow \text{Evidence} \rightarrow \boxed{\text{Judgment and Decision}}. \]
The scientific loop is therefore not value-free.
It begins and ends with choices.
What Does “Human in the Loop” Really Mean?
The phrase is used so often that it can become almost meaningless.
A useful human role should include actual authority.
For high-stakes scientific systems, that may mean humans can:
- define what goals are permitted,
- establish limits before an experiment begins,
- require review when a system crosses a risk threshold,
- inspect the evidence behind a recommendation,
- stop an automated process,
- reject an apparently optimal solution,
- authorize deployment beyond the laboratory,
- and investigate failures afterward.
The important point is not that every scientific action requires a committee.
It is that responsibility should not disappear simply because the machine performed the intermediate reasoning.
The AI Recommended It Is Not an Answer
Imagine that an automated scientific system makes a recommendation that causes serious harm.
Who is responsible?
The model?
The researcher who accepted the recommendation?
The laboratory director?
The institution that deployed the system?
The company that built it?
In practice, responsibility may be distributed among several people and institutions.
But it should not be dissolved among them.
“The AI decided” cannot become a convenient way for everyone else to step away.
An AI system may be part of the causal chain.
But the people and institutions that design, authorize, deploy, and supervise high-consequence systems must remain accountable for the conditions under which those systems act.
This requires more than moral intention.
It requires structure.
Responsibility Needs a Record
If an important scientific decision is partly automated, we should be able to reconstruct what happened.
What information did the system receive?
What did it recommend?
How uncertain was it?
What alternatives were considered?
Who authorized the next step?
Were any safeguards overridden?
What happened afterward?
This suggests an important principle for autonomous science:
The faster the loop becomes, the more important traceability becomes.
Without records, automation can make responsibility harder to locate precisely when the consequences become larger.
Auditability is therefore not merely administrative paperwork.
It is part of scientific integrity.
Some Decisions Should Be Reversible
There is another useful principle.
When uncertainty is high, prefer actions that can be reversed.
An AI system may be allowed to explore a simulation freely because the consequences remain inside a sandbox.
It may need more oversight before controlling an expensive instrument.
A physical experiment with irreversible environmental consequences deserves a much higher threshold.
The logic is simple:
\[ \text{greater irreversibility} \rightarrow \text{greater need for deliberation}. \]
This is one reason automated experimentation should not be treated as one uniform category.
The appropriate degree of autonomy depends on what can happen if the system is wrong.
Moral Values Are Not a Bug in Scientific Reasoning
Scientists sometimes prefer to keep values separate from technical analysis.
There is a good reason for this.
We do not want experimental results to change because someone wishes they were different.
Evidence should constrain belief.
But deciding what to do with evidence is different from deciding what the evidence says.
A measurement should not be altered to fit our values.
A decision inevitably reflects them.
This distinction matters enormously for AI.
We should want AI systems to be as objective as possible when estimating facts.
But there is no such thing as a completely value-free choice of goals, risks, priorities, or acceptable consequences.
Pretending otherwise does not remove values.
It merely hides them.
Reason Matters Because Values Can Conflict
Moral principles do not always point in the same direction.
Accelerating a medical discovery may benefit future patients.
Proceeding too quickly may expose current participants to unacceptable risk.
Open publication may advance science.
Restriction may reduce misuse.
Autonomy may increase productivity.
Human review may increase safety but slow the loop.
These are genuine conflicts.
There is rarely a single equation that settles them.
This is where reasoned judgment becomes essential.
Reason does not eliminate moral disagreement.
It helps us make the disagreement explicit, examine evidence, identify consequences, and explain why a decision was made.
That explanation matters because other people may have to live with the result.
Humans Must Retain the Right to Say No
One of the most important human roles in an automated scientific system may be remarkably simple:
the right to stop it.
A machine can be designed to optimize a goal.
Human judgment must retain the authority to say that the goal, the method, or the risk is unacceptable.
That remains true even if the AI is more knowledgeable than the human decision-maker in the relevant technical field.
Expertise and authority are not the same thing.
A navigation system may know the fastest route better than the driver.
The driver still decides whether the destination is worth reaching.
In scientific AI, the consequences can be much larger.
Ultimate Responsibility Comes with Ultimate Authority
If humans are expected to take responsibility for AI-assisted science, they must also retain meaningful authority over it.
Responsibility without control is hollow.
Control without responsibility is dangerous.
The two should remain connected.
This means humans and human institutions must ultimately own several decisions:
\[ \text{What are we trying to achieve?} \]
\[ \text{What are we unwilling to do to achieve it?} \]
\[ \text{How much uncertainty and risk will we accept?} \]
\[ \text{Who may be affected?} \]
\[ \text{When should the system stop?} \]
\[ \text{Who answers for the consequences?} \]
AI can make these decisions better informed.
It should not make responsibility disappear.
The Human Role Does Not Shrink to Zero
This gives us a different picture of the future scientist.
AI may increasingly perform calculation, search, coding, optimization, simulation, and even parts of experimental design.
Humans may therefore do less of some activities that once defined scientific labor.
But other roles become more visible:
choosing goals, judging significance, interpreting meaning, applying moral values, making consequential decisions, and accepting responsibility.
These are not leftovers from tasks AI failed to automate.
They are part of what makes scientific knowledge matter in a human world.
A scientific system can tell us what is possible.
It can help us estimate what is likely.
It may eventually discover things that no person could have discovered alone.
But possibility is not permission.
Prediction is not purpose.
Optimization is not morality.
And intelligence is not responsibility.
As AI moves closer to direct interaction with the physical world, this distinction will become more important, not less.
The next part of this series turns to an especially demanding case: quantum science.
There, both observation and computation behave differently from the classical world.
But the larger question remains the same.
As our machines become more capable of understanding and acting upon nature, humans must remain responsible for deciding what that capability is for.
Further Reading
- International AI Safety Report, International AI Safety Report 2026
- NIST, AI Risk Management Framework
- OpenAI, Our Updated Preparedness Framework
0 comments