Imagine an AI system that becomes extraordinarily good at molecular design.
A medical researcher asks it to search for a protein that could block a dangerous virus. The AI reads thousands of papers, proposes new molecular structures, predicts which ones are most promising, and helps plan the experiments needed to test them.
That sounds exactly like the future we hope AI for Science can deliver.
Now change only the goal.
Instead of asking how to stop the virus, someone asks how to make it harder to stop.
The underlying scientific capability has not changed.
The intention has.
This is the dual-use problem: the same knowledge, model, laboratory technique, or automated system can support both beneficial and harmful purposes.
Science has always faced this problem. AI makes it more urgent because it can accelerate not only discovery, but access to discovery.
Prefer to watch? Continue with the video below, or keep reading.
Scientific Power Has Never Been Morally One-Directional
Many technologies we depend on have a dual history.
Chemistry gives us medicines, fertilizers, batteries, and new materials. It can also produce toxins and explosives.
Nuclear physics gave us medical imaging, radiation therapy, and nuclear power. It also made nuclear weapons possible.
Biotechnology can create vaccines and lifesaving therapies. The same knowledge can sometimes be redirected toward harmful biological applications.
The scientific knowledge itself does not arrive labeled “good” or “bad.”
Its consequences depend on goals, context, access, safeguards, and decisions.
AI enters this old problem with a new feature:
speed and scale.
A capable scientific AI can search a much larger space than a human researcher. It can combine information from many fields. It can write code, use specialized models, analyze data, and increasingly connect these steps into an agentic workflow.
That is exactly why it can accelerate beneficial research.
It is also why misuse matters.
The dual-use problem became unusually concrete in a 2022 study by researchers at Collaborations Pharmaceuticals. A generative drug-discovery system that had been used to seek useful, low-toxicity molecules was reoriented computationally toward predicted toxicity. In less than six hours, it generated about 40,000 candidate molecules meeting the chosen toxicity criterion, including the known nerve agent VX and other known chemical-warfare agents. Many additional candidates were predicted to be highly toxic. The study did not synthesize these compounds, and moving from a computational candidate to a usable chemical agent remains a substantial barrier. Its warning was simpler: the same model capability can serve very different objectives depending on the goal it is given.
AI Can Lower the Barrier to Expertise
Suppose a difficult scientific task requires three things:
- knowing what information matters,
- knowing how to connect that information,
- knowing what to do when the first attempt fails.
Traditionally, those abilities may take years of training.
A sufficiently capable AI does not eliminate the need for physical materials, instruments, laboratories, and practical skill. Those barriers remain important.
But AI can make specialized knowledge easier to navigate.
It can explain a technical procedure.
It can translate jargon.
It can suggest why an experiment failed.
It can connect a user to specialized scientific tools.
It can help plan a sequence of actions rather than answering one isolated question.
That is why recent AI safety assessments pay particular attention to biological and chemical capabilities. The 2026 International AI Safety Report, for example, describes biological and chemical applications as a clear dual-use area: the same capabilities that support medicine and scientific research may also lower some barriers for malicious actors. The report also emphasizes that the magnitude of real-world risk remains uncertain because practical barriers still matter.
That uncertainty is important.
We should neither assume catastrophe nor assume safety.
The Risk Changes When AI Can Act
A language model that only produces text has one kind of risk.
An AI agent connected to tools has another.
Consider the difference:
\[ \text{question} \rightarrow \text{answer} \]
versus
\[ \text{goal} \rightarrow \text{plan} \rightarrow \text{tool use} \rightarrow \text{observation} \rightarrow \text{revised plan} \rightarrow \cdots \]
The second system can pursue an objective over time.
In AI for Science, that may mean searching databases, writing code, running simulations, controlling instruments, or interacting with automated laboratories.
This is exactly what makes scientific agents useful.
It also means that an error or malicious goal can propagate farther before a human notices.
The problem is no longer only:
Did the AI say something dangerous?
It may become:
What actions was the AI allowed to take?
Misuse Is Only One Kind of Risk
It is tempting to make AI safety a story about bad actors.
That is too narrow.
There are at least three broad ways advanced scientific AI could cause harm.
Deliberate misuse
Someone intentionally uses the system to pursue a harmful goal.
This is the classic dual-use concern.
A capability developed for beneficial science may be redirected toward biological, chemical, cyber, or other harmful applications.
Unintended failure
The goal may be legitimate, but the AI may be wrong.
Imagine an autonomous laboratory system that confidently misinterprets an anomalous measurement.
It modifies the experiment.
The next result reinforces the error.
After many automated cycles, the system has built an elaborate conclusion on a faulty assumption.
Automation can make research faster.
It can also make mistakes faster.
Systemic risk
A third danger does not require either malice or a dramatic technical failure.
Suppose scientific institutions become heavily dependent on a small number of AI systems.
Researchers begin trusting the same models.
Funding decisions increasingly follow AI-generated assessments.
Experimental priorities converge.
A hidden bias or blind spot could then influence an entire field.
The risk comes from scale and dependence.
A system can be individually useful yet collectively distort the scientific ecosystem.
The Closed Loop Raises the Stakes
Earlier in this series, we described the scientific loop:
\[ \text{hypothesis} \rightarrow \text{experiment} \rightarrow \text{observation} \rightarrow \text{revision}. \]
Closing that loop is one of the great ambitions of AI4S.
But the more complete the loop becomes, the more consequential the safeguards become.
If AI only suggests hypotheses, a scientist can ignore them.
If AI proposes an experiment, orders materials, controls instruments, interprets the result, and initiates the next experiment, there are fewer natural stopping points.
Autonomy is therefore not simply a measure of convenience.
It changes the structure of risk.
The question becomes:
At which points should a machine be allowed to proceed on its own, and at which points must someone stop, inspect, and authorize what comes next?
There is unlikely to be one answer for every field.
An automated telescope adjusting its observation schedule is not the same as an automated biological laboratory manipulating a hazardous organism.
The level of oversight should depend on the possible consequences.
The Same Capability Can Be Both Valuable and Dangerous
This is what makes scientific AI safety unusually difficult.
The most obvious response to a dangerous capability is to restrict it.
But scientific capabilities often do not divide neatly into harmless and harmful categories.
A model that understands virology deeply enough to help design better vaccines may also understand virology deeply enough to answer dangerous questions.
A model that can design proteins for medicine may have knowledge relevant to harmful biological design.
A powerful cybersecurity model can defend a laboratory network and also identify vulnerabilities.
This creates a genuine tension:
\[ \text{more capability} \rightarrow \begin{cases} \text{more scientific benefit},\\ \text{more potential for misuse}. \end{cases} \]
Good safety design cannot simply erase the underlying capability.
It has to manage who can do what, under which conditions, with what oversight.
Safety Is Not One Filter
There is another common misconception: if AI becomes dangerous, add a safety filter.
That is unlikely to be enough for high-consequence scientific systems.
A more realistic approach is defense in depth.
Different layers can address different failures:
- evaluating models before deployment,
- limiting access to especially sensitive capabilities,
- monitoring for suspicious patterns of use,
- restricting which external tools an agent can call,
- requiring human authorization before high-consequence actions,
- maintaining physical laboratory safety controls,
- protecting model weights and sensitive data,
- keeping auditable records,
- allowing independent review,
- and designing systems that can be stopped reliably.
No single layer is perfect.
The point is that failure of one safeguard should not automatically produce failure of the whole system.
This is particularly important when AI crosses from information into action.
Scientific Openness Also Has a Cost
Science has traditionally benefited from openness.
Researchers publish methods so others can reproduce them.
Data are shared.
Software is released.
Knowledge accumulates because discoveries are not confined to one laboratory.
AI creates a difficult question for this culture.
If a scientific model has a capability that is extremely useful but also unusually easy to misuse, should everything about it be released without restriction?
There is no universal answer.
Too much restriction can slow legitimate science, concentrate power, and prevent independent verification.
Too little restriction can distribute dangerous capabilities before society has adequate safeguards.
The right balance may differ by capability, maturity, and potential consequence.
That is not a reason to abandon openness.
It is a reason to recognize that openness itself can become an ethical decision when scientific capabilities become powerful enough.
The Danger Is Not Only What AI Knows
The most serious transformation may come when AI combines four things:
\[ \text{knowledge} + \text{reasoning} + \text{tools} + \text{autonomy}. \]
Knowledge tells the system what is known.
Reasoning helps it construct a plan.
Tools let it affect external systems.
Autonomy lets it continue without constant human approval.
Each component is useful.
Together they create something qualitatively different.
For AI4S, this combination may eventually accelerate discovery beyond anything possible in a conventional laboratory.
It also means safety cannot be added at the end.
The structure of the scientific loop itself must include safeguards.
A More Powerful Scientist Needs Stronger Boundaries
We often speak of AI as though intelligence were an unqualified good.
In science, more intelligence can certainly produce enormous benefit.
But intelligence is a capability, not a moral direction.
A more capable system can pursue a good objective more effectively.
It can also pursue a bad objective more effectively.
And if its objective is poorly specified, it can produce harmful consequences without anyone intending them.
That is why progress in AI for Science must include a second question alongside every increase in capability:
What should this system be allowed to do?
The answer cannot come from the model alone.
A model can estimate consequences.
It can compare alternatives.
It can reason about rules we give it.
But deciding which goals are acceptable, which risks are justified, whose interests matter, and who has authority to proceed is not merely another prediction problem.
It is a problem of values and responsibility.
That leads to the next question in this series:
If AI can increasingly propose, evaluate, and even execute scientific decisions, who should make the final judgment—and who is responsible for the consequences?
Further Reading
- International AI Safety Report, International AI Safety Report 2026
- OpenAI, Our Updated Preparedness Framework
- Anthropic, Why Do We Take LLMs Seriously as a Potential Source of Biorisk?
0 comments