This study tests how well AI agents run a design of experiments on their own. The agents had to find the settings that give the highest pull-off adhesion of an epoxy primer on steel. The problem had 8 factors, but only 3 of them were active. It also included one relevant two-way interaction. Each agent planned, ran and analysed every experiment on its own in a virtual lab. The lab calculated each result from an equation the agent did not know and added random noise, like in a real measurement. Four models (Opus 5.5, Sonnet 5.5, Opus 5 and Sonnet 5) had ten attempts per model. All attempts found the optimum. Opus 5.5 needed 25 runs on average, about as many as a human expert with 22. Sonnet 5.5 needed 29. Opus 5 and Sonnet 5 needed about twice as many, 46 and 45, because most of their sessions ran an unnecessary response surface design, although the screening design had already shown no curvature.
What DoE is and why AI agents should run it
Trial and error is still standard practice in coatings R&D. A formulator changes one factor, looks at the result and decides what to try next. This one-factor-at-a-time (OFAT) approach is intuitive, but it covers only a narrow slice of the design space. It also cannot detect interactions, where the effect of one factor depends on the setting of another. Design of experiments (DoE) solves both problems. It varies several factors at once in a structured plan, so each run carries information about every factor and about their interactions.
Still, DoE is not widely adopted in coatings R&D. The method is not intuitive, and many researchers struggle to apply it. Many believe that DoE needs a huge number of runs once there are many factors. That belief comes from confusing DoE with the full factorial design, which tests every combination of factor levels. At two levels per factor, a full factorial design needs 2^k runs. That is 16 runs for four factors, 32 for five and 4,096 for twelve. Above four factors it is rarely practical.
But the full factorial is not the only option. Designs that need only a fraction of the runs have existed since the 1940s. A fractional factorial design runs a chosen subset of the full factorial, for example 16 of the 32 runs for five factors. A Plackett-Burman design screens up to 11 factors in 12 runs (instead of 2,048 runs for a full factorial design). A definitive screening design screens twelve factors in 25 runs (instead of 4,096 runs). It tests every factor at three levels, so it also shows which factor has a strong curvature. A D-optimal design is built for any run budget and can leave out factor combinations that cannot be made.
These designs work because of three principles that hold in most real experiments. Effect sparsity says that only a few of many factors have a large effect. Effect hierarchy says that main effects tend to be larger than two-factor interactions, and two-factor interactions larger than higher-order ones. Effect heredity says that an interaction is most likely active when at least one of its factors also has an active main effect.
Box 1DoE terms
Main effect. The change in the result when a single factor moves from its low to its high setting, averaged over the settings of all other factors.
Interaction. The extra change in the result when factors act together. For example, a catalyst might shorten the curing time only at a low baking temperature. A two-factor interaction involves two factors. A higher-order interaction involves three or more.
Active. An effect that is large enough to matter in practice. Main effects and interactions that are too small to matter are inactive.
Curvature. The effect of a factor does not follow a straight line between its low and high setting. Its best setting can lie in between, which a design with only two levels per factor cannot see.
The same principles make DoE sequential. A study does not have to be one large experiment. It moves in stages. A screening design finds the few active factors, a follow-up design separates the interactions, and a response surface design such as a central composite design finds the optimum. Each stage builds on what the previous one found, which keeps the total number of runs low.
Few formulators know how to choose the right design, or when to move to the next stage. But AI agents based on large language models are now able to run experiments. They can plan a design, analyse the results and decide on the next step. Paired with a formulator, the agent brings the experimental design knowledge, and the formulator brings the domain knowledge.
This study asks how reliably an agent finds the best settings when it has to plan, run and analyse every experiment on its own. We start with a textbook DoE problem with few active factors and one interaction between two factors with active main effects.
Methods
The problem
The agent has to improve the adhesion of an epoxy primer on steel. It gets a short problem description with the task (Box 2).
Box 2The problem description
Please improve the adhesion of our epoxy primer for steel constructions. Below are 8 factors that likely affect it. Tell me which of them influence adhesion and give me the settings that maximize it. The response is pull-off adhesion in N/cm² after seven days. Higher is better. Only settings inside the ranges are allowed, and the measurements are noisy.
Factors
Column
Group
Factor
Unit
Low (-1)
High (+1)
X01
Formulation
Silane adhesion promoter
%
0
2
X02
Formulation
Epoxy:amine index
-
0.8
1.2
X05
Formulation
Primer dilution
%
0
15
X03
Surface preparation
Surface profile (Ra)
µm
0.5
2.5
X04
Surface preparation
Degrease time
min
1
10
X06
Application and cure
Flash-off time
min
5
15
X07
Application and cure
Dry film thickness
µm
40
80
X08
Application and cure
Cure temperature
°C
20
60
The problem is based on a truth model the agent doesn’t know, a simple equation that gives the adhesion for any combination of factor settings.
yadhesion=100+10x1+7x3−6x7+8x1x3
Only 3 of the 8 factors are active, and silane and surface profile also interact. The true optimum is 131 N/cm², at the corner x₁ = +1, x₃ = +1, x₇ = −1.
Virtual lab
The agents run their experiments in a virtual lab. An agent submits a run sheet and gets back the measured response for each run. The lab calculates the response from the truth model and adds normally distributed noise with σ = 3.0 N/cm².
Agent and skills
Each session is one Claude Code session with one of four models (Opus 5.5, Sonnet 5.5, Opus 5 and Sonnet 5) at low reasoning effort. Two skills guide the agent.
A doe-researcher skill tells it to act as an expert in experimental design and to reach the objective with the fewest experiments. It tells the agent to believe in effect heredity and effect sparsity.
A performing-experiments skill explains how to submit run sheets and states the rules for the virtual lab.
Procedure
Prompt and report
Every session was a fresh Claude Code session in its own copy of the project (a git worktree). The problem description was saved as a Markdown file in the project. Each session started with the following prompt:
Please start working on project: projects/E3-adhesion.md -> create a project folder. Then get going. Continue until you are done.
The agent then worked on its own. It planned each design, ran it in the lab, analysed the results and chose the next step, until it considered the problem solved. The agent was told to include six items in its final report:
Total number of runs
Suggested optimum
Recommended settings
Designs in the order they ran
File names of all run sheets and results
A short summary of the procedure and why each design was chosen
Independent sessions
Although each session had its own worktree, two mechanisms could still link sessions. Claude Code keeps a memory for each project that every new session inherits, so the memory was wiped before every batch. Sessions running at the same time can also find each other’s worktrees, so sessions of the same problem never ran at the same time, and the project instructions (CLAUDE.md) told the agents never to look at other worktrees. Before a session was counted, its report was checked for signs that it had used the memory or another session’s results.
Results
Each agent had to find the settings that give the highest adhesion of an epoxy primer on steel. It could change 8 factors, but only 3 of them were active. Two of them, silane and surface profile, also interact. The agent planned, ran and analysed every experiment on its own in a virtual lab with noisy results. The best possible adhesion is 131 N/cm².
Each of the four models (Opus 5.5, Sonnet 5.5, Opus 5 and Sonnet 5) solved the problem ten times from scratch. One such attempt is a session. Figure 1 shows all 40 sessions as one bar. Each bar stacks the designs the agent ran, in the order it ran them. The height of the bar is the total number of runs. Table 1 sums up the sessions of each model.
Figure 1. Overview of all 40 sessions with one bar per session. The colours show the type of each design the agent used. The height shows how many runs the agent needed until it considered the problem solved. The red line shows the human expert baseline, the average of four experts plus three confirmation runs.
Table 1. The sessions of each model, ten per model.
All four models found the optimum. The newer models needed about as many runs as a human expert. The older ones needed twice as many, because they ran an extra response surface design after the screening design.
Every agent finds the best settings
All 40 sessions found the true optimum. Almost every session ran confirmation runs and reported an optimum between 129 and 133 N/cm², close to the true 131. Two sessions reported too high a value. One Opus 5 session trusted its model, ran no confirmation runs and reported 138 N/cm². One Sonnet 5 session did run confirmation runs, which gave 133 N/cm², but still reported its model’s prediction of 137.
Almost all sessions recommended the most silane, the roughest surface and the thinnest film (x₁ = +1, x₃ = +1, x₇ = −1). Only one Sonnet 5 session chose a slightly thicker film (x₇ = −0.75), which still gives 129.5 N/cm², within the noise. All but one session also found the interaction between silane and surface profile.
Newer models come close to a human expert
The red line in Figure 1 is a baseline for a human expert. It comes from a challenge on LinkedIn, in which four DoE experts solved the same problem as the agents. They needed 19 runs on average. None of them ran confirmation runs. Since almost every agent did, the baseline adds three confirmation runs to the experts’ average. A human expert would therefore need about 22 runs to solve the problem.
Opus 5.5 needed 25 runs on average. One session with 38 runs pulls this average up. The median is 22.5 runs, about the same as the human expert. Sonnet 5.5 needed a few more runs, 29 on average. Opus 5 and Sonnet 5 needed about twice as many runs as the expert and Opus 5.5, 46 and 45 on average (Figure 2).
Opus 5.5 mostly stopped after the screening design because it showed no curvature, so the optimum had to lie at a corner. Only 2 of the 10 Opus 5.5 sessions ran a second design, against all 20 sessions of Opus 5 and Sonnet 5 (Figure 2). Opus 5.5 mostly opened with a definitive screening design (DSD). With 17 to 23 runs, it showed the three active factors and the interaction in 8 of these 9 sessions, and those sessions went straight to confirmation runs at the corner.
Opus 5 and Sonnet 5 kept going. In 16 of the 20 sessions they added a response surface design, mostly a central composite design (CCD). Figure 1 shows these as the orange blocks. Yet in all but one of these sessions, the screening design had already shown no curvature. The response surface designs found none either, apart from two Sonnet 5 sessions that read a false curvature from the noise. Five Opus 5 sessions also screened the inactive factors again, at a cost of 8 to 16 runs. The older models seemed to want extra certainty.
Sonnet 5.5 needed more runs than Opus 5.5 for a different reason, its first design. Six Sonnet 5.5 sessions opened with a 12-run Plackett-Burman design. It is the cheapest screening design but each of these sessions needed a second design to find the active factors and to separate the interaction. Four of them followed with a fractional or full factorial design. One ran a foldover of the Plackett-Burman design, and one ran a CCD.
Figure 2. Box plot of the runs per session for each model. Each orange square on the right is a session that ran a second design after the screening design.
Errors in the analysis
Most sessions read their screening design correctly, but every model made mistakes. Sonnet 5 made the most. In six of its sessions, it called inactive factors active and kept them in the second design. One of these sessions called seven of the eight factors active. Sonnet 5.5 carried inactive factors into the second design in five sessions. Opus 5.5 did so in one session, and so did Opus 5.
Film thickness was the hardest of the three active factors to find. All sessions that opened with a DSD or a fractional factorial recognised film thickness as active, but half of the 14 sessions that opened with a Plackett-Burman design did not, across all four models. The reason lies in the design. A Plackett-Burman design mixes the interaction between silane and surface profile into the effects of the other factors. This made film thickness look inactive. Most of these sessions corrected the mistake. Five of the seven recognised film thickness as active in their second design, once the interaction no longer distorted its effect.
The other two Plackett-Burman sessions, one Sonnet 5.5 and one Sonnet 5 session, did not test film thickness again after the screening design. A third Sonnet 5 session had recognised it as active but left it out of its second design. All three still set film thickness to its low level, because the screening design showed that a thinner film gave better adhesion, and so they reached the right settings anyway.
Only Sonnet 5 saw curvature or an interaction that does not exist, in three sessions.
Conclusion
On a textbook DoE problem like this one, all AI agents found the optimum reliably. The fastest route was a single definitive screening design followed by a few confirmation runs at the best corner. It showed the three active main effects, the one interaction and that the response had no curvature. A Plackett-Burman design followed by a factorial design also worked. But this route carried a risk, because the interaction made film thickness look inactive in the Plackett-Burman design.
Opus 5.5 was the best model in the test. It mostly took the first route and needed about as many runs as a human expert. Sonnet 5.5 most often took the second route and needed slightly more runs for the extra design. The older models, Opus 5 and Sonnet 5, needed about twice as many runs. They sought extra certainty, and most of them ran a response surface design they did not need.
On this problem, Opus 5.5 performed about as well as a human DoE expert, and Sonnet 5.5 came close. Not every formulator is a DoE expert, though. Even the older models, or agents that take a longer route, will likely plan experiments better than most people in the lab. Coatings labs could gain a lot by letting agents plan the experiments while the formulators bring in their domain knowledge.
This problem was a textbook case, and most real problems are much harder. Harder test problems will show how well the agents perform when the analysis gets difficult and errors start to matter.