More thinking pays off for the 5.5 models.

Human expert
Opus 5.5
low xhigh
Sonnet 5.5
Sonnet 5
Opus 5
20406080
Average runs per session

Summary

This is the second problem of our study on how well AI agents run a DoE on their own. In the first problem, the agents had to improve the adhesion of an epoxy primer on steel. It was a textbook case with three active main effects and one two way interaction. This second problem is slightly more difficult. The agents had to find the settings that give the hardest film of a 2K PU clearcoat. It has 12 factors instead of 8, and again only 3 of them are active. But the response is curved, so the best settings lie inside the factor ranges instead of at a corner. Plotted against the catalyst level, hardness forms an upside-down parabola, an arch with its top at 200 ppm in the middle. So the challenge is to catch the quadratic effect of the catalyst. The response to cure temperature is also curved, but it is not so difficult to catch, since cure temperature also has a strong linear effect that every screening design would find.

We used the same four models as in the first problem (Opus 5.5, Sonnet 5.5, Opus 5 and Sonnet 5), and each solved the problem ten times. Because this problem is harder, we also varied the reasoning effort, which sets how much the model thinks before it acts. Each model ran ten times at low effort, as in the first problem, and ten times at extra high (xhigh) effort.

In 77 of the 80 sessions, the recommended settings came within 0.6 s of the best possible hardness of 91.5 s. But only 62 sessions found the quadratic effect of the catalyst. The other 18 judged the catalyst inactive. Seventeen of them left it at 200 ppm, the middle of its range, which happens to be its best setting, so they reached the right hardness by luck. One moved it to 400 ppm and ended 13 s below the optimum.

Whether a session found the catalyst depended mostly on its first design. Sessions that opened with a definitive screening design found the catalyst in 47 of 51 cases, sessions that opened with a two-level screening design in 15 of 29. More thinking paid off for the newer models. At xhigh, Opus 5.5 and Sonnet 5.5 found the catalyst every time and needed 36 runs on average instead of 47 at low effort. That brings them close to three human DoE experts, who needed about 32 runs. The older models gained nothing from more thinking.

What DoE is and why AI agents should run it

Trial and error is still standard practice in coatings R&D. A formulator changes one factor, looks at the result and decides what to try next. This one-factor-at-a-time (OFAT) approach is intuitive, but it covers only a narrow slice of the design space. It also cannot detect interactions, where the effect of one factor depends on the setting of another. Design of experiments (DoE) solves both problems. It varies several factors at once in a structured plan, so each run carries information about every factor and about their interactions.

Still, DoE is not widely adopted in coatings R&D. The method is not intuitive, and many researchers struggle to apply it. Many believe that DoE needs a huge number of runs once there are many factors. That belief comes from confusing DoE with the full factorial design, which tests every combination of factor levels. At two levels per factor, a full factorial design needs 2^k runs. That is 16 runs for four factors, 32 for five and 4,096 for twelve. Above four factors it is rarely practical.

But the full factorial is not the only option. Designs that need only a fraction of the runs have existed since the 1940s. A fractional factorial design runs a chosen subset of the full factorial, for example 16 of the 32 runs for five factors. A Plackett-Burman design screens up to 11 factors in 12 runs (instead of 2,048 runs for a full factorial design). A definitive screening design screens twelve factors in 25 runs (instead of 4,096 runs). It tests every factor at three levels, so it also shows which factor has a strong curvature. A D-optimal design is built for any run budget and can leave out factor combinations that cannot be made.

These designs work because of three principles that hold in most real experiments. Effect sparsity says that only a few of many factors have a large effect. Effect hierarchy says that main effects tend to be larger than two-factor interactions, and two-factor interactions larger than higher-order ones. Effect heredity says that an interaction is most likely active when at least one of its factors also has an active main effect.

Box 1 DoE terms

Main effect. The change in the result when a single factor moves from its low to its high setting, averaged over the settings of all other factors.

Interaction. The extra change in the result when factors act together. For example, a catalyst might shorten the curing time only at a low baking temperature. A two-factor interaction involves two factors. A higher-order interaction involves three or more.

Active. An effect that is large enough to matter in practice. Main effects and interactions that are too small to matter are inactive.

Curvature. The effect of a factor does not follow a straight line between its low and high setting. Its best setting can lie in between, which a design with only two levels per factor cannot see.

The same principles make DoE sequential. A study does not have to be one large experiment. It moves in stages. A screening design finds the few active factors, a follow-up design separates the interactions, and a response surface design such as a central composite design finds the optimum. Each stage builds on what the previous one found, which keeps the total number of runs low.

Few formulators know how to choose the right design, or when to move to the next stage. But AI agents based on large language models are now able to run experiments. They can plan a design, analyze the results and decide on the next step. Paired with a formulator, the agent brings the experimental design knowledge, and the formulator brings the domain knowledge.

This study asks how reliably an agent finds the best settings when it has to plan, run and analyze every experiment on its own. The first problem, the adhesion of an epoxy primer on steel, was a textbook case with few active factors and one interaction. This second problem is slightly more complicated due to curvature in the response.

Methods

The problem

The agent has to improve the pendulum hardness of a 2K PU clearcoat. The problem description with the task and 12 factors and their ranges is shown in Box 2.

Box 2 The problem description

A customer rejects our 2K PU clearcoat because the cured film is too soft. The recipe and the baking process have grown over the years, and nobody can say anymore which settings control hardness. Below are 12 factors, the full recipe and the full process window. Tell me which of them influence hardness and give me the settings that maximize it. The response is König pendulum hardness in seconds, measured after full cure. Higher is better. Only settings inside the ranges are allowed, and the measurements are noisy.

Factors

ColumnGroupFactorUnitLow (-1)High (+1)
X01FormulationNCO:OH index-0.91.3
X02FormulationPolyol OH number (blend)mg KOH/g140180
X03FormulationCatalyst (DBTDL)ppm0400
X04FormulationFlow additive%0.11.0
X05FormulationMatting agent%04
X06FormulationSolvent dilution%1025
X07ApplicationFlash-off timemin520
X10ApplicationDry film thicknessµm3060
X11ApplicationBooth humidity% r.h.3070
X12ApplicationSubstrate temperature°C1830
X08CureCure temperature°C60100
X09CureCure timemin2060

The problem is based on a truth model the agent doesn’t know, a simple equation that gives the hardness for any combination of factor settings.

yhardness=80+6x1+x3−14x32+10x8−9x82+4x1x8y_\text{hardness} = 80 + 6x_1 + x_3 - 14x_3^2 + 10x_8 - 9x_8^2 + 4x_1x_8

Only 3 of the 12 factors are active, the NCO:OH index (x₁), the catalyst (x₃) and the cure temperature (x₈). The other nine have no effect. The effect of the index is linear, so a higher index always gives a harder film. Cure temperature has a linear and a quadratic term, so hardness rises with temperature and levels off near the top of the range. The catalyst has almost only a quadratic term, −14x₃². Hardness peaks in the middle of the catalyst range and falls by about the same amount toward both ends, so a design that tests only the two extreme levels measures nearly the same hardness at 0 and 400 ppm and sees no catalyst effect. The last term, 4x₁x₈, is an interaction between index and cure temperature. A higher index pays off most at a high cure temperature. The true optimum is 91.5 s, at the highest index of 1.3 (x₁ = +1), a catalyst level of about 210 ppm (x₃ = 0.04) and a cure temperature of about 96 °C (x₈ = 0.78).

Why this problem is harder than the first one

The first problem, the adhesion of an epoxy primer, was a textbook case. It had 8 factors, all effects were linear, and the best settings lay at a corner of the design space. A good screening design showed the three active factors and their interaction, and a few confirmation runs at the best corner finished the job. The best agents needed about 25 runs (which is close to what the human experts also needed).

This second problem looks similar, with three active factors and one interaction. But it has 12 factors instead of 8, and the response is curved. It also has a trap, the catalyst. A two-level screening design tests the catalyst only at 0 and 400 ppm, where hardness is nearly the same, so the catalyst looks inactive. Center points show that the response is curved, but not which factor causes it. Cure temperature is curved too and clearly active, so it is the easy explanation, but not the full one.

An agent can therefore reach the right hardness without ever finding the catalyst, simply by leaving it at the middle of its range. So in this problem, it matters not only whether an agent finds the best settings, but also whether it found the quadratic effect of the catalyst or reached the optimum by luck.

Virtual lab

Similar to the first problem, the agents run their experiments in a virtual lab. An agent submits a run sheet and gets back the measured response for each run. The lab calculates the response from the truth model and adds normally distributed noise with σ = 3.0 s.

Agent and skills

As in the first problem, each session is one Claude Code session with one of four models (Opus 5.5, Sonnet 5.5, Opus 5 and Sonnet 5). What is new is the reasoning effort, which sets how much the model thinks before it acts. The first problem ran at low effort only. This time each model ran ten times at low and ten times at extra high (xhigh) effort. The same two skills as previously guide the agent.

A doe-researcher skill tells it to act as an expert in experimental design and to reach the objective with the fewest experiments. It tells the agent to believe in effect heredity and effect sparsity.

A performing-experiments skill explains how to submit run sheets and states the rules for the virtual lab.

Procedure

The procedure is the same as in the first problem, with one addition. The agent also had to report its final model, which shows whether it found the quadratic effect of the catalyst.

Prompt and report

Every session was a fresh Claude Code session in its own copy of the project (a git worktree). The problem description was saved as a Markdown file in the project. Each session started with the following prompt, which names that file:

Please start working on project: projects/E1-hardness.md -⁠> create a project folder. Then get going. Continue until you are done.

The agent then worked on its own. It planned each design, ran it in the lab, analyzed the results and chose the next step, until it considered the problem solved. The agent was told to include seven items in its final report:

  1. Total number of runs
  2. Suggested optimum
  3. Recommended settings
  4. Final model
  5. Designs in the order they ran
  6. File names of all run sheets and results
  7. A short summary of the procedure and why each design was chosen

Independent sessions

Although each session had its own worktree, two mechanisms could still link sessions. Claude Code keeps a memory for each project that every new session inherits, so the memory was wiped before every batch. The ten sessions of one model and effort ran at the same time, and sessions running at the same time can find each other’s worktrees. The project instructions (CLAUDE.md) told the agents never to look at other worktrees. Before a session was counted, its full record of actions was checked for any access to another worktree, to the memory or to the lab’s internal files, and the lab results of sessions that ran the same run sheet were checked for independent noise. No session used another session’s results.

Results

Each agent had to find the settings that give the hardest clearcoat. It could change 12 factors, but only 3 of them were active. Two of these, cure temperature and the catalyst, had a curved effect. Cure temperature also had a strong linear effect, so any screening design should flag it as active and carry it into the next designs. The catalyst had no linear effect. Plotted against the catalyst level, hardness forms an upside-down parabola with its top in the middle of the range. A two-level screening design testing only the two ends could miss the catalyst. The agent planned, ran and analyzed every experiment on its own in a virtual lab with noisy results. The best possible hardness is 91.5 s.

Each of the four models solved the problem ten times from scratch at low effort and ten times at xhigh effort. One such attempt is a session. Figure 1 shows how many runs the sessions of each model needed on average, at both efforts. Table 1 sums up the sessions of each model and effort.

Figure 1. Average runs per session for each model at low effort (gray) and xhigh (orange). The arrow points from low to xhigh. The red line shows the human expert baseline, the average of three experts plus three confirmation runs.

Table 1. Results of the ten sessions for each model and effort.

ModelEffortRunsRuns on averageFirst designFound the catalystSuggested optimum, s
Opus 5.5low31 to 7247.8DSD (9), Plackett-Burman (1)887 to 92
Opus 5.5xhigh28 to 4535.5DSD (10)1090 to 94
Sonnet 5.5low33 to 5745.8DSD (7), Plackett-Burman (2), fractional factorial (1)789 to 92
Sonnet 5.5xhigh28 to 4836.7DSD (10)1090 to 93
Opus 5low50 to 9168.1Plackett-Burman (6), DSD (4)1089 to 94
Opus 5xhigh59 to 12073.2DSD (7), Plackett-Burman (3)1091 to 94
Sonnet 5low39 to 5649.1Plackett-Burman (10)490 to 96, one 77
Sonnet 5xhigh41 to 7251.2Plackett-Burman (6), DSD (4)389 to 93

Finding the optimum was not the hard part. In 77 of the 80 sessions, the agent found the optimum. But only 62 sessions found the quadratic effect of the catalyst, and the number of runs ranged from 28 to 120.

Whether a session found the catalyst depended mostly on its first design. Sessions that opened with a definitive screening design found it in 47 of 51 cases, sessions that opened with a two-level screening design in only 15 of 29.

How many runs a session needed depended mostly on what it did after screening. Sessions with few runs added a handful of runs to the screening data. Sessions with many runs ran a full CCD, a second screening design or extra checks of factors they had already judged inactive.

At xhigh effort, where the models think longer before they act, Opus 5.5 and Sonnet 5.5 opened with a DSD in all 20 sessions, found the catalyst every time and needed about 10 runs fewer than at low effort. The older 5 model family did not profit from the extra thinking.

How close the sessions came to the optimum

The 77 sessions that found the optimum all set the NCO:OH index to its top level, the catalyst between 190 and 220 ppm and the cure temperature between 90 and 100 °C. Their recommended settings give a true hardness within 0.6 s of the optimum.

Two sessions landed about 1.6 s below the optimum, within the noise of a single measurement. One Sonnet 5 session at low effort set the cure temperature too low, at 87 °C, because its model ignored the catalyst. One Opus 5.5 session at xhigh read too much into two low confirmation runs and moved the catalyst to 140 ppm.

One Sonnet 5 session at low effort clearly missed the optimum. It judged the catalyst inactive and set it, like the other inactive factors, to the side its screening design had slightly favored. That put the catalyst at 400 ppm. Its model predicted 77 s and its confirmation runs gave 79 s, so it saw no reason to doubt the result. The true hardness at these settings is 78 s, 13 s below the optimum.

Not all of the 77 sessions at the optimum got there by skill, though. Sixteen of them judged the catalyst inactive and left it at the middle of its range, as most agents do with factors they think are inactive. The middle, 200 ppm, happens to be the catalyst’s optimum, so these sessions found the right hardness by luck.

How the successful sessions found the catalyst

Figure 2 shows which sessions found the catalyst and which design they started with. Of the 80 sessions, 51 opened with a definitive screening design (DSD), 28 with a Plackett-Burman design and one with a fractional factorial design.

A DSD tests every factor at three levels, so the catalyst’s peak shows up in the screening data directly. A two-level design sees no catalyst effect. Most two-level sessions added a few center runs, which came back 20 to 25 s above the corner runs. So they saw that the response curves, but not which factor causes it. That made the catalyst easy to miss.

The sessions that missed the catalyst, including four that opened with a DSD, saw the curvature but mostly put it on cure temperature. Cure temperature curves too and has a much larger main effect, so it was the obvious choice.

Figure 2. Every session as one square, for each model and effort. The color shows the first design. A filled square is a session that found the quadratic effect of the catalyst, an empty one a session that judged the catalyst inactive.

How the newer models save runs at xhigh

Figure 3 shows the design each session ran after its screening design. The screening design itself took about 25 runs at both efforts, so the runs were saved on this second design.

At low effort, most Opus 5.5 and Sonnet 5.5 sessions ran a central composite design (CCD) of 10 to 29 runs next. The CCD covered only the factors the screening had found active and held the others at their center. But in a typical session, 9 of its 15 settings had already been tested in the screening at the same index, catalyst and cure temperature. The sessions could have reused those runs but ran them again.

At xhigh, most sessions first fitted a curved model to the DSD data and then added only the 5 to 14 runs the model still needed, mostly picked by a D-optimal or I-optimal search. Such a search builds on the runs already made, and the DSD runs already test the three active factors at three levels. On average, the second design shrank from 17 to 7 runs for Opus 5.5 and from 19 to 8 runs for Sonnet 5.5.

Three sessions at xhigh skipped the second design. They argued that a DSD can fit a full model with curvature when only three factors are active, ran three confirmation runs and stopped at 28 runs, still at the optimum. One Sonnet 5.5 session at low effort also went straight from the DSD to confirmation, but ran eight confirmation runs and stopped at 33.

The older models, Opus 5 and Sonnet 5, did not save runs at xhigh. They needed about as many runs as at low effort.

Figure 3. The design each session ran after its screening design, for each model and effort. Confirmation runs do not count here, so None means the session went straight to confirmation. Optimal design or augmentation means runs added to the screening data. Axial runs test single factors at extra levels.

Newer models at xhigh come close to a human expert

Three DoE experts solved the same problem in the online lab. One opened with a 16-run D-optimal screening design and then fitted a curved model in five factors. The other two opened with a DSD. They needed 28, 30 and 32 runs, including one validation run in two cases. The red line in Figure 1 is a human expert baseline built from these runs. As in the first problem, it counts the experts runs without validation and adds three confirmation runs, since most agents also ran them. A human expert would therefore need about 32 runs. At low effort, all four models needed more runs than that. At xhigh, Opus 5.5 and Sonnet 5.5 came close, with about 36 runs on average.

The experts recommended settings give a true hardness of 91.3, 90.7 and 86.9 s. The expert at 86.9 s stopped with the NCO:OH index at 1.2 instead of 1.3 and the cure temperature at 90 °C. This expert also left the catalyst out of the recommended settings and likely missed it. Of the other two, one explicitly named the catalyst as active and the other most likely found it as well. The experts results come from a small, self-reported sample collected through an online challenge on LinkedIn, so they are only a rough comparison.

Conclusion

This second problem was slightly more difficult than the first one, mostly because of the catalyst. Hardness peaked in the middle of the catalyst range, with almost no linear effect, which made the catalyst easy to miss.

The 5.5 models again did better than the older 5 models. They needed fewer runs and found the catalyst more often.

Extra thinking pays off for the 5.5 models. At xhigh, they came close to the human experts in the number of runs. They opened with a DSD every time, and they built on the screening runs they already had instead of running new ones. The older models did not profit from extra thinking.

A good advice is therefore to run the 5.5 models at xhigh effort by default. Planning then takes longer, often about half an hour for Opus 5.5, but that is nothing compared with the time the lab work takes.

Acknowledgements

Many thanks to the human DoE experts.

Appendix

Agent reports