How do you get AI Scientist to focus on replication rather than make new ideas

[BradKML]

Sorry for phrasing it this way, but if there are ways of using AI to check and expand on other papers, and synthesize new knowledge through meta-analysis adjacent work, that would be sweet.

• replication requires direct access to the codebase (or creating new code copying the same methodologies) + injecting new data
• multiple replications can be done if multiple papers tends towards the same set of topics
• synergizing multiple improvements methods should be as favorable as adding a new method of improvements
• the benchmark of "novelty" should be calibrated from time to time

[clxyder]

Considering that the research creation process of experimentation, code, and paper has been demonstrated.

It would be extremely interesting to reverse the process, i.e., go from a research paper with results to a code base that can replicate said results.

It would further accelerate the AI Scientist's ability to innovate.

[clxyder]

OpenAI just released PaperBench for exactly this use case.

https://arxiv.org/abs/2504.01848