Depositphotos
Sakana, a Japanese artificial intelligence startup, has announced the review of the first scientific article generated by AI.
However, the company’s statement should be taken with many reservations. At present, a large number of researchers believe that artificial intelligence is not ready to work on a par with scientists in creating research materials. Others are of the opinion that AI has the appropriate potential but this area of artificial intelligence activity is currently at an early stage.
Developers at Sakana consider themselves to be of the second type. The company stated that they used The AI Scientist-v2 AI model to create scientific material, which was then presented at the ICLR seminar as part of the authoritative conference on artificial intelligence.
Sakana representatives claim that the organizers of the seminar and the ICLR conference management agreed to work with them as part of an experiment with double-blind review of manuscripts, created with the help of AI.
The company claims to have collaborated with researchers from the University of British Columbia and Oxford to present three generated by AI documents for the aforementioned workshop for review. AI Scientist-v2 fully generated all three scientific publications, including scientific hypotheses, experiments, experimental code, data analysis, visualizations, text, and titles.
«We generated research ideas by providing the AI with an abstract and description of the seminar This ensured that the generated articles were relevant to the topic and suitable for presentation», — said Robert Lange, Sakana’s researcher and founder.
One of these three scientific materials, namely, an article critically reviewing methods of training AI models, was allowed to be considered at the seminar. Sakana stated that it immediately withdrew the article before it was published in the interests of transparency and respect for the ICLR conventions.
«The accepted paper simultaneously introduces a promising new method for training neural networks and shows that empirical challenges remain It provides an interesting data point to spur further scientific research», — Robert Lange emphasized.
However, this story, which seems very promising at first glance, has many important nuances. Sakana admitted that from time to time, the model AI Scientist-v2 made mistakes when referencing other related scientific materials. For example, it mistakenly attributed the analyzed method is based on material from 2016 instead of the original scientific article from 1997.
Material generated by AI Scientist-v2, was also not as thoroughly reviewed as some other peer-reviewed publications. Since the company withdrew it after the initial review, the article did not receive an additional «meta-review», during which the seminar organizers could theoretically reject it.
At the same time, the approval rate for conference workshops is usually higher than the approval rate for the main event of the conference, as Sakana representatives also mentioned in their message. The company admitted that not one of the scientific articles generated by AI passed the internal review for publication as part of the main conference event.
«Sakana employees selected articles from a certain number of generated articles, i.e. they used human judgment to select the results they thought would be relevant I think it shows that humans plus AI can be effective, not that AI alone can create scientific papers», — said the artificial intelligence researcher and associate professor at the University of Alberta Matthew Guzdial.
Technical disadvantages of artificial intelligence including a tendency to fantasize, make many scientists avoid endorsing this tool for serious scientific work.
Sakana does not claim that its AI model is capable of producing groundbreaking or fundamentally new scientific works. Rather, the goal of the experiment was to examine the quality of AI-authored research and emphasize the urgent need to adopt regulations that would apply to scientific publications created by artificial intelligence.
Source: TechCrunch