Following Sora by OpenAI, which generates digital virtual worlds in the style of Minecraft Google’s DeepMind generative model has distinguished itself with its own talent for creating games.
How to MIT Technology Review reports, Genie can take a short text description, a hand-drawn sketch, or a picture and turn it into a video game in the style of classic 2D platformers like Super Mario Bros. You can’t expect a fast pace from AI creations at the moment — they run at one frame per second compared to the typical 30-60 frames per second in most modern games.
Genie was trained on 30,000 hours of gameplay videos from the Internet of hundreds of 2D platformers. Other models use a similar approach — Nvidia trained in this way GameGAN, which was famous for creating a Pac-Man clone.
All the previous examples have trained models for input actions (such as pressing buttons on a controller), and combining them with video (video of Mario jumping is combined with the action «Jump») — but adding such tags is very labor-intensive. Instead, Genie only learns from videos and analyzes which of the possible actions will make the character change its position
Genie generates each new frame of the game on the fly depending on the player’s action: press «jump» and it will update the current image to show the game character jumping; press «left» and the image will change to show the character moving to the left
As for speed limits, it’s only a matter of time, says Tim Rocktel, a researcher at Google DeepMind:
«There are no fundamental limitations that prevent us from achieving 30 fps. Genie uses many of the same technologies as today’s large-scale speech models, where significant progress has been made in improving output rates».
Genie has also learned some typical visual features of platformers — such as parallax, when the foreground moves sideways faster than the background — and often adds this effect to games.
The model is currently in the status of an internal research project and is not publicly available. At the same time, games are not the only area of interest for Genie’s creators.
The team is working on AI-controlled bots that can stay in a virtual environment and solve various tasks by trial and error. In 2021, another DeepMind team developed a virtual a playground called XLand where similar bots worked. Apparently, Genie may well create such an environment in the future.