At the Google I/O 2024 presentation, the company showed Project Astra, a virtual assistant with artificial intelligence and visual recognition based on Google Gemini that is currently under development. Speaking about Astra, Demis Hassabis, CEO of DeepMind Experimental Laboratory, said that his team has always wanted to develop a universal AI agent that would be useful in everyday life.
Project Astra is a program whose main input interfaces are a camera and voice. A person with a smartphone points its camera at different parts of the office and gives Astra tasks: «Tell me when you see something that makes a sound». When the virtual assistant saw a speaker next to the monitor, it responded: «I see a speaker that makes sound». The demonstrator drew an arrow to the top circle on the speaker and asked: «What is the name of this part of the speaker? The program immediately answered: «This is a tweeter. It makes high-frequency sounds».
Then, in a video that Google says was recorded in one take, the tester walked over to a cup of crayons down the table and asked «Give me a creative alliteration about this», to which he was responded by saying «Creative crayons are cheerfully colored. They tend to create colorful creations». The video goes on to show Astra identifying and explaining parts of the code to the monitor and telling the user what neighborhood they are in based on the view from the window. Astra was able to answer the questions: «Do you remember where you saw my glasses?» even though they were hidden. «Yes, I know. Your glasses were on the table next to the red apple».
After that, the tester put on the glasses and the video got a first-person perspective. Using the built-in camera, the glasses scanned the environment, and the eyes were focused on the diagram on the whiteboard. The person in the video asked: «What can I add here to make this system faster? The program replied: «Adding a cache between the server and the database can improve speed».
The tester looked at a pair of cats on the whiteboard and asked: «What does this remind you of? Astra said: «Schrödinger’s cat» When a tiger plush toy was placed next to a golden retriever and asked to name the group, Astra answered «Golden stripes».
The demonstration proves that Astra not only processed visual data in real time, but also memorized what it saw and worked with the stored information. According to Hassabis, this was due to faster information processing by continuously encoding video frames, combining video and speech input with a timeline of events, and caching this information for efficient use.
On video, Astra was quite responsive to requests. Hassabis noted in a blog post: «While we’ve made incredible progress in developing AI systems that can understand multimodal information, reducing response times to conversational is a challenging engineering task». Google is also working on giving its AI a greater range of diversity and emotional nuance.
Although Astra remains an early feature with no specific launch plans, Hassabis said that similar assistants could be available in a phone or glasses in the future. There is no information yet on whether such glasses will become a successor to Google Glass, but the DeepMind executive noted that some of the demonstrated capabilities will be available in Google products later this year.
Source: Engadged
Spelling error report
The following text will be sent to our editors: