Project Astra
A universal AI agent that is helpful in everyday life
Building on our Gemini models, Project Astra explores the future of AI assistants that can process multimodal information, understand the context you’re in, and respond naturally in conversation.
This demo shows two continuous takes: one with the prototype running on a Google Pixel phone and another on a prototype glasses device.
Putting Project Astra to the test
Explore more of Project Astra’s capabilities.
All demos were taken in one continuous take, in real-time, on a Google Pixel phone or a prototype glasses device.
Under the hood
To be truly useful, an agent needs to understand and respond to the complex and dynamic world just like people do — and take in and remember what it sees and hears to understand context and take action. It also needs to be proactive, teachable and personal, so users can talk to it naturally and without lag or delay.
Continuously encoding video frames
Combining the video and speech input into a timeline of events
Caching information for efficient recall
While we’ve made incredible progress developing AI systems that can understand multimodal information, getting response time down to something conversational is a difficult engineering challenge.
Over the past few years, we've been working to improve how our models perceive, reason and converse to make the pace and quality of interaction feel more natural.
By leveraging our leading speech models, we also enhanced how our AI agents sound giving them a wider range of intonations. These agents can better understand the context they’re being used in, and respond quickly in conversation.
What’s next
With technology like Project Astra, it’s easy to envision a future where people could have an expert AI assistant by their side, through a phone or glasses. And some of these capabilities are coming to Google products, like the Gemini app and web experience, later this year.
Acknowledgements
This work was made possible by the exceptional contributions of: Agustin Dal Lago, Alexander Chen, Alistair Muldal, Bibo Xu, Chen Yan, Dan Motzenbecker, Darren Carikas, Federico Carnevale, Gregory Wayne, Joe Stanton, Jonas Degrave, Jordan Griffith, Kitty Stacpoole, Mehdi Mirza, Michael Chang, Misha Dashevskiy, Nikolai Grigorev, Pavel Dubov, Praveen Srinivasan, Reed Roberts, Suz Chambers, Toshiyuki Fukuzawa.
We extend our gratitude to Adrian Bolton, Alexandre Moufarek, Alex Lince, Alexey Guseynov, Aliya Ahmad, Amy Stuart, Ana Salazar, Andrew Rhee, Ankesh Anand, Antoine He, Antoine Yang, Antoine Miech, Antoine Yang, Arielle Bier, Bethanie Brownfield, Brona Robenek, Brooke Taylor, Charlie Chen, Charlie Deck, Charlie No, Arslan Chaudhry, Chongyang Shi, Chris Lock, Chris Ocana, Daan van Esch, David Allin Reese, Demetra Brady, Dilan Gorur, Doug Fritz, Duncan Williams, Emanuel Taropa, Emma Wang, Enrique Piqueras, Folake Abu, Fred Alcober, Gaby Pearl, Grant Yoshida, Hsiao-Yu (Fish) Tung, Ian Graetzer, Jacob Austin, James Carr, Jamie Hayes, Jean-baptiste Alayrac, Jerry Li, Jerry Torres, Jessica Gottlieb, Jessica Lo, Johnny Lee, Jonas Fromseier Mortensen, Jonathan Fildes, Joost Korngold, Josip Djolonga, Juliette Love, Junwhan Ahn, Juston Payne, Karel Lenc, Leland Rechis, Louise Griffiths, Matthew Mauger, Mario Lučić, Matt Miller, Mehdi Bennani, Mikel Rodriguez, Minh Truong, Mónica Carranza, Mor Hazan Taege, Natalie Clay, Natalie Vegh, Neil Rabinowitz, Nevena Lazic, Nishtha Bhatia, Omar Estrada, Oskar Bunyan, Peter Choy, Phil Chen, Piermaria Mendolicchio, Ross West, Sam Lawton, Sarah Chakera, Sarah York, Sean Sechrist, Shashwath Santosh, Sholto Douglas, Sina Samangooei, Soheil Hassas Yeganeh, Soyeon Kim, Sridhar Thiagarajan, Tarik Abdel-Gawad, Tianqi Fan, Timothy Nguyen, Tony (Tuấn) Nguyễn, Trudy Painter, Vijay Bolina, Woohyun Han, Yana Lunts, Yash Katariya, Yury Kartynnik, Zeina Oweis.
We also acknowledge the many other individuals who contributed across Google DeepMind and our partners at Google.