Google Assistant is now powered by Gemini — sort of

Google Assistant as we knew it is on the way out. Well, at least its guts.

Late last year, Google added AI capabilities from its chatbot Bard to Google Assistant, allowing the virtual helper to make sense of images and draw on data in emails, documents and more. Now, the company’s giving users the choice of replacing the AI models driving Google Assistant’s conversational skills with Gemini, Google’s family of newer GenAI tech.

Soon, via a new Gemini app for Android devices (which replaces the existing Google Assistant app) and an updated Google app on iOS (set to roll out in the coming weeks), users will be able to access a new Gemini-powered Google Assistant experience. It’ll be mobile-only; Google Assistant devices like Nest and Home speakers and displays won’t see changes just yet.

The experience comes in the form of an overlay that acts as a frontend for the Gemini models and provides contextual recommendations and suggestions, Sissie Hsiao, Google Assistant VP, told TechCrunch in a press briefing.

“This overlay allows you to do some really interesting things with Gemini,” she said. “For example, you can generate a caption based on a picture that you’ve already taken that’s behind the overlay. You might be reading an article, and bringing up the overlay allows you to not only understand the article more deeply, but ask questions about the article that you’re reading now.”

Indeed, the Gemini-powered Assistant — which, depending on the device, can be summoned with a corner swipe, power button tap or “Hey Google” hotword — can accept images as well as text and voice commands. For example, you can upload a picture of a plant with the instructions “I just bought this plant — give me tips and YouTube videos to help me take care of it” and the Gemini-powered Assistant will do its best to satisfy the request.

If you’ve ever tried a Gemini model on the web via Bard, you’re probably aware that Gemini’s capable of a range of GenAI tasks — and that’s not changed with the Assistant integration. Google gives a few example prompts in the onboarding flow, like “Help me craft a text response to my friend who is stressed at work,” “Give me some ideas to surprise my concert-loving friend on their birthday” and “Help me incorporate more plant-based options in my diet.” As on the web, Gemini in Assistant can also create images, although it’s not clear which model’s doing the generating; Bard previously tapped Imagen 2 for this purpose.

As you’d expect, conversations with the Gemini-powered Assistant on mobile will carry over to Gemini-powered experiences on the web — and vice versa — if you’re signed into the same Google Account in both places.

If for whatever reason the Gemini-powered Assistant experience isn’t to your liking, a menu screen in the Assistant app on Android and Google app on iOS lets you switch back to the old models at any time.

Now, the Gemini models aren’t completely replacing the old models underpinning Google Assistant — at least not at this juncture. Features including quick phrases, which let you skip saying “Hey Google” for specific tasks, will continue using the old models. So will features relating to setting timers, making calls and controlling smart home devices. In addition, Google notes on the aforementioned settings menu that, even when the Gemini models are selected as the default models in Assistant, apps with Assistant integration like Google Maps may keep using the old models.

In a move that might irk some longtime Google Assistant users, the most capable of the Gemini models, Gemini Ultra, isn’t available to power Assistant for free. It and the capabilities it brings, like better reasoning, coding and instruction-following skills, are gated behind a new product, Gemini Advanced, that’s part of a new subscription, the Google One AI Premium Plan, priced at $20 per month. The free Gemini-powered Assistant leverages the less-capable Gemini Pro, which got a middling reception when it launched in Bard in December.

Google’s positioning the Google One AI Premium Plan — an expansion of its existing Google One program, initially available with a two-month free trial — as the only way to access its latest and greatest GenAI features through Assistant, in fact. Users will have to maintain a subscription to get future Gemini Ultra updates including expanded multimodal capabilities, more interactive coding features and deeper data analysis capabilities.

To sweeten the pot, the AI Premium Plan comes with 2TB of storage and, in the near future, Gemini-powered features in Gmail, Google Docs, Google Slides, Google Sheets and more. But I wonder whether that’ll be enough to pacify Google Assistant users who perceive the AI Premium Plan as an effort to charge for model updates that were previously free.

The new Gemini-powered Assistant is available in English starting today in the U.S. It’ll come to more locations — and Japanese and Korean — starting next week.