Google IO 2024 Google Unveils Mind Blowing AI Changes for Android in 2024 911

Google I/O 2024 - Google Unveils Mind-Blowing AI Changes for Android in 2024 - 9/11

Full tutorial link > https://www.youtube.com/watch?v=DFkYs6_PKhM

🚀 Dive into the future of Android with us at Google I.O. 2024! Discover groundbreaking AI innovations that are set to transform your Android experience. From AI-powered search features to the all-new Gemini assistant, Android is getting smarter in ways you have to see to believe. Watch as we unveil the latest upgrades and new features that promise to redefine how we interact with our devices. Don't forget to like, subscribe, and hit the bell icon to stay updated on all things Google AI! 🌟

00:00:00 Introduction to Google I.O. 2024

00:00:06 Welcome Back to Google I.O.

00:00:11 Overview of AI in Google Products

00:00:19 Innovations Coming to Android

00:00:29 The New AI Era on Smartphones

00:00:58 Three Major AI Breakthroughs

00:01:34 Introduction of Circle to Search

00:02:10 Enhancements in Circle to Search

00:03:42 Gemini Becomes Core to Android

00:04:01 Launch of Gemini on Android

00:05:30 Interaction with Gemini and Live Demos

00:08:05 Exclusive Features on Android with Gemini

00:09:00 Enhancements in Accessibility with Gemini Nano

00:10:03 Protecting Users from Fraud with AI

00:11:22 Future Potential of On-Device AI

00:12:02 Announcements for Developers

00:12:17 Preview of Tomorrow's Session

00:12:28 Wrap Up and Handover

At Google I.O. 2024, the focus was significantly on AI enhancements across various Google products including Gemini, Search, Workspace, and especially Android. The session began with an exhilarating introduction to the transformative capabilities AI is poised to bring directly to users through their Android devices. The overarching theme was the deep integration of AI to make smartphones not just tools but intelligent partners in everyday activities.

The presentation outlined three major AI breakthroughs for the year. First, it introduced "Circle to Search," an AI-powered search tool integrated within the Android operating system, allowing users to search directly from any screen or app without needing to switch between interfaces. This feature was demonstrated to be particularly useful in scenarios ranging from shopping for fashion items to helping students with homework directly from their mobile devices.

Secondly, the Gemini AI assistant's capabilities were significantly expanded. Initially launched a few months prior, Gemini has been integrated at the system level to provide seamless assistance without needing to navigate away from the current task. This includes context-aware suggestions and actions that anticipate the user's needs based on their activity. For instance, when receiving a text about playing pickleball, Gemini could assist in generating humorous responses or relevant information right in the conversation window.

Furthermore, Gemini's role in enhancing accessibility through the new Gemini Nano was highlighted as a pivotal upgrade. This version of Gemini assists users with disabilities by providing richer descriptions of images or complex PDFs directly on the device, thus ensuring privacy and reducing latency. The example showcased involved an accessibility feature named TalkBack, which aids users through audio feedback.

A significant portion of the presentation was dedicated to demonstrating how these AI enhancements work in real-time scenarios, emphasizing the speed and privacy benefits of processing data directly on the device rather than in the cloud. This was illustrated through examples such as Gemini preventing fraud by identifying and alerting users about potential scam calls instantly.

Looking ahead, Google announced that these AI features, including the newly introduced on-device capabilities, will soon be available on hundreds of millions of devices. This rollout is part of a broader strategy to make Android the first mobile OS with a built-in foundational model, enabling multimodal interactions where the phone understands inputs not just through text but also through visuals and audio.

The session concluded with a forward-looking statement about the potential of AI in Android, teasing further announcements in the upcoming developer keynote and subsequent updates on Android 15. The advancements presented signify a substantial leap towards a more integrated, intuitive, and intelligent mobile experience, emphasizing Google's commitment to leading innovation in AI for mobile technology.

Video Transcription

• 00:00:00 Ok, we continue to the Google I.O. 2024 with the part 9.

• 00:00:06 Hi everyone. It's great to be back at Google I.O.

• 00:00:11 Today, you've seen how AI is transforming our products across Gemini, Search, Workspace and more.

• 00:00:19 We're bringing all these innovations right onto your Android phone.

• 00:00:23 And we're going even further to make Android the best place to experience Google AI.

• 00:00:29 This new era of AI is a profound opportunity to make smartphones truly smart.

• 00:00:36 Our phones have come a long way in a short time, but if you think about it,

• 00:00:41 it's been years since the user experience has fundamentally transformed.

• 00:00:45 This is a once in a generation moment to reinvent what phones can do.

• 00:00:51 So we've embarked on a multi-year journey to reimagine Android with AI at the core.

• 00:00:58 And it starts with three breakthroughs you'll see this year.

• 00:01:03 First, we're putting AI-powered Search right at your fingertips,

• 00:01:08 creating entirely new ways to get the answers you need.

• 00:01:12 Second, Gemini is becoming your new AI assistant on Android, there to help you anytime.

• 00:01:20 And third, we're harnessing on-device AI to unlock new experiences that work as fast as you do

• 00:01:27 while keeping your sensitive data private.

• 00:01:31 Let's start with AI-powered Search.

• 00:01:34 Earlier this year, we took an important first step at Samsung Unpacked

• 00:01:38 by introducing Circle to Search.

• 00:01:41 It brings the best of Search directly into the user experience.

• 00:01:45 So you can go deeper on anything you see on your phone without switching apps.

• 00:01:51 Fashionistas are finding the perfect shoes, home chefs are discovering new ingredients

• 00:01:56 and with our latest update, it's never been easier to translate whatever's on your screen,

• 00:02:02 like a social post in another language.

• 00:02:05 And there are even more ways Circle to Search can help.

• 00:02:10 One thing we've heard from students is that they're doing more of their schoolwork

• 00:02:14 directly on their phones and tablets.

• 00:02:17 So we thought, could Circle to Search be your perfect study buddy?

• 00:02:23 Let's say my son needs help with a tricky physics word problem like this one.

• 00:02:28 My first thought is, oh boy, it's been a while since I've thought about kinematics.

• 00:02:34 If he's stumped on this question, instead of putting me on the spot,

• 00:02:38 he can circle the exact part he's stuck on and get step-by-step instructions

• 00:02:43 right where he's already doing the work.

• 00:02:46 Ah, of course, final velocity equals initial velocity plus acceleration times elapsed time.

• 00:02:53 Right.

• 00:02:55 I was just about to say that.

• 00:02:58 Seriously though, I love how it shows how to solve the problem, not just the answer.

• 00:03:04 This new capability is available today.

• 00:03:08 And later this year, Circle to Search will be able to tackle more complex problems

• 00:03:13 involving symbolic formulas, diagrams, graphs, and more.

• 00:03:18 Circle to Search is only on Android.

• 00:03:22 It's available on more than 100 million devices today,

• 00:03:26 and we're on track to double that by the end of the year.

• 00:03:31 You've already heard from Sissy about the incredible updates coming to the Gemini app.

• 00:03:42 On Android, Gemini is so much more.

• 00:03:46 It's becoming a foundational part of the Android experience.

• 00:03:50 Here's Dave to share more.

• 00:04:01 Hey, everyone.

• 00:04:02 A couple of months ago, we launched Gemini on Android.

• 00:04:06 And like Circle to Search, Gemini works at the system level.

• 00:04:10 So instead of going to a separate app, I can bring Gemini right to what I'm doing.

• 00:04:16 Now we're making Gemini context aware so it can anticipate what you're trying to do

• 00:04:22 and provide more helpful suggestions in the moment.

• 00:04:26 In other words, to be a more helpful assistant.

• 00:04:29 So let me show you how this works.

• 00:04:31 And I have my shiny new Pixel 8a here to help me.

• 00:04:35 So my friend Pete is asking if I want to play pickleball this weekend.

• 00:04:43 And I know how to play tennis, sort of.

• 00:04:45 I had to say that for the demo.

• 00:04:47 But I'm new to this pickleball thing.

• 00:04:49 So I'm going to reply and try to be funny.

• 00:04:51 And I'll say, is that like tennis but with pickles?

• 00:04:56 This will be actually a lot funnier with a meme.

• 00:04:59 So let me bring up Gemini to help with that.

• 00:05:01 And I'll say, create image of tennis with pickles.

• 00:05:06 Now, one thing you'll notice is that the Gemini window now hovers in place above the app

• 00:05:11 so they stay in the flow.

• 00:05:13 Okay.

• 00:05:14 So that generated some pretty good images.

• 00:05:16 What's nice is I can then drag and drop any of these directly into the messages app below.

• 00:05:21 So like so.

• 00:05:22 Cool.

• 00:05:23 Let me send that.

• 00:05:25 All right.

• 00:05:30 So Pete was typing.

• 00:05:31 And he says he's sending me a video on how to play pickleball.

• 00:05:34 All right.

• 00:05:35 Thanks, Pete.

• 00:05:36 Let's tap on that.

• 00:05:37 That launches YouTube.

• 00:05:38 But, you know, I only have one or two burning questions about the game.

• 00:05:41 And I can bring up Gemini to help with that.

• 00:05:44 And because it's context aware, Gemini knows I'm looking at a video.

• 00:05:49 So it proactively shows me an ask this video chip.

• 00:05:53 So let me tap on that.

• 00:05:54 And now I can ask specific questions about the video.

• 00:05:58 So, for example, what is the two bounce rule?

• 00:06:05 Because that's something that I've heard about but don't quite understand in the game.

• 00:06:09 By the way, this uses signals like YouTube's captions, which means you can use it on billions

• 00:06:13 of videos.

• 00:06:14 So give it a moment.

• 00:06:15 And there.

• 00:06:17 I get a nice distinct answer.

• 00:06:20 The ball must bounce once on each side of the court after a serve.

• 00:06:23 Okay.

• 00:06:24 Cool.

• 00:06:25 Let me go back to messages.

• 00:06:26 And Pete's followed up.

• 00:06:27 And he says, you're an engineer.

• 00:06:29 So here's the official rule book for pickleball.

• 00:06:32 Okay.

• 00:06:33 Thanks, Pete.

• 00:06:34 Pete's very helpful, by the way.

• 00:06:35 Okay.

• 00:06:36 So we tap on that.

• 00:06:37 Launches a PDF.

• 00:06:38 That's an 84-page PDF.

• 00:06:39 I don't know how much time Pete thinks I have.

• 00:06:41 Anyway, us engineers, as you all know, like to work smarter, not harder.

• 00:06:45 So instead of trolling through this entire document, I can pull up Gemini to help.

• 00:06:51 And again, Gemini anticipates what I need and offers me an ask this PDF option.

• 00:06:57 So if I tap on that, Gemini now ingests all of the rules to become a pickleball expert.

• 00:07:03 And that means I can ask very esoteric questions like, for example, are spin serves allowed?

• 00:07:13 And let's hit that.

• 00:07:14 Because I've heard that rule may be changing.

• 00:07:16 Now, because I'm a Gemini advanced user, this works on any PDF and takes full advantage

• 00:07:21 of the long context window.

• 00:07:23 And there's just lots of times when that's useful.

• 00:07:25 For example, let's say you're looking for a quick answer in an appliance user manual.

• 00:07:30 And there you have it.

• 00:07:32 It turns out, nope, spin serves are not allowed.

• 00:07:35 So Gemini not only gives me a clear answer to my question, it also shows me exactly where

• 00:07:41 in the PDF to learn more.

• 00:07:43 Awesome.

• 00:07:44 Okay.

• 00:07:45 So that's a few of the ways that we're enhancing Gemini to be more context aware and helpful

• 00:07:55 in the moment.

• 00:07:57 And what you've seen here are the first really many new ways that Gemini will unlock new

• 00:08:03 experiences at the system level.

• 00:08:05 And they're only available on Android.

• 00:08:08 You'll see these and more coming to hundreds of millions of devices over the next couple

• 00:08:12 of months.

• 00:08:14 Now, building Google AI directly into the OS elevates the entire smartphone experience.

• 00:08:20 And Android is the first mobile operating system to include a built-in on-device foundation

• 00:08:26 model.

• 00:08:27 This lets us bring Gemini goodness from the data center right into your pocket.

• 00:08:31 So the experience is faster while also protecting your privacy.

• 00:08:36 Starting with Pixel later this year, we'll be expanding what's possible with our latest

• 00:08:40 model, Gemini Nano with multimodality.

• 00:08:44 This means your phone can understand the world the way you understand it.

• 00:08:48 So not just through text input, but also through sights, sounds, and spoken language.

• 00:08:54 Let me give you an example.

• 00:08:56 2.2 billion people experience blindness or low vision.

• 00:09:00 So several years ago, we developed TalkBack, an accessibility feature that helps people

• 00:09:06 navigate their phone through touch and spoken feedback.

• 00:09:10 Helping with images is especially important.

• 00:09:12 In fact, my colleague Caro, who uses TalkBack, will typically come across 90 unlabeled images

• 00:09:18 per day.

• 00:09:20 Thankfully, TalkBack makes them accessible.

• 00:09:23 And now we're taking that to the next level with the multimodal capabilities of Gemini

• 00:09:27 Nano.

• 00:09:28 So when someone sends Caro a photo, she'll get a richer and clearer description of what's

• 00:09:33 happening.

• 00:09:34 Or let's say Caro is shopping online for an outfit.

• 00:09:37 Now she can get a crystal clear description of the style and cut to find the perfect look.

• 00:09:43 Running Gemini Nano on device helps minimize the latency.

• 00:09:47 And the model even works when there's no network connection.

• 00:09:51 These improvements to TalkBack are coming later this year.

• 00:09:55 Let me show you another example of what on-device AI can unlock.

• 00:09:59 People lost more than $1 trillion to fraud last year.

• 00:10:03 And as scams continue to evolve across texts, phone calls, and even videos, Android can

• 00:10:09 help protect you from the bad guys no matter how they try to reach you.

• 00:10:13 So let's say I get rudely interrupted by an unknown caller right in the middle of my

• 00:10:19 presentation.

• 00:10:22 Hello?

• 00:10:24 Hi, I'm calling from SafeMore Bank Security Department.

• 00:10:26 Am I speaking to Dave?

• 00:10:28 Yeah, this is Dave.

• 00:10:29 Kind of in the middle of something.

• 00:10:31 We've detected some suspicious activity on your account.

• 00:10:33 It appears someone is trying to make unauthorized charges.

• 00:10:37 Oh, yeah.

• 00:10:38 What kind of charges?

• 00:10:40 I can't give you specifics over the phone.

• 00:10:42 But to protect your account, I'm going to help you transfer your money to a secure account

• 00:10:46 we've set up for you.

• 00:10:50 And look at this.

• 00:10:52 My phone gives me a warning that this call might be a scam.

• 00:11:02 Gemini Nano alerts me the second it detects suspicious activity, like a bank asking me

• 00:11:08 to move my money to keep it safe.

• 00:11:10 And everything happens right on my phone.

• 00:11:12 So the audio processing stays completely private to me and on my device.

• 00:11:17 We're currently testing this feature, and we'll have more updates to share later this

• 00:11:20 summer.

• 00:11:22 And we're really just scratching the surface of the kinds of fast, private experiences

• 00:11:27 that on-device AI unlocks.

• 00:11:29 Later this year, Gemini will be able to more deeply understand the content of the screen

• 00:11:35 without any information leaving your phone, thanks to the on-device model.

• 00:11:40 So remember that pickleball example earlier?

• 00:11:43 Gemini on Android will be able to automatically understand the conversation and provide relevant

• 00:11:48 suggestions, like where to find pickleball clubs near me.

• 00:11:52 And this is a powerful concept that will work across many apps on your phone.

• 00:11:57 In fact, later today at the developer keynote, you'll hear about how we're empowering our

• 00:12:02 developer community with our latest AI models and tools, like Gemini Nano and Gemini in

• 00:12:08 Android Studio.

• 00:12:09 Also, stay tuned tomorrow for our upcoming Android 15 updates, which we can't wait to

• 00:12:14 share with you.

• 00:12:17 As we said at the outset, we're reimagining Android with Gemini at the core.

• 00:12:22 From your favorite apps to the OS itself, we're bringing the power of AI to every aspect

• 00:12:28 of the smartphone experience.

• 00:12:30 And with that, let me hand over to Josh to share more on our use for developers.

• 00:12:35 Thank you.