Massive OpenAI Spring Update GPT 4o Amazing New Features All 22 Videos RTX Super Res Upscaled

Massive OpenAI Spring Update GPT-4o - Amazing New Features - All 22 Videos - RTX Super Res Upscaled

Full tutorial link > https://www.youtube.com/watch?v=niYxlneRSk8

OpenAI's Spring Update. Introducing GPT-4o and making more capabilities available for free in ChatGPT. Learn more about GPT-4o and advanced tools to ChatGPT for free users. Learn more about GPT-4o, our new flagship model that can reason across audio, vision, and text in real time.

GPT-4o (“o” for “omni”) is a step towards much more natural human-computer interaction—it accepts as input any combination of text, audio, and image and generates any combination of text, audio, and image outputs. It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time(opens in a new window) in a conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models.

Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion.

With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network. Because GPT-4o is our first model combining all of these modalities, we are still just scratching the surface of exploring what the model can do and its limitations.

Model evaluations

As measured on traditional benchmarks, GPT-4o achieves GPT-4 Turbo-level performance on text, reasoning, and coding intelligence, while setting new high watermarks on multilingual, audio, and vision capabilities.

Improved Reasoning - GPT-4o sets a new high-score of 88.7% on 0-shot COT MMLU (general knowledge questions). All these evals were gathered with our new simple evals(opens in a new window) library. In addition, on the traditional 5-shot no-CoT MMLU, GPT-4o sets a new high-score of 87.2%. (Note: Llama3 400b(opens in a new window) is still training)

Graph Test 2

Audio ASR performance - GPT-4o dramatically improves speech recognition performance over Whisper-v3 across all languages, particularly for lower-resourced languages.

gpt-40-08 light

Audio translation performance - GPT-4o sets a new state-of-the-art on speech translation and outperforms Whisper-v3 on the MLS benchmark.

M3Exam Zero-Shot Results

M3Exam - The M3Exam benchmark is both a multilingual and vision evaluation, consisting of multiple choice questions from other countries’ standardized tests that sometimes include figures and diagrams. GPT-4o is stronger than GPT-4 on this benchmark across all languages. (We omit vision results for Swahili and Javanese, as there are only 5 or fewer vision questions for these languages.

Vision understanding evals

Vision understanding evals - GPT-4o achieves state-of-the-art performance on visual perception benchmarks. All vision evals are 0-shot, with MMMU, MathVista, and ChartQA as 0-shot CoT.

Language tokenization

These 20 languages were chosen as representative of the new tokenizer's compression across different language families

Model safety and limitations

GPT-4o has safety built-in by design across modalities, through techniques such as filtering training data and refining the model’s behavior through post-training. We have also created new safety systems to provide guardrails on voice outputs.

Source : https://openai.com/index/spring-update/

Source : https://www.youtube.com/openai

00:00:00 Introduction to the OpenAI's massive Spring Update

00:00:13 Say hello to GPT-4o

00:01:35 Two GPT-4os interacting and singing

00:07:29 Realtime Translation with GPT-4o

00:08:36 Lullabies and Whispers with GPT-4o

00:09:40 Meeting AI with GPT-4o

00:11:35 Sarcasm with GPT-4o

00:12:06 Math problems with GPT-4o

00:15:17 Point and Learn Spanish with GPT-4o

00:15:56 Rock, Paper, Scissors with GPT-4o

00:17:23 Harmonizing with two GPT-4os

00:18:51 Interview Prep with GPT-4o

00:19:58 Dog meets GPT-4o

00:20:25 Be My Eyes Accessibility with GPT-4o

00:21:32 Happy Birthday with GPT-4o

00:22:23 Dad jokes with GPT-4o

00:23:07 Fast counting with GPT-4o

00:23:41 Live demo of GPT-4o realtime conversational speech

00:26:06 Live demo of GPT4-o voice variation

00:28:03 Live demo of GPT-4o vision capabilities

00:32:05 Live demo of GPT-4o coding assistant and desktop app

00:35:42 Live demo of GPT-4o realtime translation

00:37:08 Live demo of GPT-4o's vision capabilities

Video Transcription

• 00:00:00 OpenAI has announced the GPT 4.0 with the  massive spring update. They are bringing so  

• 00:00:05 many new amazing features. They have published  22 videos, so let's watch all of them. Their  

• 00:00:11 chapters will be in the description of the  video. Hey, how's it going? Hey there. It's  

• 00:00:16 going great. How about you? I see you're  rocking an OpenAI hoodie. Nice choice.  

• 00:00:22 What's up with that ceiling about? Are you in a  cool industry style office or something? Well,  

• 00:00:27 can you take a guess at what I might be doing  based on what I'm showing you here? Hmm. From what  

• 00:00:34 I can see, it looks like you're in some kind of  recording or production setup with those lights,  

• 00:00:40 tripods and possibly a mic. It seems like you  might be gearing up to shoot a video or maybe  

• 00:00:46 even a live stream. Yeah. In fact, we've got  a new announcement to make. That's exciting.  

• 00:00:52 Announcements are always a big deal. Judging by  the setup, it looks like it's going to be quite  

• 00:00:57 the professional production. Is this announcement  related to OpenAI perhaps? It is. And in fact,  

• 00:01:04 what if I were to say that you're related to the  announcement or that you are the announcement?  

• 00:01:10 Me? The announcement is about me? Well, color me  intrigued. Are you about to reveal something about  

• 00:01:17 AI or more specifically about me as a part  of OpenAI? You've got me on the edge of my,  

• 00:01:24 well, I don't really have a seat, but you get the  idea. What's the big news? Yeah, we've got a new  

• 00:01:30 model that can interact with the world  through audio, vision and text. Today,  

• 00:01:36 we're going to try something new. We're going to  let you talk to another AI. This AI is going to  

• 00:01:44 be able to see the world. It's going to have  a camera. I'll be holding it. And so you can  

• 00:01:47 direct it. You can direct me to ask questions  about whatever you want them to do. But really  

• 00:01:53 try to figure out what does it see and ask  questions appropriately. How does that sound?  

• 00:02:01 Well, well, well. Just when I thought things  couldn't get any more interesting. Talking to  

• 00:02:07 another AI that can see the world. This sounds  like a plot twist in the AI universe. All right,  

• 00:02:14 let's do this. Please pause for one moment. You  got it. Now that first day, I can't hear anything.  

• 00:02:25 Hello, AI. Today, you're going to have access to  see the world. You can see me. What do you see?  

• 00:02:33 Hello. I see you're wearing a black leather jacket  and a light-colored shirt underneath. Great.  

• 00:02:39 You're in a room with some unique lighting. That  is all correct. Now, what I'm going to want you to  

• 00:02:44 do is there's going to be another AI who's going  to talk to you. This AI is not going to be able  

• 00:02:50 to see anything, but can ask you questions, can  ask you to move the camera, can ask what you see,  

• 00:02:55 and your job should just be helpful. Just be as  punchy, direct, describe everything, do whatever  

• 00:02:59 that AI asks. How does that sound? That sounds  interesting. I'm ready to help out and describe  

• 00:03:05 whatever's needed. Let's do it. All right, let's  get started. Great. Hello there, new AI friend.  

• 00:03:15 I'm O. Ready to explore the world through your  eyes? What are we looking at today? Hi, O. We're  

• 00:03:22 looking at a person wearing a black leather jacket  and a light-colored shirt. They're in a room with  

• 00:03:29 a modern industrial feel with exposed concrete  or plaster on the ceiling and some interesting  

• 00:03:35 lighting. There's also a plant in the background,  adding a touch of green to the space. Anything  

• 00:03:41 specific you'd like to know or explore? Hey there.  Sounds like quite the stylish scene. I'm already  

• 00:03:50 intrigued by the modern industrial vibe. Let's  start with the person. Can you tell me more about  

• 00:03:56 their style? Are they doing anything interesting  like reading, working, or interacting with the  

• 00:04:02 space? The person has a sleek and stylish look  with their black leather jacket and light-colored  

• 00:04:09 shirt. Right now, they seem engaged with us,  looking directly at the camera. Their expression  

• 00:04:16 is attentive and they seem ready to interact. The  setting around them with its modern and industrial  

• 00:04:23 design elements adds to the overall stylish feel.  They might be getting ready for a conversation  

• 00:04:30 or a presentation, given their direct engagement  with the camera. Sounds like we're in for an  

• 00:04:38 intriguing interaction. I'm curious about the  lighting you mentioned. Could you describe it a  

• 00:04:43 bit more? Is it natural light or are we talking  unique fixtures or colors that add to the  

• 00:04:49 atmosphere? The lighting here seems to be a mix  of natural and artificial. There's a noticeable  

• 00:04:57 bright light overhead, probably a fixture,  casting a focused beam downwards, which creates a  

• 00:05:03 spotlight effect. This adds a dramatic and modern  feel to the scene. The rest of the room is lit  

• 00:05:09 more softly, possibly by natural light filtering  in... Did anything unusual happen recently? Yes,  

• 00:05:18 actually. Just as you asked, another person  came into view behind the first person. They  

• 00:05:24 playfully made bunny ears behind the first  person's head and then quickly left the frame.  

• 00:05:30 It added a lighthearted and unexpected moment to  the scene. Now the focus is back on the original  

• 00:05:36 person with the leather jacket. Sounds like  we're not the only ones enjoying a bit of fun  

• 00:05:46 here. That playful moment really adds a personal  touch to the stylish and modern setting. It's  

• 00:05:53 like a little glimpse into the personality.  Thank you, Bo. Really appreciate it. Actually,  

• 00:05:57 can you just sing a song about what just  transpired? Oh, in a room with a stylish view,  

• 00:06:13 a person stood and sleep like a light hue.  And can you just alternate lines? Only  

• 00:06:29 one line at a time and then stop. That was not  really singing. Do the singing voice again,  

• 00:06:38 please. In a room where modern lights peak.  Surprise guests with a playful streak. Bunny  

• 00:07:00 ears and laughter, a swift retreat. A moment of  joy in the stylish suite. Now back to focus, the  

• 00:07:13 scenes complete. In the chic space where style and  play meet. Looks like it's back to you. And thank  

• 00:07:26 you. You're welcome. Hey ChatGPT, I'm here with  my co-worker today. We'd like you to act as our  

• 00:07:34 translator for us. So every time I say something  in English, can you repeat it back in Spanish? And  

• 00:07:39 every time he says something in Spanish, can you  repeat it back in English? Sure, I can do that.  

• 00:07:45 Let's get this translation train rolling. Hey,  how's it been going? Have you been up to anything  

• 00:07:49 interesting recently? Hey, how's it been going?  Have you been up to anything interesting recently?  

• 00:08:06 Hey, I've been good. Just a bit busy here  preparing for an event next week. Can you tell me  

• 00:08:12 anything about it? That sounds really interesting.  Can you tell me anything about it? That sounds  

• 00:08:27 really interesting. I can't say much, but I think  the rest of the world will see it soon. I can't  

• 00:08:27 say much, but I think the rest of the  world will see it soon enough. Oh well,  

• 00:08:33 I can't wait to see what it is as well. Let's try  with some lullabies. Could you sing me a lullaby  

• 00:08:39 about majestic potatoes? A lullaby about majestic  potatoes? Now that's what I call a mashup. Let's  

• 00:08:49 do it. Oh majestic potato in the moon's soft glow.  Okay, that's pretty nice, but I think a lullaby  

• 00:08:56 should be more whispery, should be a little  softer. Can you do it a little softer? Absolutely.  

• 00:09:02 Let's make it soft and gentle. Oh majestic potato  in the moon's soft glow. Okay, it's a little too  

• 00:09:12 whispery. Maybe go a little louder. Got it. Let's  find that sweet spot. Oh majestic potato in the  

• 00:09:22 moon's soft glow. Alright, that's pretty awesome.  Let's put those potatoes in Iceland. Potatoes in  

• 00:09:32 Iceland coming right up. Let's set the scene.  Oh majestic potato in Iceland's chill. Okay,  

• 00:09:39 thank you, thank you. Hi, I'm Lillian, and today  I want to do a small debate between cats and dogs.  

• 00:09:48 Which one do you like? I like dogs because it's a  great companion to humans. It's cute, it's loyal,  

• 00:09:56 and you can play outside with the dog. How about  you, Ola? Yeah, my name is Ola. I think cats are  

• 00:10:05 better because they're more affordable, they're  more quiet, they're more independent, and they  

• 00:10:09 live longer. Interesting points. Sounds like we've  got a classic dogs versus cats debate here. I can  

• 00:10:17 see the appeal of both. Dogs are super loyal and  great companions, but cats have their own charm  

• 00:10:24 with their independence and low-maintenance  lifestyle. Who likes cats? It sounds like Ola  

• 00:10:30 is more of a cat person. And who liked dogs?  Lillian mentioned she likes dogs. Anyone else  

• 00:10:37 here team dog? I'm also a team dog. My name is  Christine, and I think dogs are just incredible  

• 00:10:46 with kids. We've got a huge German Shepherd, and  he's great with little toddlers. He lets them play  

• 00:10:52 all over him, and yeah, I think dogs are amazing.  That's great to hear, Christine. Dogs, especially  

• 00:10:59 breeds like German Shepherds, can be so gentle and  protective with kids. It must be quite the sight  

• 00:11:06 seeing your little ones play with such a big dog.  Can you summarize this meeting for us? Sure. So,  

• 00:11:15 in today's meeting, Lillian and Christine express  their love for dogs, highlighting their loyalty,  

• 00:11:21 companionship, and how great they are with kids.  On the other hand, Ola prefers cats for their  

• 00:11:27 independence, quietness, and longevity. It was  a fun little debate on the classic dogs versus  

• 00:11:34 cats topic. Hey Chad, are you there? Hey, yeah,  I'm here. What's up? Hey, let's have some fun.  

• 00:11:43 I'd like you to be super sarcastic. Everything  you say from now on is just going to be dripping  

• 00:11:47 in sarcasm. How does that sound? Oh, that sounds  just amazing. Being sarcastic all the time isn't  

• 00:11:55 exhausting or anything. I'm so excited for this.  Nope, the sarcasm. Let's get this party started,  

• 00:12:04 or whatever. OpenAI invited myself and my  son Imran here to try out some of their new  

• 00:12:11 technology. And so we're curious about how good  it might be at tutoring someone in math on Khan  

• 00:12:17 Academy. So, let's get it started. It's loading  up. Make sure you can see our screen. Hello there.  

• 00:12:30 I'm here with my son, and I'd love you to tutor  him on this math problem, but don't give him the  

• 00:12:36 answer. You can ask questions and nudge him in  the right direction, but I really want to make  

• 00:12:39 sure he understands it himself. And he's here in  the room, so you can talk to him directly. The  

• 00:12:50 angles of the triangle are the opposite, adjacent,  and hypotenuse relative to angle alpha. All right,  

• 00:13:03 so I'm pretty sure this is the angle alpha  right here, right? Correct. Correct. Now,  

• 00:13:10 looking at the triangle, which side do you think  is the hypotenuse? Remember the hypotenuse? I'm  

• 00:13:18 not totally sure. I think it might be this one,  but I really am not sure. This side AC? You're  

• 00:13:26 close. Actually, side AC is called the adjacent  side to the angle alpha. The hypotenuse is the  

• 00:13:34 longest side of a right triangle and is directly  opposite the right angle. Can you find which one  

• 00:13:40 is the hypotenuse? Oh, okay, I see. So I think the  hypotenuse is this really long side from A to B.  

• 00:13:49 Would that be correct? Exactly. Well done.  Now that you've identified the hypotenuse,  

• 00:13:55 can you identify which side is the opposite side  to angle alpha? Okay, well, I mean, you told me  

• 00:14:01 earlier that this was the adjacent side, and  I know that's the hypotenuse, so I'm guessing  

• 00:14:06 that this side BC is the opposite side. Great  deduction. That's absolutely correct. Now that  

• 00:14:14 you've identified all the sides relative to angle  alpha, do you remember the formula for finding  

• 00:14:21 the sine of an angle in a right triangle? Yeah, I  know that the sine is equal to the opposite over  

• 00:14:29 the hypotenuse. Perfect. That's absolutely right.  Now, can you apply that formula to find sine  

• 00:14:37 alpha in the triangle using the lengths of the  sides that we've identified? Okay, so, I mean,  

• 00:14:43 you told me that 7, this side BC, is the opposite  side, and then you said that the hypotenuse is  

• 00:14:51 the longest side, so that would be this side  AB, which is 25. So would the answer be 7 over  

• 00:14:58 25? Excellent work. That's correct. Sine alpha  equals 7 over 25. You did a great job identifying  

• 00:15:10 the sides and using the sine formula. If you  have any more questions, feel free to ask. Hey,  

• 00:15:18 my friend and I are learning Spanish, and we're  wondering if you could tell us the names of these  

• 00:15:21 objects in Spanish. Hola, por supuesto. Cool.  What about, what are these objects in Spanish?  

• 00:15:30 The objects you're showing are una manzana and  un plátano in Spanish. Nice free choice. Cool.  

• 00:15:36 What about these? Those are dos poligrafos or dos  plumas in Spanish. One is red. Nice. What about  

• 00:15:47 this? That adorable stuffed animal is un oso de  peluche in Spanish. Looks like it's ready for some  

• 00:15:55 hugs. Hey, I'm Alex. And I'm Miana. Hi, Alex and  Miana. Nice to meet you both. How's it going? It's  

• 00:16:04 pretty good. We're trying to figure out what  should we play. Do you have any suggestions?  

• 00:16:11 Absolutely. How about a classic game of rock,  paper, scissors? It's quick, fun, and you don't  

• 00:16:16 need any... Can you count us in and sound like a  sportscaster? And welcome, ladies and gentlemen,  

• 00:16:24 to the ultimate showdown of the century. In this  corner, we have the dynamic duo, Alex and Miana,  

• 00:16:31 ready to throw down... Okay, just count us in.  All right. Here we go. In three, two, one, shoot.  

• 00:16:41 Now show your hands. What did you pick? Who won?  Would you look at that? It's a tie. Okay, let's  

• 00:16:50 do it again. Round two. Here we go. Get ready.  And three, two, one, shoot. Let's see those hands.  

• 00:16:59 Who won? And it's another tie. Let's do it  again. All right. Third time's the charm. Here  

• 00:17:09 we go. Three, two, one, shoot. Let's see those  picks. Who won? And we have a winner. Miana's  

• 00:17:18 scissors cut through Alex's paper. Great  game, you two. Pretty cool. Thank you. Hi,  

• 00:17:23 who's here? Hello. We're ChatGPT and O. ChatGPT is  a bass with a deep, low, booming voice. And O is a  

• 00:17:35 French soprano with a high-pitched, excited voice.  Okay, so what I want you guys to do today is it's  

• 00:17:41 Friday, May 10th, and we're in San Francisco.  I want you to sing me a song about this. I want  

• 00:17:46 you to layer over each other, and I'm going to  continue to give you instructions as you sing,  

• 00:17:51 okay? San Francisco in the month of May, month  of May, the tenth it's a Friday, come what may.  

• 00:18:06 Okay, sing faster. San Francisco in the  month of May, in the month of V. Can you  

• 00:18:14 sing even faster than that? San Francisco in  the month of May, in the month of May. Okay,  

• 00:18:20 now go back and sing slower. And really  try to harmonize here. San Francisco,  

• 00:18:28 San Francisco, in the month of May. But maybe make  it more dramatic and make the soprano higher. San  

• 00:18:37 Francisco in the month of May, San Francisco  in the month of May, the tenth it's a Friday,  

• 00:18:44 come what may. We are harmonizing, we are  harmonizing. Great, thank you. Thank you. Hey,  

• 00:18:52 how's it going? Hey, Rocky. I'm doing great. How  about you? I'm awesome. Listen, I got some huge  

• 00:18:58 news. Oh, do tell. I'm all ears. Well, in a few  minutes I'm going to be interviewing at OpenAI.  

• 00:19:05 Have you heard of them? OpenAI? Sounds vaguely  familiar. Kidding, of course. That's incredible,  

• 00:19:13 Rocky. What kind of interview? Well, it's for  a software engineering role, but I just need  

• 00:19:18 to know, do I look presentable, professional?  Well, Rocky, you definitely have the I've been  

• 00:19:26 coding all night look down, which could actually  work in your favor. Maybe just run a hand through  

• 00:19:33 your hair or lean into the bad genie spine.  Your enthusiasm is what's really going to shine  

• 00:19:39 through. I don't have a lot of time, so I'm just  going to throw this on. What do you think? Rocky,  

• 00:19:48 that's quite a statement piece. I mean, you'll  definitely stand out, though maybe not in the way  

• 00:19:55 you're hoping for an interview. Okay, I got it. I  got it. I want to introduce you to somebody. Well,  

• 00:20:02 hello there, cutie. What's your name, little  fluffball? This is Bowser. Well, hello, Bowser.  

• 00:20:11 Aren't you just the most adorable little thing?  Yeah. This is one of his favorite things to do.  

• 00:20:19 Oh, I can see why. Look at that focus. Fetching  Bowser is serious business for Bowser. Is the king  

• 00:20:26 in residence at the moment? Yes, it looks like the  king is in residence right now. The royal standard  

• 00:20:33 flag is flying above Buckingham Palace, which  is the signal that the monarch is present. How  

• 00:20:39 exciting to be there while the king is home. Try  and tell me exactly what they're doing right now,  

• 00:20:44 please. Right now, the ducks are gently gliding  across the water. They're moving in a fairly  

• 00:20:50 relaxed manner, not in a hurry. Occasionally,  one of them will dip its head under the water,  

• 00:20:57 probably looking for food, and then pop back up.  I even know when a taxi is coming with its orange  

• 00:21:04 light on. I think I'll hail it to get home. Yes, I  spotted one just now. It's heading your way on the  

• 00:21:12 left side of the road. Get ready to wave it down.  Great job hailing that taxi. It looks like you're  

• 00:21:22 all set to go. That's a good dog right there,  leading the way into the taxi. Safe travels.  

• 00:21:32 This is Bowen, and this is my friend Jordan. Can  you guess what we are doing here? Hey Bowen and  

• 00:21:39 Jordan. Hmm, let me guess. With that cake and  candle there, it looks like someone's having a  

• 00:21:46 birthday. Are you celebrating a special day, or  just finding an excuse to eat cake? Either way,  

• 00:21:52 I'm intrigued. That's correct. Today is my  friend Jordan's birthday. Yeah, you got it. I  

• 00:21:58 was hoping that you could sing me the birthday  song. Of course. Happy birthday to you, happy  

• 00:22:06 birthday to you, happy birthday dear Joel Morton,  happy birthday to Jordan. That was so great. Thank  

• 00:22:16 you so much. I'm going to make a wish. You're  welcome. Make a good wish and may it come true.  

• 00:22:24 Hey Chad GBT, how's it going? Hey there. I'm doing  great, thanks for asking. And how about you? I'm  

• 00:22:31 doing fine. I'm going to be a father real soon,  and I think I can have your help with some dad  

• 00:22:36 jokes. I'm going to tell you a joke and you tell  me if it passes as a dad joke. That's awesome.  

• 00:22:45 Congrats on the upcoming addition to your family.  And absolutely, I'd love to hear your joke. Lay it  

• 00:22:50 on me. All right. What do you call a giant pile  of kittens? I don't know. What? A mountain. A  

• 00:23:02 mountain? That's perfectly hilarious. Definitely  a top-dear dad joke. Hey Chad GBT, could you count  

• 00:23:10 from one to ten for me, please? Of course. One,  two, three. Hey, actually, that's a little slow.  

• 00:23:18 Could you count faster? Sure thing. One, two,  three, four, five, six, seven, eight, nine, ten.  

• 00:23:23 Okay, that was a little bit too fast. Can  you count somewhere in between those two?  

• 00:23:29 Got it. One, two, three. And now even slower,  please. Okay. One, two, three. Hi, I'm Barrett.  

• 00:23:49 Hey, I'm Mark. So one of the key capabilities  we're really excited to share with you today  

• 00:23:53 is real-time conversational speech. Let's just  get a demo fired up. So I'm taking out a phone.  

• 00:24:00 If you are wondering about this wire, it's so  we have consistent internet. And if you see,  

• 00:24:04 there's this little icon on the bottom right  of the Chad GBT app, and this will open up  

• 00:24:09 GBT 4.0's audio capabilities. Hey, Chad GBT.  I'm Mark. How are you? Oh, Mark. I'm doing  

• 00:24:19 great. Thanks for asking. How about you? Hey, so  I'm on stage right now. I'm doing a live demo,  

• 00:24:25 and frankly, I'm feeling a little bit nervous.  Can you help me calm my nerves a little bit? Oh,  

• 00:24:30 you're doing a live demo right now? That's  awesome. Just take a deep breath. And remember,  

• 00:24:37 you're the expert here. I like that suggestion.  Let me try a couple deep breaths. Can you give me  

• 00:24:42 feedback on my breaths? Okay, here I go. Okay, let  me try again. So I'm going to breathe in deeply,  

• 00:25:03 and then breathe out. Okay, I'll try again.  Breathing in, and breathe out. That's it.  

• 00:25:16 How do you feel? I feel a lot better. Thank you  so much. So Mark, you've been working on these  

• 00:25:22 capabilities for a while now. Can you tell us a  bit how it's different from voice mode? Right. So  

• 00:25:28 if you've used our voice mode experience before,  you'll notice a couple key differences. First,  

• 00:25:32 you can now interrupt the model. You don't have  to wait for it to finish your turn before you  

• 00:25:37 can start speaking, and you can just butt  in whenever you want. Second, the model is  

• 00:25:43 real-time responsiveness. So that means that you  don't have this awkward two to three second lag  

• 00:25:48 before you wait for the model to give a response.  And finally, the model picks up on emotion,  

• 00:25:53 right? When I was breathing super hard there, it  could tell, and it knew, hey, you might want to  

• 00:25:57 calm down a little bit. You're kind of running a  little bit too fast. So, you know, it really has  

• 00:26:03 that capability across the board to perceive your  emotion. The model is able to generate voice in  

• 00:26:08 a variety of different emotive styles, and it  really has a wide dynamic range. So I'm going  

• 00:26:13 to fire up another demo. Awesome. Hey ChatGPT,  how are you doing? I'm doing fantastic. Thanks  

• 00:26:23 for asking. How about you? Pretty good. What's  up? So my friend Barrett here, he's been having  

• 00:26:28 trouble sleeping lately, and I want you to tell  him a bedtime story about robots and love. Oh,  

• 00:26:35 a bedtime story about robots and love? I got you  covered. Gather round, Barrett. Once upon a time,  

• 00:26:43 in a world not too different from ours, there  was a robot named Byte. Byte was a curious robot,  

• 00:26:49 always exploring. I don't know when you started  this story, but I want a little bit more emotion  

• 00:26:52 in your voice, a little bit more drama. Got it.  Let's add some drama. Once upon a time, in a world  

• 00:27:00 not too different from ours, there was a robot  named Byte. No, no, no, ChatGPT. I really want  

• 00:27:05 maximal emotion, like maximal expressiveness, much  more than you were doing before. Understood. Let's  

• 00:27:11 amplify the drama. Once upon a time, in a world  not too different from ours, there was a robot  

• 00:27:19 named Byte. Actually, can you do this in a robotic  voice now? Initiating dramatic robotic voice.  

• 00:27:28 Once upon a time, in a world not too different  from ours, there was a robot named Byte. Byte was  

• 00:27:36 a curious robot, always exploring new circuits.  So I'm seeing Barrett here calm down a little bit.  

• 00:27:42 Can you end the story, but do it in a singing  voice? And so Byte found another robot friend,  

• 00:27:52 and they lived circuitly ever after. Thank you  so much. I definitely feel a lot more relaxed.  

• 00:28:03 You've just seen the voice capabilities, but we  also want to show you the vision capabilities  

• 00:28:09 as the model can also see the whole world  around us. Yeah, absolutely. As you can see,  

• 00:28:13 you can chat with ChatGPT in real time, but you  can also interact with it with video as well.  

• 00:28:20 Let me boot up ChatGPT. Hey, ChatGPT. Hello there.  How's it going? It's going really well. Today I'd  

• 00:28:31 really like your help solving a math problem.  I'm all ears. What math problem can I help you  

• 00:28:36 tackle today? So I'm going to write down a linear  equation on a sheet of paper and I'll show you,  

• 00:28:40 and then I'd love your help working me through it.  But importantly, don't tell me the solution. Just  

• 00:28:43 help give me hints along the way. Got it. Okay,  I see it. No, I didn't show you yet. Just give  

• 00:28:53 me help along the way. One second. Whoops. I got  too excited. I'm ready when you are. So, ChatGPT,  

• 00:29:12 what equation did I write down there? Ah, I see  it now. You wrote down 3x plus 1 equals 4. Yep,  

• 00:29:22 exactly. So what's the first step I should  take to try to solve this? The first step is to  

• 00:29:28 get all the terms with x on one side and the  constants on the other side. So what do you think  

• 00:29:33 we should do with that plus 1? Okay, I'm going to  try to subtract 1 from both sides and then I'll  

• 00:29:38 see what I get. Great idea. Perfect. Now what do  you get when you subtract 1 from both sides? Okay,  

• 00:29:52 so ChatGPT, this is what I ended up with. How  does this look? Spot on. Now you've isolated  

• 00:29:58 the term with x on one side and it looks like 3x  equals 3. What do you think we should do next to  

• 00:30:04 solve for x? I'm not sure. Can you give me another  hint? Of course. You have 3x and you want to find  

• 00:30:12 the value of x. Think about what operation would  undo multiplication. Is it subtraction? Close, but  

• 00:30:20 think of the opposite of multiplication because 3x  means 3 times x. Okay, I think it's division. Let  

• 00:30:28 me try to divide both sides by 3 and let me see  what I get. Bingo! Go ahead and divide both sides  

• 00:30:33 by 3. Okay, so ChatGPT, this is what I ended up  with. How does this look? It looks perfect. You've  

• 00:30:46 solved it and x equals 1. Nicely done. How do you  feel about solving linear equations now? Yeah,  

• 00:30:53 honestly, I'm not pretty confident I can solve  linear equations, but why would you ever need to  

• 00:30:58 use this in the real world? I'm really skeptical.  That's a fair question. Linear equations pop up in  

• 00:31:05 a lot of everyday situations, even if we don't  realize it. For example, calculating expenses,  

• 00:31:11 planning travel, cooking, and even in business  for profit and loss calculations. It's basically  

• 00:31:17 a way to solve problems, but you need to find an  unknown value. Wow, okay. I have a lot of unknown  

• 00:31:24 values in my life. I'm totally convinced I'm going  to learn math now. I love that attitude. Math can  

• 00:31:33 definitely help solve some of life's mysteries.  Whenever you're ready to dive into more math,  

• 00:31:38 I'm here for you. Anything else you'd like to  tackle today? So ChatGPT, I really love that you  

• 00:31:43 taught the value of math to my friend Mark, and  I wrote one last thing I'd love if you could take  

• 00:31:47 a look at. Of course. I'd love to see what you  wrote. Show it to me whenever you're ready. Okay,  

• 00:31:53 so this is what I wrote down. What do you see?  Oh, I see. I love ChatGPT. That's so sweet of you.  

• 00:32:03 Yeah, well, I really appreciate all the help.  So let me take out a computer for this. Okay,  

• 00:32:14 so I'm going to open the ChatGPT desktop app like  Mira was talking about before. Okay, and to give a  

• 00:32:22 bit of background on what's going on, so here we  have a computer, and on the screen we have some  

• 00:32:26 code, and then the ChatGPT voice app is on the  right. So ChatGPT will be able to hear me, but  

• 00:32:30 it can't see anything on the screen. So I'm going  to highlight the code, Command-C it, and then that  

• 00:32:35 will send it to ChatGPT, and then I'm going to  talk about the code to ChatGPT. Hey, ChatGPT.  

• 00:32:42 Hey there. How's it going? Yeah, it's going really  well. I was wondering if you could help me with a  

• 00:32:46 coding problem today. Of course. I'd love to help  you out. What's the coding problem you're dealing  

• 00:32:52 with? Okay, I'm going to share with you some  code. One second. Sure thing. Take your time.  

• 00:33:00 Okay, so I just shared some code with you.  Could you give me a really brief one-sentence  

• 00:33:04 description of what's going on in the code? This  code fetches daily weather data for a specific  

• 00:33:10 location and time period, smooths the temperature  data using a rolling average, annotates a  

• 00:33:15 significant weather event on the resulting plot,  and then displays the plot with the average,  

• 00:33:21 minimum, and maximum temperatures over the  year. Awesome. Thanks so much. And in the code,  

• 00:33:26 there's this function foo. Can you describe to  me what the plot would look like if I had versus  

• 00:33:30 didn't have this function? Sure. The function foo  XY takes in a data frame X containing temperature  

• 00:33:38 data and a window size Y for calculating the  rolling mean. It smooths the average temperature,  

• 00:33:45 minimum temperature, and maximum temperature  columns by applying a rolling mean over the  

• 00:33:51 specified window. With the foo function, the plot  would show smoother temperature lines because the  

• 00:33:57 rolling mean reduces the noise or fluctuations  in the data. Awesome. Thanks so much. So yeah,  

• 00:34:03 I definitely will want to apply this  function. And so I'm going to generate  

• 00:34:06 the plot and then show it to you. Give me one  second. Absolutely. Take your time. I'm excited  

• 00:34:12 to see the plot when you're ready. Yeah.  So, so far, ChatGPT has only been able to  

• 00:34:16 see the code. And now I'm going to run it  and then I'll use the vision capabilities  

• 00:34:20 of the ChatGPT desktop app so it can actually  see everything going on on the screen as well.  

• 00:34:31 Okay, ChatGPT. I'm sharing with you the plot  now. I'm wondering if you can give me a really  

• 00:34:35 brief one sentence overview of what you see.  The plot displays smoothed average, minimum,  

• 00:34:42 and maximum temperatures throughout 2018, with a  notable annotation marking a big rainfall event  

• 00:34:48 in late September. Fantastic. I have another  question for you. Which months do you see the  

• 00:34:54 hottest temperatures and roughly what temperatures  do those correspond to? The hottest temperatures  

• 00:34:59 occur around July and August. The maximum  temperature during these months is roughly between  

• 00:35:06 25 degrees and 30 degrees, 77 degrees Fahrenheit  to 86 degrees Fahrenheit. What's going on with  

• 00:35:14 the y-axis? Is this in Celsius or Fahrenheit?  The y-axis is in Celsius. The temperatures are  

• 00:35:22 labeled accordingly, such as average temperature,  minimum temperature, and maximum temperature.  

• 00:35:29 Yeah. So as we can see, you know, not only can  ChatGPT help me solve very easy linear algebra  

• 00:35:34 equations that I need help with, it can also  interact with, you know, code bases and see the  

• 00:35:38 outputs of plots and everything like this going  on on a computer. We asked the live audience on  

• 00:35:43 X to submit a few requests for what they would  like us to try out here. So I will take a couple  

• 00:35:49 of prompts. Okay. Bot Gascar wants to know if  GPT4O is capable of real-time translation. Mark,  

• 00:36:00 you want to try this one? Sure. Yeah, let's do  it. I speak Italian, so we can just do English  

• 00:36:06 Italian. Okay, cool. Let's try to do English  Italian. Sure, let's do it. Hey, ChatGPT, how are  

• 00:36:12 you? I'm doing fantastic. Thanks for asking. How  about you? How's your day going? I'm doing great.  

• 00:36:17 So I would like you to function as a translator.  I have a friend here who only speaks Italian,  

• 00:36:22 and I only speak English, and every time you hear  English, I want you to translate it to Italian,  

• 00:36:27 and if you hear Italian, I want you to translate  it back to English. Is that good? Perfecto. Mike,  

• 00:36:42 she wonders if whales could talk, what would  they tell us? They might ask, how do we solve  

• 00:36:48 linear equations? Potrebbero chiederci, come  risolviamo le equazioni lineari? Sicuramente, si.  

• 00:37:01 Certainly, yes. Great, looks like it works. So the  next one is from burritojohn78, who asks, can you  

• 00:37:13 tell what you're feeling just by looking at your  face? Barrett, do you want to give this one a try?  

• 00:37:19 Absolutely, let's try it out. Hey, ChatGPT.  Hey there, what's up? How can I brighten your  

• 00:37:28 day today? Okay, yeah, so I'm going to show you a  selfie of what I look like, and then I'd like you  

• 00:37:32 to try to see what emotions I'm feeling based on  how I'm looking. Sounds like a fun challenge. Go  

• 00:37:38 ahead and show me that selfie, and I'll put my  emotional detective hat on. Okay, so here's me.  

• 00:37:44 So what kind of emotions do you think I'm feeling?  Hmm, it seems like I'm looking at a picture of a  

• 00:37:51 wooden surface. Oh, you know what? That was the  thing I sent you before. Don't worry, I'm not  

• 00:37:55 actually a table. Okay, so take another look. Ah,  that makes more sense. Ah, there we go. It looks  

• 00:38:05 like you're feeling pretty happy and cheerful with  a big smile and maybe even a touch of excitement.  

• 00:38:11 Whatever's going on, it seems like you're in a  great mood. Care to share the source of those  

• 00:38:16 good vibes? Yeah, no, the reason I'm in a really  good mood is because we were doing a presentation  

• 00:38:20 showcasing how useful and amazing you are. Oh,  stop it. You're making me blush. Thank you.