third thing, this premise that bot tutors are unlikely to be as good as the best instructors.18:29We had a few colleagues at Harvard who tested this for a course called Physical Sciences 2. This is one of the most popular courses, and by the way, the instructors are very good in that course. They've been refining active learning, teaching methods for many years.18:43What they did as an experiment was, say for half the students every week, we'll give them access to the human tutors for the other half, give them access to an AI bot. And by the way, the nice thing about the experiment is they flip that every single week.18:58So some people always had access to the humans. some people had access to the AI for that week, But then they'd flipped the next week, every single week, they tested your mastery of the content during that week. And what was interesting was the scores of the students using the AI bots were higher than with the human tutors.19:20And these are tutors who've been refining their craft year in and year out. What was even more surprising is engagement was higher by the way, this is a first experiment. The only point is we better take this seriously. Uh, next, will it level the playing field in education part of the premises because everyone has access any individual in a village, a low-income area is basically going to have access to the same technology as those who are in elite universities.19:52And this is going to level everything. there's a possibility it might go exactly the other way, which is the benefits might ACR disproportionately to those who already have domain expertise. Why do I say this? Think about a simple example when you have knowledge of a subject and you start using generative AI or chat.20:14pt the way you interact with it, asking it prompts, follow on prompts, you're basically using your judgment to filter out what's useful and what's not useful. If I didn't know anything about the subject, I basically don't know what I don't know.20:29So in some sense, the prompts are garbage in, garbage out. By the way, this is being shown in different studies. There was a metaanalysis summarized by the economist a couple of weeks ago, where they basically talk about different kinds of studies that are showing for certain domains and expertise.20:46The gap between high performance high knowledge workers and no knowledge workers is actually increasing. We better take this seriously, Why and this is not the first time this has happened 12 years ago, there was a big revolution in online education.21:03Harvard and MIT got together created a platform called EDEX, where we offered free online courses to anyone in the world. By the way, they still exist if you want to take a course from Harvard for free, pay $100 for a certificate, you can get it on virtually every subject.21:21What happened As a result EDEX reached 35 million learners as did Corera and Udacity and other platforms. What was beautiful is roughly free 3,000 courses. The challenge was completion rates less than 5%. Why? By the way if you're used to a boring lecture in the classroom, the boring lecture online is 10 times worse.21:43So there's virtually no engagement People take a long time to complete or may not complete. But here's what's interesting. The vast majority 75% of those who actually completed these courses already had college degrees meaning the educated rich were getting richer.22:01Now think about that. That's very sobering. Why is that? Because those are people used to curiosity. Intrinsic motivation by the way they're used to boring lectures. They've gone to college but this has big implications for how we think about the digital divide.22:16So I just want to keep that in your mind. And the last thing I just want to say is rather than going out and trying to create tutor bots for as many courses as possible, I think what we really need to do is have a strategic conversation about what's the role and purpose of teachers, given the way the technology is proceeding.22:35The one thing I will say here is that when we think about what we learned in school, okay, think back, think back many, many years, we learned many things. Tell me, honestly, how many of you have used geometry proofs since you graduated from high school.22:56Three people, why did we learn state capitals and world capitals of every single country? Okay. Uh, foreign languages. And by the way, this is Italian. Davy is not a goddess. Davy in Italian says you must. Okay, they have similarities.23:16um, why did we learn foreign languages when we think about business concepts in our curriculum? I often get my students who come back 10 years later and say those two years were the most transformative years of my life life.23:28I often asked them, what were the three most important concepts you learned? They said, we have no idea. I'm like, no, no. Okay, Give me one. no, no, we have no idea. I'm like. So why do you say this was transformative? The point simply being they're saying this was transformative, Not because of the particular content, but because of the way we were learning, we were forced to make decisions in real time.23:50We were listening to others, we were communicating, what are they saying? they're saying that the real purpose of case method was listening and communication. The real purpose of proofs was understanding logic. The real purpose of memorizing state capitals was refining your memory.24:07By the way, that example, there is the poem If by Rard Kipling, some of you might remember this from school. it goes something like this. If you can keep your head when all about you are losing theirs and blaming it on you.24:18Uh, I have PTSd because. my nephew, when he was reciting this, to me, preparing for his 10th grade exams, I was like, what the heck are you doing? But it was basically refining memory skills. And for, for languages, it was just learning cultures and syntax.24:30When we go deep down and think about what we were actually teaching, I think that probably gives us a little more hope because it means it doesn't matter if some of these things are probably accessible through gen, When calculators came along we thought it's going to destroy math skills.24:47we're still teaching math thankfully 50 years later and it's pretty good. So this is something that I think is going to be an important strategic conversation. This is the slide I'd love for you to keep in mind which is basically everything i've just said If you want to take a screenshot this is the slide to take a screenshot.25:04Thank you all so much. Um and I hope to be in touch and keep this here at HBS. I took Professor Anan's class on economics for managers listening to him feels like being back in class. fortunately he didn't call call anyone which is terrific.25:21So thank you for that Now I have a few questions. We've got young children and you've got so much of knowledge available now on chat prompts. What's your advice to everyone who's got young children and are wondering about what should they be teaching their children so that when they grow up and when we don't know what the actual capabilities of these machines are that what they've learned is still useful.25:44How old are your kids? Rul so my son is nine and my daughter is five. What are you telling them right now? Now I want to learn from you and I know we telling them a lot of stuff with a good, bad ugly.25:55I don't I'm trying to refine that and give them a framework of what we should be telling them. So there's two things so I think first of all this is probably one of the most common questions I get, uh by the way it's really interesting that the tech experts and there was an article in the Wall Street Journal about this 10 days ago, are basically telling their kids don't learn computer science that skill at least basic computer programming is gone.26:20advanced computer science, advanced data analysis. If you want to do that, that's going to be fine. What are they telling their kids to learn? They're telling their kids to learn how to teach dance, they're telling their kids to learn how to do plumbing.26:34they're telling their kids to learn about the humanities. Why are they saying that implicitly, they're saying, what are those skill sets that are robust to machine intelligence? Now I will say it is virtually impossible to predict that given the pace at which this improvement is occurring, I probably have a slightly different kind of answer by the way, my daughter's majoring in psychology without me telling her anything.26:60So the kids I think, know, basically where this is going But the one thing I'll say Rahul is I don't know when you started out college what were you measuring in journalism? Journalism. You started out with journalism. Okay that's enlightened.27:12I started out doing chemistry and then the reason I switched to economics was probably like many of you. There was one teacher who inspired me and that's what made me switch. And I would say to kids follow the teachers who inspire you.27:30And the reason is if you can get inspired and passionate about a subject that's going to build something that's going to be a skill that would last all your life which is curiosity which is intrinsic motivation. We talked about in the last session.27:46This is no longer about learning episodically, it's about learning lifelong and that's I think going to be the most important ST in the way that Indian families operate. And as do so many Asian families too. Parents want to equip their children with the skills that are likely to be most useful when they grow up.28:02So it used to be, say, engineering and doctors back in the day, then, uh, it a few years ago. So if you were looking ahead, what do you think the children should be learning? So they acquire skills which are useful in the job market.28:17years down. Yeah, I think that's honestly being too instrumental. As I said, 10 years ago, a lot of my students were talking to me and saying, what should I major in? I never told them computer science. if I told them that that I would have regretted it.28:30But I genuinely mean this. that's looking at it things too narrowly. What I would say is think about things like creativity, judgment, human emotion, empathy, psychology, those are things that are going to be fundamentally important regardless of where computers are going, by the way, you can get those skills through various subjects.28:48It doesn't matter. It's not a one-o--one mapping between those skills and a particular topic or disciplinary era. This is partly what I'm saying. really think about where their passion is how do we teach our children how to think? Because everything's available on Google Co--pilot chat GPT, you can just chat GPT it So joining the dots, giving them a framework to be able to interpret, analyze and think, how do you tell them that when the easiest thing is go? Yeah.29:16So uh, it's a good question, just two things on that. The first is there was an interesting study done by colleagues at MIT recently, where they had groups of students and they were asked to undertake a particular task or learn about a topic.29:32Some students were given AI chat bots. Some students were only given Google search with no AI. What they found is the students with access to AI intelligence, learned the material much faster. But when it came time to apply it on a separate test, which was different from the first one, they found it much harder.29:55The students who learned the material through Google search with no other access took longer, but they did much better on those tests. Why is that part of the issue is? learning is not simple. It takes effort. Okay. And so part of the issue is, you can't compress that effort.30:14Um, the harder it is to learn something, the more likely you'll remember it for longer periods of time. And so I think for me, the big implication is when I tell my students, look, all these technologies, a are available, It depends on how you use it.30:34My basic approach to them is just saying study, because if you get domain expertise, you will be able to use these tools in a much more powerful way later on.