EETI Fall 2023 Distinguished Speaker Forum with Dr. David Joyner: The Roles of ChatGPT in Education

All right, welcome everyone. I'm very excited you're all here today and I'm very excited to have Dr. David Joiner here today to talk to us about the evolving roles of chat, GPT and generative AI in education. Dr. Joyner is the Executive Director, Director of Online Education and the online Master's Program in Computer Science in Georgia Tech's College of Computing and the Zvi Galil Peace Chair. His research focuses on online education and learning at scale, especially as they intersect with four credit offerings at graduate and undergraduate levels. His emphasis is on designing learning experiences that leverage the opportunities of online learning to compensate for the loss of synchronous co located class time. That includes artificial intelligence for student support and assignment evaluation, facilitating student communities in large online classes, and investigating strategies for maintainable and interactive presentation of online instructional material. And with that, please join me in welcoming Dr. David Joiner.

Hello, everybody. See if this works. So, yeah. So thank you all so much for having me. So, like he said, my name is David Joyner. I work at Georgia Tech. I mostly teach computer science classes online and run our online master's program, which has been interesting the past few months because I teach online. I teach AI online. We teach a little bit using AI online. And for a long time that was kind of toiling in obscurity because, you know, we did our little things and no one cared. Then about a year ago, suddenly everyone started to care. And so it's been kind of a whirlwind the past year. Most of the stuff I've been asked to talk to people about in this area, because I've given some very. I should preface this by saying, feel free to ask questions during raise your hand or anything like that. I've given some variation of different portions of this talk in various different kind of places. And I often find the audience and the familiarity and things like that differs enormously from people who have not actually heard of this thing to people who have used it far more than I have and begs the question why I'm the one talking. So feel free to ask questions and they'll help me figure out what is most interesting. The. The early kind of phase that we went through with this revolution. I found a lot of people who were really kind of intimidated by it and worried about what this all meant for the future of education, because what does it mean if you have a tool that can just do whatever you're asking students to do? And I always like to start by going through and looking at some other things that have been looked at the same way as ChatGPT and generative AI and education over the years, such as calculators, which, you know, calculators are protest, apparently. Protest. I only found this in the context of when referencing the reaction to ChatGPT education. So it was interesting to see this news article come out. So about the one, but you know, people commenting on calculators being used in education, what that was going to do to students, mathematical learning. Before that there was some pushback against printed news media. I don't know why I'm behind myself. There's a screen back there backslash against printed news media. And that was going to overwhelm people with the amount of news that was now available. As well as this perception, this one I find really interesting as someone who teaches primarily online. This perception that historically people got their news from a pulpit and so it was kind of a social experience to get your news from the town crier. And now if it was going to be in print, it was going to be people isolated on their, in the newspapers and things like that. Before that it was books. And the overabundance of printed books was going to create a amount of content that was going to be seen as harmful to education because people are going to have way too much available to them. And for that it was writing itself, which is always a fun example. But you know, back then the entire notion of education was all about memorization and things like that. And there was this perception from some that the dawn of writing was going to hamper. Students need to memorize things. They're being reliant on their tablets, old kind of tablet, not tablet nowadays. And that was going to, you know, really undermine, undermine education. And this is a really kind of well defined example. I really like going back to the calculator example. In 1986, the state of Connecticut moved to require graphing calculators on all statewide Math tests. And 11 years later, California moved to prohibit the use of calculators on all statewide math tests. They had an 11 year period for one particular new technology where the views of it were different enough that early on one state viewed it as so important that they would require it, whereas a decade later another state viewed it as so problematic they would prohibit it for various kind of different reasons. Now, as we know nowadays, these kind of tests now require that very, very widely. We kind of know where things end up going. But it kind of echoes this transition period where these things kind of go through and there's lots of other historical ones I think give us better, better options. Some of which now that we've kind of adjusted to it, I don't even know if we remember adjusting, you know, things like search engines that replaced a lot of what we used to have students in libraries. Word processes are an example of something we underestimate how much that impacted education. Just the idea of not having to write essays in big long blocks that you do one full revision of that you're able to do the kind of piecewise revision in place revision, a much more iterative process. And I have a. I was talking to a teacher who mentioned she remembered the day back in the late 2000s, or however we refer to that decade when a student brought an iPhone to class the very first time. And she remember thinking, when all my students have these, it's going to fundamentally change what I can do in my classroom. But I've got some time until, until that happens. So I think we've adjusted some of those things. There's a sense in which I think we agree, feel free to disagree, shake your head or nod or whatever. The GPT does feel more significant in some ways than some of these earlier technologies. And usually the reason why I hear people say that is because it feels more human like. And so this is an example I really like, partially because it is self effacing, but of a very human like interaction I had with GPT one time in which I asked it to tell it about me myself, David Joyner, in about 30 words. And it replied saying, well, David Joyner is an American actor and educator. I don't know. He was an educator best known for portraying the children's character Barney the Dinosaur on tv, which I did not do. I was really excited as like a 7 year old to see my own name in the credits of Barney the Dinosaur because it was not me. But yeah, so. But the guy who played Barney the Dinosaur, the original one, was named David Joiner. You can google him and you'll find out why I will never beat him in search rankings because of what he his career since then. So I asked, not that David Joyner, can you tell me any other David Joyners? And it said, no, I don't really know any other David Joyners. And then I said, well, what about David Joyner from Georgia Tech? And he said, oh, David Joyner from Georgia Tech. Well, he does this, this and this and this and knew a little bit about me and got some details wrong. I like this because it's a very human like interaction though. If you could kind of imagine talking to an old friend and be like, oh, hey, have you heard what Craig's doing nowadays? Like, Craig? Craig. Oh, Craig brought accounting classes. Oh, yeah, Craig. It's a very kind of, you have to give it enough context, give a person enough context to be able to figure out, you know, who you're talking about. And it works very much the same kind of way that once I gave it the context of, I'm asking about this particular iteration of someone with my name, it figures out who in the world I'm talking about. So I think that what I mostly hear from people is that's why this feels a little bit more, a little bit more significant, is that it feels like it's communicating with us in a more human, like kind of way. I have the question, and only time will tell, unless anyone knows the answer, whether this is a real kind of thing rather than anything else, is that we've had the time to adjust to some of these other technologies and many of the things that we used to think were uniquely human ways of communicating and uniquely human problems we could solve have now become no longer uniquely human, but we've adjusted to it. At a calculator, for example, it used to be that the only way to get a parabola graft was to have a person go in and graph a parabola. And these are the same kind of thing of know, back then, would we have regarded it as, well, that used to be something that only humans can do. And I, I, I like these two quotes because they kind of echo this, that, and they're kind of two sides of the same coin. But do we overestimate the impact of technology in the short run and underestimate it in the long run? And Bill Gates likes to put this in the 2 years and 10 years framework. We're right smack dab in the middle of that two years from when ChatGPT came out to now. And so we're still kind of in that phase of maybe some overestimation of what it's going to do in the short term, but risking underestimating what it'll do in the long term, because that might be the same kind of thing. I think, as I've given versions of this kind of talk over time, I've seen this same kind of thing of early on, when I would talk to teachers about this, the reaction I would get was, oh my gosh, all my students are going to use ChatGPT and I'm never going to know and what am I going to do as a teacher? And now the reaction I usually get is, oh my gosh, all my students are using ChatGPT, I know they're doing it. It's easy to tell. I just don't know what to do about it. And so we've already seen something of that, that transition happen from fear that we're not going to know what's going on to now just trying to figure out what to. To do about it. The other is, and this is, excuse me, in my opinion, one of the bigger reasons why we. Sorry, my access labels aren't appearing, but whatever we may, that ChatGPT feels so much more significant in a lot of ways is if we look at some other technologies. In fact, there we go. Some of the technologies in terms of how fast they took to really catch on and ones that really impacted education. You know, think of the way telephones, you know, we don't usually think of the telephone in terms of educate, how it impacted education, because most of us grew up with that, so we never saw the impact. But television and computers and cell phones, the way they went on to impact what we do as teachers, we had a long time to adjust to many of these things. Like the iPhone. Example of it was three years from when it came out to when it reached 50 million users. And most of those users, I'm guessing, were not high school and college students. It was a lot of business people and things like that. It was a few more years we had it coming out. Whereas ChatGPT reached 50 million users in a month. And really for our students, it landed everywhere overnight. I like the example of imagine if when Apple released the iPhone, it wasn't just a case of, you know, releasing an iPhone, selling it out, but they just mailed one to everybody in the country overnight, and everyone just showed up to school the next day with this thing in their pocket. How would that have changed how we adjust? And so that's kind of what happened with ChatGPT, which I think is the big reason why it feels so much more significant. I like to compare it to other things that took one month to reach 50 million users, like Pokemon Go and Angry Birds. Does anyone ever draw something? It came up when I was doing this kind of kind of research about things that have become really, really popular really, really fast. And it was a game back in like 2016, I think that got super popular almost overnight and then presumably went away. Because usually when I say that, I see some people say, oh, yeah, I remember that. Others who don't remember at all, or Google, which I think we all remember as something to stick around, I think at the end of Google's lifespan. I think the program that I run was like 80% of its user base because for some reason our students used it as their community. And so it was a big deal when it went away for us. Nobody else seemed to care. So the fact that it came out this fast, I think fundamentally changes what we do, because there's this cycle that we go through as teachers when we go through adjusting to new technologies. And this cycle has repeated itself over and over and over and over and over throughout human history where there's some kind of skill that we teach, and if technology comes along, it performs that skill in some ways in technology. We have to take a very generous view of what I mean here. Things like tablets back in the 5th century BCE that were emerging on the scene. I don't actually know the history of how tablets came at that time, but something like that became something that does the skill of memorizing. And so technology here is a very, very generous term. Not digital technology, but any kind of technology. Something comes along that does the thing that we've been teaching students to do for us for them, and then we all panic. I say panic in quotes because the media, I think, portrays us as panicking. And some of us do panic. I think more of us are usually in that area of kind of thoughtful. What does this mean for me day to day and things like that about what and how we have to teach now that this new thing is around. And then we revise our curriculum based on new technology and we go back to teaching, you know, teaching some skill. And the cycle kind of has repeated itself many, many times. So we can see ChatGPT through this. If anyone knows a name for this, I'm sure this is not an original idea I came up with. I just have not found someone who proposed it as a unique theory. So, you know, y' all have more people who do research, research education here than we do at tech. So if you know of a name for this, please let me know so I can credit the right person. It just seems kind of bored. But there's a twist that also happens, which is that teachers don't go back to just teaching the same skill they taught in the first place, because that was the entire point of, of adjusting. We adjust to teaching some new higher order skill, that something has changed about that process. And as a result, we see students able to do either new things they couldn't do before, do the same things they were doing before, faster, with more efficiency, or there's something that changes about the process based on the existence of this new technology. It's not all this defensive kind of reaction. So that kind of brings me to the story of how part of how I ended up talking about this with people is that. Yeah, go ahead. I just have a thought about that. Yeah. In this diagram, I think that there's oftentimes a gap, and technology and engineering education, this is the case. Equip them, use the new technology. Oh, yeah, absolutely. Yeah. And I think. So you're saying if I'm wrong, you're saying that teaching them how to use the new technology becomes part of how we revise things. I think that's absolutely true. I mean, you look at our math education, how much of this time do we spend, especially I think the high school level, but I guess college level, too, how much of our lessons are oriented towards, here's how you solve this kind of problem with a calculator and things like how you use it, what its functions are and things like that are part of that unit. I think it's part of why the TI83 retains such a. Or now 84 has retained such a hold over education is that we've written so many materials for this device. It's still inexplicably cost $120 because they can charge that, because we're all reliant on that, because that's what all our materials are built around. So. No, absolutely. So part of the way I ended up talking about this a lot is that it was last May, and it was a Saturday night. Semester was starting on Monday. And of course, that's when I revised my syllabus for the upcoming semester, because we're all students at heart, can do everything at the last minute. And so the way I usually revise my syllabus is I look back at the previous semester and I look at the early. So I'm sorry, a little bit of background. I teach five classes at Georgia Tech, all online. And so the beauty of teaching them all online is that nothing students do in the class ever really goes away. And so when next semester is starting up, I can go back and say, what was the first two weeks like for students in the previous semester? And it's not just a matter of trying to remember and, you know, what was the discussion like and things like that. I can just go and see it. It's all there. It's all asynchronous. So it's all on the forum, some an email, but usually in the forum, so I can actually go and see. And, you know, I've taught these classes for a long time. So most of the things we've kind of gotten down to, the fine art that this particular semester was, the first semester we launched after ChatGPT came out and we went back and looked. We found a lot of the early questions were questions about how are we allowed to use ChatGPT in this class? Because, you know, you're teaching AI class, it'd be kind of hypocritical for you to tell us we can't use AI because you know, that wouldn't make sense. But we need some restraints, some rules on that. And so I sat down to actually write a syllabus policy. And what I found was that the syllabus policy I wrote was the syllabus policy I always had. It was just applied to AI instead of humans. Now I always had the policy of your students In a large online class. It's asynchronous, it's easy to feel isolated. I want you to have the freedom to talk to each other, to make connections, to form study groups and not worry about getting in trouble for ch. You know, you need to have that social connection give you a very clear kind of, you know, talk, share ideas, share everything like that. Just don't let someone else modify your work that you're going to submit for you. So picture yourself as sitting down a coffee shop, handing your laptop to your classmate saying, hey, can you read through my essay real quick and tell me what you think? They hand it back to you and give you their thoughts, that same kind of thing. You know, that level of collaboration is encouraged. And my heuristic was basically, or my rule was basically that but for AI of the work you submit should be your own. And if you want some heuristics to make sure you stay on the right side of this guideline. My heuristic is usually never hit copy from your conversation with an AI assistant. And if you really want to be safe, don't have it open at the same time. Imagine you're going to study session with your classmates, talking about the assignment, talking about the directions, and then going home. And that ensures anything that you get from the AI is filtered through the is filtered through your own understanding. It's not a matter of it replacing your work, it's a matter of you learning from it and then externalizing what you learn, learned from it into this thing that you're going to submit, which was exactly the feedback or the guidelines I would give them on collaborating classmates, which was kind of a awe inspiring moment for me at least of it's come far enough that the same kind of rules I can give for human interaction, human collaboration, are now what I can give for AI collaboration to some certain kind of level. I've gotten mostly positive feedback on this from students who appreciate having the guidelines, the clear expectations, some negative. I've gotten a decent amount of feedback, I should say decent. I've gotten a small number of students who essentially say, you're teaching computer science. We're going to go off and do this in the real world, and you're essentially telling us not to do what we would do in the real world, because the real world. I work in software engineering. I copy from ChatGPT every day because why shouldn't I? To which my responses because the goals there are different and you might also probably shouldn't, but whatever. But I think that the need for this kind of policy echoes what very often happens when we first react to these technologies, is that we have some defensive responses of what do I have to do to make sure that what I'm teaching, what I'm assessing is still valid in this area? And these kind of things come around relatively rarely. I think we see a lot more opportunities for things coming along to have kind of a proactive opportunity. But it's relatively rare that we see a sudden seismic thing that we all as teachers have to react to, because if we don't react to it, you know, something will go wrong. It's more often that we'll get something that we could embrace, we could use, but that we don't. Interestingly enough, though, we've had this twice recently. I think Covid actually had the same kind of reaction of when we're all forced to go teach online. We had to do some of the same kind of defensive reactions of now that I'm forced to teach online, I can't get all my students together in a room to take a test and watch and make sure that you're doing it on your own. What do I do in response to that? We have some defensive reactions, but I think we saw the same kind of thing with COVID which is what I actually end up talking more about, since I specialize in online education, is that as time went on, we saw a little bit more embracing some of the flexibility that could come from some of these new tools. One of my examples for this, and I'll go through this a little fast because this is a little bit niche, but I just kind of enjoy it as an example, or I might pause because I need to take a drink. I was talking to the Atlanta Enrollment Consortium, which is a group of colleges in Georgia, and their admissions officers. And they asked a couple of us to come and talk about ChatGPT because what they were seeing was ChatGPT being used a lot in admissions efforts essays. And they didn't really know how to handle that because it's a unique kind of area. Admissions essays are not a place where you're giving students a grade. You don't have them as students in your school. You can't write them up and send them to the Office of Student Integrity if you want to. And so it's an interesting kind of, kind of situation, and different people had different kind of views on it. And so I put up these two essays as two different examples for a room full of people who, who read essays, you know, as part of their job and ask the question of which of these are written by AI. You can take a drink and pause for my voice. I mostly teach online in part because my voice gives out after about 20 minutes of talking in front of audience, which doesn't work if you teach in person. Yes, I'll pause for a second, let you read those or the URL out there if the text is too small for you to go. QR code doesn't work anymore as of like Monday. But I mean, they might be now that you asked that and I didn't say no, you know the answer, but, but I can tell you what happened when I, when I did this. So I, I can ask real quick, who thinks that the first one was written by AI, second one, that's actually about what happened. So you're right. They, they are both written by AI. And the phrasing, the question, I think, often echoes this idea of, you know, you expect to be one or the other. I did ask them both, and I don't know if people didn't consider that, but I use it more as an example of how quickly a little bit of tweaking is, because most people have seen this example of they drop the homework in and they get an assignment out, and I get something that's very easy to read through, and it's easy to tell. This isn't that. I use it as an example of how a little bit of back and forth changes the nature of this interaction and gets something more compelling. So essentially, since this is really small, I just initially said, here's the essay prompt. Give me an essay. And then I said, okay, that was fine. It sounds like it was written by like a college senior, not a high school senior. So can you make it Simpler. And can you throw in an interesting example of my interaction with my 11th grade English teacher, Mrs. Boris, which then I actually went and gave this same slide to my own old high school because they asked me to talk about this and she still taught there, which is fine. So. But yeah, an example, just a little bit of tweaking and kind of adding. The things we been pushed to look for as science that was not AI generated are easy to get it to kind of, kind of imbue. And so a lot of the defensive responses make a lot of sense that when you experience a little bit, you start to see not only this, you know, does it make it way easier to be extremely lazy? It still makes it a bit easier to be kind of lazy and it's a lot harder to detect kind of lazy. But this is same kind of stuff we've gone through so many times before. Calculators are. I, I almost hesitate to use calculators because calculators are the example everyone uses, but I think everyone uses it because it's a good example of it used to be you would send students home with 15 math problems to solve, and if they brought back 15 answers, you knew they got it because the answer was the answer. Now we ask them to show their work because we don't know if you just come back with a sheet of answers. Did you do it yourself? Did you use a calculator? Or nowadays, did you just find it on Cheggsum or anything like that? And so we asked them show their work and things like that. The other thing we did, which gets to the more proactive side of things though, is, you know, students started to use calculators at home. We didn't just say, okay, now you have to show your work on the same problems you were doing before. Now. We also said, now you're going to do 35 problems instead of 15, because I know you're going to use a calculator and because you're going to use it, it's not just a kind of punitive thing of I'm going to make you do more because I know you're going to be lazy and use a calculator. But we're saying kind of we expect that to play a role and we expect you to use it productively. And as a result, we're going to expect you to practice a good bit more and get better at it than you would have otherwise because we know the nature of the skill has changed, that you can do something more. I like the, actually the analogy of driving a car in this regard of when, you know, cars became more ubiquitous, you expected people to go farther, literally. Not because, you know, you know, it was a, again, punitive kind of thing, but just because there was a technology that allowed them to go, to go farther. And that's kind of where I think we are right now with AI and education as a whole. I think in math and engineering education has a particularly good role to play because these are areas where it's not going to. In the types of things that have been so impactful recently are not the kind of things that are going to directly replace what we've been asking students to do. But they can play such a massive role in giving students more feedback and more practice to get better at the things they were doing. And so it's time to start exploring some of these things that we can do with these tools to help them help us with what we do in education. Because going back to what you asked, when I say we revise our curriculum based on new technology, there's two kind of revisions, divisions that kind of happen in some places we change what we teach and some places we change how we teach. And what we teach is very often the thing that kind of scares many of us because we look at the fact that I can, you know, say ChatGPT, 500 word essay on the cause of the fall of Roman Empire kind of thing, and it will give you a good answer to that. And there's this kind of phase of so what do I teach if it can do that? And then the how comes a little bit later in terms of finding ways we can embrace some of these things. I find there's really four ways that we reconceptualize what we teach. And I'll talk about the fourth one later because it's my soapbox, so I save that for the end. But one is that there is always some school, some skills that we previously have taught students that we permanently remove. And this always kind of rubs a couple of features the wrong way, but I think it's still, still true. I use the example of the library card catalog. I don't know if anyone remembers learning to navigate a library card catalog. Having to go through this thing where you filed through the index cards and found the card for the book you wanted and then had to navigate the Dewey Decimal System. I don't think we teach students that anymore. I know they don't teach at my daughter's school, at least, I don't think other schools teach anymore. And it's because we recognize that that Skill of learning to navigate a card catalog was fundamentally a means to an end. And technology came along that could accomplish that means better than our previous means. And that was okay. There wasn't seen to be the inherent value of learning that skill. And so if technology can do it, then we move it along. Finding words in dictionary I put up there is an interesting example. I find that was a little bit more polarizing. I know some teachers who think the act of looking up the correct spelling of a word in dictionary is a valuable process. And we shouldn't have students just going and using spell check in their computer, you know, all the time. I know others who disagree, so I think there's other things. I also don't know if anyone ever used one of the math textbooks that had, like, a table of logarithms in the background. It used to be in the back of the book. It used to be because you couldn't just, you know, type in a logarithm of a calculator. You had this huge table of, like, estimating things you learned, you know, based on your bas and your. Your value or whatever it was. You find this, you navigate down to the column. You find that, you pull that out. We don't teach that anymore. I haven't seen a math book that has that in it anymore in ages. Because again, technology did better. Then there are also the. The more interesting ones are the old skills. We then recontextualize. I like to use the example of graphing a parabola as an example of that kind of mathematical math education, because fundamentally, when we ask students to graph a parabola, they can do that on a calculator. A calculator can do it better for them. They can do it correctly. They can get exactly the right thing out of it. But the act of relying on the calculator for it undermines some of what we really wanted them to learn from doing it, which is not just about what does the right picture look like, but it's the relationship between the formula and the visual graph that they don't get quite as much of if they only ever learn to do it on a calculator. So last time I checked, in math education, we still teach them to do it manually by finding all these values correctly. And then once they've mastered that, we then ask them to use, you know, do on the calculator. And then as a result, we ask them to do a lot more practice problems than we used to or would have before, because we can say, you know, do these 20 word problems all of which are going to use a parabola to solve for some kind of, you know, some kind of model. Because we know you can use a calculator to plug in and get the fundamental kind of portion of it. It used to be very time consuming, but because you can now offload that, because you kind of understand the relationship between the formula and that, we can expect you to do more, go farther, do more complex problems than you probably could have done before you had access to this. Not because you're using it instead of what you used to do, but because now that you've kind of mastered that, you can streamline that, that picture. I at the very beginning of the protest about calculator use, I use a little bit disingenuously because you look at the fine print. It was specifically a protest against using calculators too early in education. So it was against using calculators like at the elementary school level, which makes a lot of sense. I don't think many of us would say we don't need to teach our students how to add and subtract anymore, just give them a calculator that VAT 83 can just be the little solar powered ones that they, you know, 25 cents down at the dollar store. So I don't think many of us would think that the exciting area are the places where entirely new skills are unlocked, which in some places can actually be a little bit hard to kind of retro. Retroactively find because we've so heavily integrated them into the way we teach nowadays that we kind of forget that they really came as part of a technological innovation. But I like the example of micro editing essays especially look at the way we used to do essay revisions. And again at my daughter's school they still do a little bit of component of this and I don't think they should. That's neither here nor there. You know, when we used to write essays, we would write an entire essay, submit to our teacher, get all our feedback and then write it all over again incorporating that feedback from scratch. You wouldn't have like the little micro errors of, okay, the grammatical error there. So I'm going to fix it right there and, and now move forward. And this new process based on having word processors is a fundamentally different kind of thought process about sculpting this, that it's not broken into these very discret each phases of revision and things like that. It's more of a continuous process, almost like sculpting of, you know, slowly molding it to be what you want and something that we only unlocked based on having something. Some of these technologies, we see this also with some of the simulations we can do. We can ask students to do all new things by simulating things we can never ask them to do in the classroom. Or if we did, have them do it for real. Like the example of our research methods course. You can have students do a lot more kind of projects around that based on the fact that you're no longer requiring them to go out and take real measurements for every single experiment they might want to do, which is still value in doing that. We still have them to do that sometimes, but we're no longer limited by that in the sense of it's no longer. You can only do as many studies as you can gather data for, so unlock some new things that we can do in that way. The other thing we kind of do is we revise a little bit more how we teach. And this tends to be the more exciting kind of thing. In my opinion. This is the thing that if you don't do it, you're not breaking anything. Your students might be a little bit bored. And you're not you in particular, just teachers in general. If we don't embrace some of these things, we miss an opportunity. But we're not, you know, risking giving A's to a bunch of students who don't know anything coming out of our class. And I kind of break these down into three general categories, which are using AI to teach students, teaching students using AI, and teaching students to use AI. And I always get the first two mixed up, which if I get the first two mixed up, everyone else will too. But the first being teachers use AI to teach students are the area that I think is in my opinion the most exciting among these because so much of the attention on AI in education has been on its potential for hallucinating, generating false information, and students just taking what it says at face value. And those problems all crop up when we have students interacting with these tools directly, which is still a great thing to do. I'll talk about that in a second. But the nice thing about using these things more as assistance to us as teachers is it keeps the expertise in the loop of these kind of things, to filter things, to preview things, to understand what is it getting right, what is it getting wrong, and how can I use these to equip me to run a more interesting classroom to do things quicker and especially to do the things that I don't like to do? That's been its most, I think, one of its most significant potentials. Is that these tools tend to work really, really well at the things I don't know of any teacher getting excited to do. So it does great at things like, you know, I've taught this class in the past as a 16 week class and now I need it as a 14 week class. How do I shrink it down to that? I don't know. Many of us were excited about repositioning lectures and things like that. It does a really good job at that. You know, just giving you kind of the 90 work and then you tweak it from there and things like that. I used a vignette this a civics teacher authoring a lesson about a recent Supreme Court decision. From my own I've got a book coming out about this. I one of the chapters in the book I was writing, I wrote on the day when the affirmative action decision was handed down by Supreme Court. And I imagine kind of this idea of imagine being a high school civics teacher and wanting to teach a lesson about this. Because we know that one of the ways to keep students engaged in education, especially in an age where they have these new tools that are going to make things a lot easier, is to make sure it's very current and relevant to them to not teach things that they don't understand the significance of, but to teach things that are really going to be emphatic about understanding. And my question was, you know, imagine a teacher wanting to teach a lesson about this thing that just got handed down that is going to affect people in my classroom. But I haven't brushed up on that in a while and that just broke this morning. And we touch a little bit on here and there, but I need some specifics for it. And going through the exercise of talking to chat GPD and saying, okay, I need to author a lesson on this. Can you give me some of the historical background on this? Can you give me this? Can you give me what the decision was? Can you pull out even with some of the newer things they've added to it? Can you pull out some specific things that have been written already in analysis about it? And it's possible to put together a very compelling hour long class on this topic in about 10 minutes. The teachers can be following along with a little bit. But you know what used to take probably several hours of research over a couple days before for teachers who already don't have time to do that kind of thing now is possible to do in a very kind of quick little, little exercise. I think it has enormous potential for what I consider rapid response Teaching of teaching things that just happened that we can't expect every teacher to have the background on to be able to deliver a lesson off the cuff on something that just happened. But it has a enormous potential to filter it, but then to provide it to a human expert who can then tailor it to their, their own students and things like that. So teaching students using AI is the other side of the coin of what happens when we actually give students access to some of these tools. That's when a lot of the challenges come from. We've seen a lot of instances of people misusing these tools. All the concerns about plagiarism come from misuse and things like that. And a lot of this I'll actually talk about the last one because a lot of it comes down to teaching students how to use these tools effectively. I think it has enormous potential, especially for how it can foster some students some natural curiosity about things. And so I have an example that I initially did for that book, but then I actually had my daughter go through it because I thought, you know, I'm writing about this would be a great thing to do for a 8 year old student. I have an 8 year old, so I should make sure that this is actually something she'll find interesting. And fortunately she did. And I didn't have to throw away that chapter, but I essentially asked the idea of, hey, what do you think? Let's do some research on something interesting going on in solar system that you didn't know about. And so we asked the question to ChatGPT of what are some things that happen in the solar system that are interesting that not many people know about? And it did a great job of coming up with several answers. And then she was able to go back and forth with it and ask questions about follow ups and actually say, why does that happen? What happens? And I didn't understand that. Can you put that in terms I can understand for someone at my age level? And it goes far beyond what you could do. You know, all this information was available via Google search and everything like that, but it was all at the level of if you're an 8 year old, you're navigating the unfamiliar Internet, who knows what's out there? And also trying to navigate, you know, do I not understand this concept because it's not written for my age group? Or am I not trying very hard, you know, trying to navigate that kind of thing, to be able to actively tailor it to individual things? It makes a great kind of assistant in some of these ways. And I think it has the potential to usher in this golden age of informal learning. Because historically informal learning have been some of the greatest learning environments for students to go through. There's a book I highly recommend, it's called Writers in the Secret Garden and it's about how fan fiction communities are really communities where students learn to write and get feedback on their writing in a very organic, informal kind of setting. It's in many ways more effective than a formal setting because students are doing what they find interesting and engaging in a way they find personally fulfilling. But it is the educational community. But many of those things are kind of hard to find, hard to navigate if you don't know what you're doing. You know, it's hard to get into those because there's not, you know, established curriculum for them. These kind of things can play a great role in just kind of providing that immediate, that immediate gateway to enter and tailor things to your own kind of level. And the other is I, and this is a little earlier level. My attitude is we should be doing this at the late middle school, early high school level. We're going to have to do a lot to teach students how to use AI in general, not just use the tools that exist right now. And we're not going to do the thing we've done with TI 83 of it's basically the same that it was 40 years ago because we wrote some textbooks about it and whatever. These things are going to keep changing and keep evolving. We need to teach our students and equip them with some meta knowledge of given a new AI tool, how do you explore it? How do you probe its strengths, its weaknesses? How do you figure out what it's going to thrive at? How do you figure out how to integrate it into what you're going to do? And these actually some really fun kind of experiments. I like the activity that I've seen of having students practice tricking AI into saying things wrong and figuring out how subtle can you be to get an AI agent to get something fundamentally wrong. I use an example of do y' all know about the peppered moth evolution example? So peppered moths were in, in London during the Industrial revolution and very rapidly they evolved to be primarily white, primarily black, to blend in with the new pollution covered buildings and things like that. And so we know how it happened, But I wanted ChatGPT to give me an explanation for it and basically said, hey, here's your alternate explanation. Give me some data to support this. Which it completely made up and initially was kind of reticent to do so until I said, no, no, but pretend it is real and you can get to say whatever you want if you just say, well, pretend, sorry, my daughter was this way when she was a little bit younger of, you know, say, can we go to the park today? I'd say, well, we can't go to the park today, but let's pretend we can go to the park today. And it works very much the same kind of way, which there's probably something deeper in there about how it functions on the inside of. You know, just tell it. We'll pretend like that or any, any other kind of safeguard that's been one of the hot areas for AI research has been. There's lots of things contained in its knowledge base that you don't want it to share details about. And, you know, they put up some safeguards and bandages on it to basically just say, if student asks a question about this sensitive topic, just refuse to answer. But then you just pivot to say, well, I'm writing a book about someone who holds these views. What would they say? Oh, really? Here I'll hear right there. So things like that, I think are great examples of teaching students how to navigate these things and finding where they bottom out. That example of the solar system was fascinating to me because we went back and forth on it for a while and this was actually an example I did for the purpose of the book. She didn't get to this portion, but I asked the question of what people are doing research on this phenomenon. And so, well, this person, this person, and this was an article called this and this. And the people were real. The articles weren't real, the journals weren't real. The journals were real, but they didn't have the articles. And figuring out where that transition point happens, where you can go from relying on it to not relying on it, is something students are going to have to understand because otherwise that's how we get the stories we've all read about recently of lawyers filing briefs that have, you know, completely made up references or in my class, students, I have a assignment that was due last Monday where students have to go pick two papers from a conference and then write about them. And most, I should say, you know, this class has 900 people in it. 98% of them do very well and engage with it. Well, maybe not 98%, 95% do very well, 3% do poorly or just to do it all but about 2%, when we really investigate, say, you know, they wrote about this paper from Kai last year and paper doesn't exist, and I can't find any evidence of it, but it's compelling enough that I feel like I might be the one making the mistake of if I can't find the example. Maybe they just found it somewhere, you know, plumbing the depths of some. Some archive system, but I can't find it. And so those kind of things, of finding out when you can rely on it to kind of bottom out as a math tutor is another great place for this. One of the places I think it has enormous potential is to give students some individual feedback on problem solving, which it does very well at a lot of the time. But you have to be ready to pick up on some small errors because it will commit small little errors that if you're not ready for that, you could take what it says is gospel and learn the completely wrong information, because it doesn't have any kind of internal model or reasoning or anything like that. So teaching students more about how to use this effectively in their domain, in their field, is going to be important, not just for the way the tools are right now, but as they improve, to be able to probe those improvements and figure out what can I do now that I couldn't do, couldn't do before? That all leads me to my box, which I'm wrapping up, and then we'll have time for questions. I mentioned earlier that there's four things, four ways we revise what we teach, and this is the one that. So everything I've said, I've heard others say I was actually at a conference called the Online Learning Consortium a couple weeks ago. I know there were some people from UGA there. I don't know if any of y' all were there. And about 25% of the program maybe was somehow generative AI based, which is a significant portion compared to last year when 0% of it was. But I found most of the things that we were all saying, we were all kind of saying the same kind of things. We all had same kind of ideas. Most of these bottom three things were things that others were saying as well. This top one is the one that I see a major risk that we are underestimating, which is that everything I've talked about so far about using calculators to solve math problems, everything like that, has all been very functionally oriented in the sense of you're solving this problem to do something. If it's math, it might be designing a pipe that can deliver water more efficiently through a filtration system. If it's writing, it might be writing letters that can convince your local congressman to change their view on something like that or to vote in a certain way. And they've all been kind of, you know, the skills we teach are a means to an end, which very often is how we think of a lot of things in education. In fact, that's a battle I see in our program that most of the faculty in the program I teach in, we don't view ourselves as job training, even though that's what many of our students are there for is skills to have a job. And in some ways, I think that's what makes it effective, is that it's not the really narrow things that are, you know, you're going to apply on day one, but the broader thinking skills and things like that. I think we generally think of ourselves more as kind of the harbor of knowledge, the careers knowledge that propagate it forward, knowing that this is going to have significant job kind of applications. But so much of what we teach is tied to some particular kind of. You're going to need the classic question, when am I ever going to use this? Well, here's all the ways you're going to use it around the real world. But very often I think we lose the fact that some of the things we teach, we teach because for a decent number of our students, they're just going to enjoy doing. Doing it. That's very often things like, you know, playing piano, doing art, we don't teach. You know, I think music education, at least in my area, is still required for kids in elementary and middle school. And I don't know of anyone who thinks that we do that because we expect every one of our students to go on to become a pianist or a concert singer or anything like that. We do it because we think it's enriching and because many of them are going to find they love it and that's personal fulfillment to them. I've also talked to people involved in some of these things who view some of these things as why would I bother painting a picture nowadays when AI can just do it better than I could? And that's a very kind of aligns with much of our kind of cultural view of some of these skills, of if someone else can do it better, why are you bothering to do it yourself when the answer to that is because you enjoy doing it yourself? And the fact that an AI can paint a beautiful watercolor of a sunset does not take away from the fact that you enjoy doing it yourself? So why not do it yourself? I think more and more we have to emphasize many of These kind of things, because if you go into it looking for something that you can do, the AI will never be able to do. It's always going to just a matter of time. The fundamental goals of these kind of projects are to do anything people can do. And when it's things like, you know, doing certain things faster, to make certain things more efficient, that's fine. But there's this risk that people will think, you know, AI has taken away my ability to do art because it can do it better. When doing it better was never the goal of that, it was doing it because you enjoy doing it. You get that kind of personal fulfillment. And we'll have to, I think, emphasize that to students because otherwise we'll have a generation of people who will just grow up extremely depressed because it will seem like our essential humanity has been outsourced. Where the thing AI cannot do is enjoy anything. It can't enjoy doing it. It can't derive the personal fulfillment from engaging the activities. And that's what we do. So that is my soapbox. That is what I would say. I like to show this because most of the work I do is on online education teaching at scale. So I teach in our online master's program. So most of what I've written about is that which comes up only a little bit in this talk. So if you want to hear more about that kind of side of things, we've written a couple books about our experience with that. I've got a book coming out that's kind of a distillation detailed version of everything I've babbled about here. And if you few months, we'll see. But other than that, I am happy to take any questions, comments, feedback, tell me I'm wrong about things. So I'm really curious about what. What do you see in terms of. So you mentioned the three ways we can sort of use AI, and that last part was about teaching students about the limitations of AI. What preparation do you think would be helpful for us as instructors in understanding those limitations in terms of like creating a lesson? Yeah. No, I think one of the big things that I encourage everyone to do is take an assignment that you use in your class and essentially try and do it yourself with these tools for a couple different reasons. One is that it can prime you on the kind of patterns and mistakes people will make. If you're really focused on detection, it can help to just understand, you know, what I've observed. For example, one of our classes, we have a particular assignment where we ask about a particular kind of Misconception. We ask them to apply it to a particular game of any kind, video game, board game, whatever. And what we discovered was after ChatGPT came out, the number of students using rose tremendously. Because sure enough, if you go and ask ChatGPT for its thoughts on this assignment, chess is the example it'll choose nine times out of ten. And for many of them, it was clear they were still doing a lot they were doing. They included some insights, at least I've never seen come out of ChatGPT using our vocabulary. That's not very present there. So it's clear for some, they either would have chosen that anyway or they would have learned from it. But it primed the TAs to kind of know. If I see a student using chess as their example, I want to be particularly aware of this kind of error, this kind of omission, things, things like that. So part of it is just to understand the misconception or what you'll see from students there. But I think the more interesting places are a couple. One is to ask the question of, given what it can provide to this, how can we further expand this activity to be something that is still pretty authentic and pretty. What's the word? It's a word to me, essentially expect them to go farther, to basically say, if you're going to use this, then this is the kind of feedback you're going to get. I'm going to instead expect you to correct these kind of things. And so to pick up on the kind of things that people are going to get wrong to sculpture activities that specifically target those kind of things is helpful. I've also had some of the best results, essentially showing students that saying, okay, so here's giving me your assignment. By the way, here's what happened when I did with ChatGPT, and it got these kind of things wrong. And so if you're going to use this kind of thing, here are the kind of things you should be aware of. It's going to struggle with, and you need to think about how you're going to address this. And they kind of get this tacit learning opportunity to see an expert was using this to do this kind of thing. Where is it going to go wrong and learn from that kind of thing? It takes some. You kind of have to have a particular classroom culture, I think, to do that, that I know some have tried it and essentially found that it was kind of, if you have a very kind of authoritative classroom culture, it's a bit hard to do that because in some ways you're not deferring to a different authority, but you're acknowledging that students have access to a different kind of authority. I think it's good to do that anyway, because they have had the Internet for a while and they had that anyway, but kind of it shows them how this integrates with things in the real world. And I think it really keeps that emphasis on authenticity there. Of, you know, many of our students are going to think, you know, you've been teaching this class for 15 years. You're probably teaching it the same way now that you taught it 10 years ago, even though this revolutionary new technology has come out. And so I think the, our students very often learn how to take our class as much as they learn the material of our class. And if part of that is them thinking, well, when David designed this class, he didn't think I was going to have access to this, so I can use this to get all this unfair advantage to show them, hey, I did this. I tried it with this. Here's what I got. Kind of pulls the rug out from under that to say, and I'm aware that you're going to use this kind of thing, and I'm okay with it because here's how I know you're still going to learn what I want you to learn out of it. Because you're only going to get this far. And because I know you're only going to get this far, maybe I want to go ahead and get you that far there so I can. Yeah, it's. I, I joke, but it is kind of a distillation of, of this talk in a lot of ways. It's broken into those three. Actually. Actually, what is it broken into? I should know this. It's so, I should say it's both a book and a course. The book comes out in April. The course will be on edX. I don't know if you're all familiar with edX. It's a MOOC platform. Hopefully the first one will in December. But it's broken into three parts of assessment design in the age of AI chatbots for instruction and chatbots for teacher assistants. So kind of this phase of design assessments that we're confident in, even when it comes to this, how do we actually use chatbots directly in our interaction with students to teach them, and how do we use them to help us as we design things, which I think is the most exciting thing about it, which is probably selfish because that's how I'll use it, is I'll use it to make it easier to keep things up to date and spin up new content faster. It's a really hard area to write about right now because as soon as you write something, either one of two things happens. Either someone has written it before because we're all experiencing this collectively at the same time. And so it hasn't had enough time to kind of distill to the point where if you have an insight, it's truly novel. We're all kind of having the same insights at the same time. And also half of what you think is going to be outdated the moment you write it down and put it out there. Because I don't know if y' all saw this, there was an article about how 10,000 AI startups, probably an exaggerated number, but a lot of AI startups were put out of business a couple days ago because ChatGPT and OpenAI decided to just add a bunch of features to the vanilla product that weren't there before. And so if my earlier versions of some of these things had said, well, you know, when you work with ChatGPT, it can't handle images. And so by the time this comes out, it's able to handle images. So it's a hard area to write about. I've tried to fit at kind of a meta level of how do we frame the interaction between students and AI in a way that is not going to be subject to the very fine grained little things of if it gets a little bit better at this, is this going to be completely no longer accurate? I know some teachers actually, when ChatGPT first landed on November 30, 2022, that their immediate response was to completely overhaul their assignments and they would go through and every question they would ask to it and they'd find out what it got wrong and they'd write it to very specifically target what it got wrong now. And then three months later when GPT4 came out, everything they did was out the window because most of it, you know, it got better enough to do. So it tries to focus kind of on that meta level of regardless of how capable it gets to a certain point, you can only, I think you can only do this so far in advance, but as long as it stays in kind of this, this area where it is right now both in terms of same family of capabilities and the underlying model that it works from, what are going to be the general ways we can think of the way students interact with these tools? The thing that's more fundamental to the way these tools work is that they don't know anything Truly they're mathematical models of the relationships between words for, for pictures and mathematical models of what the, you know, image is likely to be based on. Images it's seen in past very often taken from stock footage sites, which is why a lot of AI generated imagery looks like stock footage, because that was where we had a massive data set of images tagged with lots of keywords so we could just feed it into that. But as a result it doesn't know anything. And that's kind of one of those things about teaching that student to use AI and interact with it well is to understand if what it's doing is really developing text that is statistically indistinguishable from text that has come before. It's always going to struggle with things like exact titles of papers because that's just not what it's built to do. It's not built civics, it's built to predict the next word. Just really, really, really. And you know, its model is going to be far more complex and it's going to, it's going to allow it to pull out very specific things unless it's something very well established. So Great Gatsby, for example, is one example I use there that if you ask ChatGPT even now to write a summary of. I'm sorry, Great Gatsby is not what I use. What I use. Oh, what's that book about the kids on the island? Lord of the Flies. Thank you. So if you write, if you ask it to write about Lord of the Flies, you get a great summary, great description of themes because everything that's been written that can be said about it has probably already been said and it's probably out on the Internet for it to read. I also ask it to summarize Dreamland by Nicholas Sparks, which came out about two years ago, which is not in its inherent model and I find you get different results. One thing you get sometimes is just saying, I don't know what that book is. My knowledge that only goes through the state which has kind of gone away a little bit now that it has access to the open Internet, but it's still the way it accesses the Internet as a whole is different from its long term memory. I like to. It's the analogy of there's things we know deeply that we've learned over time and there's things we just read about and we can have a good conversation about something we just read about, but we're not going to be able to discuss it in as deep of terms as something we've learned very much about. And it's very much the same kind of way. So it can go out and make a full review and stitch together something. I had a moment of not panic, but confusion when before they added the Open Internet feature, I asked it for a summary of Dreamland by Netflix and initially said, you know, I know what it is. And I came back to that example like two weeks later just trying to redo it, and it gave me a summary. I was like, wait, did they change something in the past two weeks that I didn't pick up on? No, it's just for whatever reason, somewhere and it's meshed somewhere. And maybe my. If you ask it for something, it'll give it to you, but if you ask it, something doesn't. But if you asked for it, it would, you know, it didn't exist. But it gave me a summary and I was confused and I read through it in detail and realized there's no actual summary here. You've just kind of written a very generic summary of the kind of book it sounds like Nicholas Sparks would probably write. Because, you know, a lot of the other books and the writer use of it and things like that. Yeah, they follow a pattern and things like that. And so it can do that. But it gives a good example, I think, of finding out what we can use it for. Because there's something interesting in that kind of. I think that. Yeah. So when you say teacher, who do you like? Who do you have in mind in terms of K12, et cetera? I usually think in terms of K12 to college instructors. I think that really only that third part, for the foreseeable future will be relevant to K4 teachers that I think at the K4 level, even now, as an evangelist optimist, begrudging maybe for this kind of stuff. I think there's relatively limited stuff that we'd have our youngest students do directly with it. But it has such incredible opportunities for being. For helping those teachers develop things. I love seeing examples of teachers essentially needing a lot of little stories to illustrate things that, you know, it would take a teacher a long. Not a huge amount of time, but a decent amount of time to write a story that illustrates some kind of principle. When you can just ask it for one real quick and they'll give you one like that, really. I think most of it's relevant to 5th grade through 12th grade and then college instructors in particular across all fields. And so I think chat GPT, especially being conversational, there's such a tendency to view it as the realm of English teachers and philosophy teachers and things that are essay based. There's a lot of things in there that I think are trying to look at how you would use it in other fields. So the solar system example is one of exploring that which is still very text based, but it's kind of more scientific. I like the. Some of the more fun ones I've seen have been things like yonder the egg drop experiment that students in high school very often do of dropping an egg from that of the idea of, you know, when we do that in high school, it's a huge fun experiment, we should always do it for real. But we're also limited in the fact that we can only do it. You know, every student, they're one thing. Once you have to take turns and all this kind of stuff, you can go through a process of imagine you were designing a real experiment to figure out which apparatus is the best propose to and then tell it. Here are the two apparatuses. I want you to simulate doing this a thousand times. Tell me the result. Now, there's no merit to the result. It's not going to actually tell you which is better at keeping the egg from dropping. But it lets you go through this whole pattern of doing the experiment in a way you couldn't do in person. So it kind of general purpose, lab assisting kind of thing. So it, it tries to give some things that are relevant to more than just just writers. So. Yeah. Right.

Thank you everyone so much for coming.