OTJtv #53 David Joyner - Online Education in the Age of AI

Here we are with Dr. David Joiner, everyone. Good morning. Good morning, David.

Good morning. Thank you for having me.

Thank you very much. How's your throat? I think you were having a few problems with it before.

Yeah, I think it's okay. I think it's just unfamiliar pollen. Although I did discover it shouldn't be that unfamiliar. The tree that I thought looks so familiar at my house is actually a Japanese cherry tree. So I don't know what I'm reacting to since I have one.

Do you have a cedar pollen allergy? Because we have a lot of cedar pollen. Give people.

It might be that. Yeah. Well, I don't know if it's out in Atlanta right now, or maybe it just has come out since I left and when I get home, it'll be the same.

Speaking of Atlanta, Dr. David Joiner joins us from Georgia Tech, where he is the, I guess the chief of the graduate program in remote learning. Did I get that title right?

Kind of. I'm the executive director of the online program in computer science. So we've got three really big online master's programs, and like, 19 total. I run the biggest one, which is the Master of Science and Computer science program. Okay, good.

And I was really interested in speaking to you when Jeffrey Cross introduced the two of us because for a lot of us here, we gathered in online teaching Japan because we were sort of forced into the whole idea of remote teaching, and. And a lot of us were hardly familiar with computers. So you're an expert in the two things that I think most of us here probably, like, sheepishly say, I'm not really good at that, you know, computers and remote teaching. And I wanted to get your expert outlook on not only the things that we went through, the things that are in front of us, but especially as people might have seen in the PR card that I brought to the front to talk about this episode. It's about the future of artificial intelligence and large language models like ChatGPT and. And where, you know, where that all comes to a Nexus. But tell me, you know, what for. For you in your program for remote education and computer science, what does that like? If you had to describe that to somebody who, who was going, gee, what's that all about? How would you describe it?

Yeah, no, it's been interesting. I think one of the interesting things about it that I always have to remember to tell people because I forget about it because it's just obvious to us. We were online long before the pandemic. We launched in 2014. And so the pandemic has been kind of surreal for many of us because suddenly everyone was doing the thing that we've been doing for a while. I joked, because I became executive director of online education in our college on January 30, 2020, and they changed the title for that position at the last minute to be all of online education, not just our online graduate program, which at the time was kind of a trivial change, because our online graduate program was really 99% of the online stuff that we did. And then two months later, everything we did was online. And so it was interesting to be in that position. Now we launched in 2014 as a fully online graduate program in computer science. We were trying to capture what we were seeing in a lot of the massive open online courses, the MOOCs that were really making education more accessible and more flexible and more affordable. But MOOCs, you know, they were never attached or very rarely attached to full college credit. And so they were kind of their own category of things that no one really knew what to make of. And we wanted to see, can we take some of what we're learning from MOOCs and apply it to for credit education? And have had a lot of success. It's been almost 10 years. We're the biggest graduate program of our kind in the world. If I'm wrong about that, please somebody let me know. I've been saying we're the biggest for about five years, and no one has corrected me. But there's also no easy place to go and look up who the biggest ones are. But we've tripled in size since I started saying that we're the biggest. So I reasonably confident that we're the. We're the biggest.

When you say the biggest, are you counting modules offered, faculty enrollment?

Enrollment, yeah. See, we've 11,500 students this semester. We get about 5,000, 6,000 applications per year. And it's the really unique thing about the program. Well, part of what makes it unique is it's highly affordable, at least by American standards, where higher education is extremely expensive. Our program is US$7,000 for the entire degree. I'm kidding. Yeah. It's a small fraction of what a comparable degree in the United States would cost.

Seven.

Wow. Yeah. And that's what gets a lot of the press. If you Google our program, you'll see there's an article in the New York Times eight years ago that was, you know, Georgia Tech, $7,000 master's degree experiment and things like that. And that's great. What I think is more interesting and Exciting about the program is it has a very inclusive admissions policy, which is that, you know, on campus we always say, you know, here's our bar for admissions. But when you only have so many spots to fill, you only ever accept. Back up, to show my analogy, you only ever accept like the top 8 or 10%. And so it really doesn't matter where you set your minimum bar, because you're only ever going to accept that top 8 or 10% based on how many physical seats you have in your classrooms. We don't have physical classrooms for our students, and everything is asynchronous. So it's not even like a matter of zoom room capacities. And so we can accept anyone who we think has a chance of success. And we don't have to rank order people and decide we want this person or this person. It's. It's very individual. If you are qualified, you get in. And I find that far more exciting. In a traditional program, you usually find that the students who enroll, especially Georgia Tech's program, is pretty highly ranked in the United States. And you'd very often find that the students who enroll in the program are the ones who went to a great undergrad institution, had a 4.0, and they're really great students, but it's a very homogenous population. And. And so there's not a lot of life experience going around in that audience. And you're really serving students who've had a lot of advantages already in life. The ones who were able to go to the really expensive good school were able to afford the resources necessary to get a 4.0, whereas we're able to open the opportunity to people who haven't had quite as many advantages. So we see a lot of people going into career transition. We see a lot of people who their prior career is getting automated or is getting outdated. They're looking for a career over here. I also see a whole lot of people who just think what we teach sounds interesting. And so they come without any real career aspirations because it sounds like cool stuff to learn. And they improved the class for everybody else in it because they asked great questions, they had great discussions. And so, yeah, it's a great program. I can say that because I had nothing to do with starting it. I joined working on it shortly after they decided to create it. So I had nothing to do with creating it. I've just been working on it since it was founded.

Speaking of great programs, there's yours. I'm stunned. There's a bunch of things going in my head and when you said $7,000, I had a guest on, I think it was the last episode before this one, John Gorham, really good guy. One of the members of otj, and he was talking about a similar program, an online program. And unfortunately, the name of the university escapes me, but it also was a graduate program, an online masters. That was for about 7,000American dollars. Is. Because that stunned me there, and I thought, wow, that's me. And then here it is. It's not. Is there. Is there some kind of trend that's happening towards, like, really lowering the cost from the astronomical numbers that it was or still is in most cases in America to. To something a little bit more affordable, like 7,000American dollars?

Yeah, I'd be. I'd be curious. The program he's talking about, there is kind of a movement.

I'll look it up for you.

Yeah, it's called Affordable Degrees at Scale. Georgia Tech was kind of the founder of it, along with what we call our sister program up at uiuc. It's University of Illinois Urbana Champaign. They have an online MBA program that is run in partnership with Coursera, but it started kind of this movement towards more of these. So There are about 50 or 60 such programs, but most aren't quite that affordable. Most of the programs in this area are between 10 and $20,000, not under 10. The only ones I know of that are under 10. So UT Austin, University of Texas at Austin has a program in computer science and in data science. I think they just launched one in AI that are under 10,000. So there are. There are friends, but they're also our nearest competition. Their programs are very similar and similarly priced. There's also the American. Oh, shoot. What does it stand for? I think it's ace American Something of Education. It's an online education. It may have been that they. They have $7,000 master's degree, and they've had it for a long time. American College of Education. That's what the C stands for.

Sounds familiar.

Yeah. Yeah. And, yeah, they're very affordable. I don't think they're at scale. I don't know what their enrollment is, but I think they're a more traditional online program that just also is very affordable.

When you were watching the pandemic like the rest of us, your program at Georgia Tech is all asynchronous. It's all, I would imagine, modules where there are forums to be discussed and assignments and programming code to be handed in. But when you were watching the pandemic, and you were watching the Dust from everyone shuffling around trying to figure out what to do online when they're very much used to doing chalk and talk lectures. What did you think to yourself? Gee, I wish, you know, they understood this more about remote teaching or at least asynchronous. Doesn't work that way. What, what did you think that even, you know, universities, maybe even Georgia Tech itself, some professors weren't doing right. About remote teaching.

Yeah. And I think the asynchronous component, or I've heard here it might be called on demand more often.

Yeah, that's right.

Is that, that was the big one, is that everyone kind of defaulted to this approach where I have to go online now I'm comfortable getting in front of a class, live with my whiteboard. So now I'm going to get in front of a class, live with my, you know, some of them figured out how to do a whiteboard digitally, others did different things. And it was very much a kind of the most familiar kind of transition, which I understand why. You know, it was a crazy time and we were all doing the best we could, but I think a lot of people just assumed the right way to do it was to go synchronously and do a synchronous remote teaching. And I mean it's possible to be, to do synchronous remote teaching very well. I find it among the hardest of the modalities because you don't have a lot of the control that you have in an in person classroom to kind of move around, monitor individuals attention, to detect if people are getting bored and involve them in the discussion. But you also don't have some of the advantages of teaching asynchronously. And so it's possible to do it well, but I think it's among the most difficult. I wish more people had experimented with some asynchronous teaching, some on demand teaching. And I know people who did ended up really liking it. I usually try to emphasize the idea that you're unfamiliar with technology, you're making this transition. Chances are if you're like many of us, you really don't want to embarrass yourself. You don't want to be the person who gets posted about on Reddit about. My professor lectured for 30 minutes without realizing his mic was off and he wasn't looking at the chat. So he didn't realize you don't want to be that person.

No.

And the nice thing about asynchronous teaching is that you get to sit down and record your content and preview it and make sure you like it and make sure you didn't say anything that's, you know, going to embarrass you and you get to be perfectly comfortable with it before anyone else sees it. Compared to live presentation, where you have to be good on your feet and if you make a mistake, it's, you know, it's there permanently. And I've talked to people who could see that benefit. I think the other thing, a lot of people didn't adopt that idea because they really liked the interpersonal. They were worried about how isolating asynchronous might be. They thought that the students would be separate, which is a very legitimate concern. I like asynchronous, but I also communicate relatively well via email and forums. And so I feel connection to my students via asynchronous communications because I grew up on Internet forums. That's just kind of how I, how I got to know the Internet. But others are not nearly as comfortable in that area. The interesting thing about this, some of the asynchronous options. We proposed a model that I call the hosted model for teaching asynchronously and synchronously kind of together. Hosted in this context, it's not like hosted like web hosting. It's hosted like.

An mc.

Yeah. Like a radio host kind of thing. Like something like this, where essentially the idea was record your video, be comfortable with it, and then during your synchronous class time, turn it on and have everyone watch it together. And then when students get confused and ask a question, you can pause and answer their question live. If they're asking questions in the chat, you can kind of be in two places at once. And then you don't have to record that. So you're not committing your live presentation, where you might make some mistakes, to immutable video forever. You have a really nice video that you can share with the students who couldn't attend live. And so they still get a nice resource, but you still have this avenue to have some more interaction with students. And the teachers I know of who experimented with it said they really liked it. It was my idea. And so I don't know if they come back and say, david, that was a terrible idea because they're nice people. But at least I've heard from some that they really liked it. We did a similar kind of experiment in my classes before the pandemic on co watching among students, where the goal was more to give online students something to. To an environment to interact as part of their studies. And they liked it too. There it was held back a little bit by the fact that we were trying to do it in virtual reality. And virtual reality headset is still. The adoption of them is still kind of low, so it's limited who even can do that.

Also, it's really weird not seeing people's lower halves of their bodies.

The one we did actually had kind of more normal avatars, but.

Yeah, yeah, but still, I don't know. I don't. I don't know if I'll ever get used to that. Well, what do you just, really quickly, what, what do you think of the Metaverse as in its current iteration, as a educational vehicle?

I want to be more optimistic about it than I am. I. I feel right now, just looking at the trends right now, I feel towards virtual reality the way I feel towards 3D TVs, that for a while they were the big next big technology. But the number of sacrifices and constraints on using them was so high that adoption remained relatively low. With the idea of wearing the goggles or sitting in the right place and everything like that just didn't work into how people live their lives. VR, I think, has many of the same kind of challenges of it requires you to completely put aside your regular life to move into this. And so things like video games you might play socially, like, you know, I like to play a game while my kids are, you know, playing and they half watch and half play their own stuff. You know, VR, it's much more. I'm going in my tube and you can't. So I'm, I'm skeptical about how much adoption we'll see. Augmented reality, on the other hand, I think, is going to be huge once it's, you know, kind of figured out, because then you're not making the same kind of sacrifices. You're not saying, either I'm in VR or I'm not. It can augment. Well, it can be augmented reality. Imagine that. So that's where I'm a little bit more optimistic. The technology, I think, for augmented reality is just tougher and still be able to put more time before it sees widespread adoption. Google Glass, I think, was ahead of its time, but it'll come back around.

Well, that's where I think Apple is putting a few extra eggs in the basket. And what they're trying to do with the goggles that they're going to try to, to present to the public is they're more into AR than anything else. So we'll see. Like you said, for a lot of the conversation that we had on Thursday, setting up to this, it was fascinating. I couldn't believe that we actually spent an hour, nearly an hour, as quickly as we did, because I am so very interested in it, mostly because I'm. I feel almost a little threatened by artificial intelligence.

Oh, yeah.

But in terms of that feeling of like, whoa, what's going to happen to my job? I had that feeling even a little while ago, maybe 10 years ago, when the whole idea of robots and I don't know if you would call it artificial intelligence, but when new technology came out that was replacing very rote routine work, like finding, looking at. Being able to look at an X ray and then say that, yeah, that's the spot you have to be concerned about. These machines were actually what I've just quotation marks. Machines were better than most X ray techs at doing it. There were certain law firms that were using, I don't know if they were LLMs or not, but to write out contracts and stuff. And I was thinking to myself, yeah, but that's not even good. That's not really going to take a creative job like mine that isn't about routine work. Now here we are, and there is a little bit of talk about how, you know, no teachers actually do a lot of routine work like marking and stuff, and that is maybe on the chopping block. So just in general terms, are there risks for education or educators with the coming wave of AI, do you think?

Risk isn't the word I'd use, but it wouldn't be wrong to call it risk, only because the places where I think there's the most risk of AI displacing humans are places where humans are already doing the job about as well as can be done in terms of the end product, and AI can come along and do it more efficiently and things like that. So the example of junior law associates, for example, who review lots and lots of case law and are putting together citations and finding all the precedents and things like that. I really don't know law, so I apologize if any lawyers are listening. And I just radically misrepresented what you do. I've heard this is one area that is ripe for replacement because the work is perfectly suited for large language models that what they're doing really is operating in a very similar way to what large language models really thrive on. And in that case, it's a case of humans are already doing about as well at the task as one can expect. Large language models and AI can come along and do it much more efficiently. And that's a place where I think There's a lot of risk for humans being replaced. I see less risk in AI. Not no risk, but less risk in AI in education because we can do so much better in teaching people. There's so much room for improvement that equipped with AI. There's still plenty of room for us to say now that I have an AI who's going to make it 80% more efficient for me to grade students work. That's time that I as a teacher can now reinvest into producing good new materials to investigating misconceptions, to doing things that are more, that rely more on some human nuance or even just some human touch. A lot of the things come down to the fact that, you know, so one thing I do in my classes is I send regular check in emails to all my students. One of my classes, I should say my undergraduate class, and it's a regular check in that just says, you know, my class is semi asynchronous. So there's a schedule as a calendar, but there's not like a daily schedule. So you have to get things done by certain deadlines, but you don't have to tune in at a given time. So there's not the live back and forth. And so I'll send emails to students every couple weeks that just say, hey, don't forget. It looks like you're about three weeks behind where I think you should be. You can catch up. I just want to make sure, you know, and I get, you know, responses from students that say those are very highly appreciated. AI can't replace that. Not because AI can't do literally what I do, but because it's the fact that it's coming from a person that makes that impactful. Having that reminder from an AI just wouldn't have the same impact as feeling like a person is actually seeing what you're doing. And so in education I feel like there's a lot of potential for it to free up our time to do the things that really improve the student experience that really have to be done by a human because it leverages that, that kind of social side of things. I also, I'm an educator. I think everyone thinks their own career is the hardest career there is, but we're correct about it. Yeah, I know of no career like education that requires us to fulfill so many different functional roles at the same time that we have to be a coach and a cheerleader, but also sometimes a judge and sometimes a police officer looking for plagiarism, things like that. It'd be a counselor for students who are going through mental health kind of things. We have to be so many things at once. And so if AI was ever able to replace all the things that we have to do, we have bigger issues, because now what are we even bothering to teach in the first place? Because if AI can replace educators, then AI can replace anything we're teaching people to do anyway. And so it's kind of a dystopian view, but it's, you know, the reason I'm not terribly concerned. I see a lot of optimistic potential for AI education is just that if AI was ever able to be a better teacher than us in every single way and serve all those different roles, then what is it teaching people to do? Because whatever it's teaching people to do is probably something that I also could do even better than those people.

When you're, when you're saying AI, are you talking about the AI that we know now, which is, you know, I don't know if you want to get into that discussion about what GPT is as opposed to a real AI, or are you talking about AI as you see it? You know, like, this is potentially the end game for AI. This is what AI will be, and that's what I'm talking about, and that's how far it'll go in education. Are you talking about the future AI or what we have now?

Yeah, it's actually a really interesting question. Just because AI has a long history of as soon as something, I need to find the quote so I can reference it more easily. There's a fantastic quote from someone like 40 years ago that captures this idea. And the fact that it's from 40 years ago is evidence of how true the statement is that everything. So AI constantly is making itself or constantly disproving itself, because we'll look at a task and say, we need to design AI to do blah. But once we've designed an AI agent that can do the thing, we no longer consider it AI because now we know how it works and we know how to do it, and it's just a computer program. A good example of that is one of the earliest, really big moonshots in AI was the optical character recognition. So that's the ability to scan in a document and have it be automatically transcribed into editable text. Most of us wouldn't consider that AI nowadays, that's just, you know, computer vision. It's. It's recognizing letters and putting them together into, you know, into editable text. It doesn't feel like AI now that it's solved and now that we can look inside and kind of understand how it's working. And so we kind of have to have a definition of AI in these kind of questions that is impervious to that, because otherwise we never really know what we're talking about. My definition of AI in this context is it's anything we create that can do something that previously only humans could do. I might even say it might not be necessary to say humans. It might be animals too. But previously something that only human effort could do. Now AI effort can do now, defined that way. A lot of things are AI. You know, search engines in some ways are AI, and the search engines in a lot of ways are AI. They're intelligently looking at your search query and finding related resources. So it's kind of a spectrum in that regard. When I say AI in this context, I'm essentially referring to it that way, though, of when we look at the things that we do as educators. If an AI were able to do that same thing, what does it mean, mean for us? So we talked around the question.

No, no, no. I think that gives us room to actually go into different directions. What we do is want to remind everyone that while I still have everyone muted and cameras off, the chat is still open. So please feel free to type in any questions you have for Dr. Joiner, people who are on Facebook watching us. You know, the. You can write your comments and I'll send them up to David as soon as I can in the little pauses in the conversation. Online learning you mentioned, I think is really well suited to incorporating AI into things. Just like you mentioned before, something I never thought of or maybe are thought of in a very cloudy way. The idea that a sort of hybrid, asynchronous kind of approach, where you're playing the video to the students at a set time and you're actually there, I thought, hey, that's a really good idea. That might actually even work in some of my classes where not a hybrid asynchronous synchronous, but a hybrid sort of like, because I have what should be a hybrid classroom, or I have mostly students in the classroom, but a couple of kids might be phoning it in. Okay, that's the wrong way to say. It might actually be phoning in. Phoning it in, phoning in through zoom and watching the class, and maybe that would actually be better. So in terms of what you're thinking about AI, how does it complement specifically remote education, synchronous or asynchronous?

Yeah, absolutely. I think that the. It's almost kind of trite to say this nowadays because technology has become so ubiquitous. I'm thinking back to when I was doing my dissertation work about 10 years ago, and in that context. So my work was on using AI to support middle school students learning how to do scientific inquiry. And at the time, the biggest challenge in that was getting records or evidence of what the students were doing accessible to an AI agent in the first place. Because typically what students were doing was they were in the classroom mixing chemicals, looking at pictures, flipping pages in books, doing a bunch of stuff that's really, really hard for an AI agent to parse. And so, excuse me, my approach to that was to create a virtual environment in which they could do a lot of the things that they would do in classroom inquiry. But the goal of creating the virtual environment for me was not necessarily to make what they were doing in the classroom better, although there are virtual environments that are geared towards that, but was really just to make it visible to an AI agent. When you're working with online learning, everything you're already doing is already digitized, and so it's already there for an AI agent to operate on. So I'm working on a book right now that's on AI and education. And some of the interesting initiatives are initiatives for AI in the classroom, which are things like robot teachers. Like literally robot teachers. Not to replace teachers, but to kind of augment the classroom. And the challenges they face are way, way higher because they're processing so much more data, and they have to take everything in visually and try and make sense of it, as opposed to in online learning, where everything is already digitized. So if I want to build an AI agent, for example, that I'll take a more reasonable one. So an AI agent, for example, that just coaches students not on content, but just on study habits. If I wanted to do that for a traditional environment. How do you build an AI agent that's aware of whether or not your students are opening their books and studying at home? The only way to do it is kind of, you know, kind of some creepy ways.

You know.

Like my daughter's school. It's, you know, the way they check to see if they studied is that I have to sign the binder to indicate that, yes, my daughter really did study. So, you know, but online it's, you know, trivial. It's. Everything is logged already. And so it'd be very easy to have an AI agent that just comes along and says, you know, hey, Lucy, you know, you set a goal to study for 45 minutes each night in that class, it looks like you've been studying for about 25 minutes each night. So just, just be aware. Maybe you can find some more time this way. That'd be trivial to do with things being digitized already. And so that's, I think, where online learning lends itself to a lot more AI is that everything is already in the language that an AI would be able to speak.

How about language education? I know you're not specifically a language teacher, but are you seeing anything towards how we work with second language learning or in our case here in Japan, foreign language learning that is maybe hindered, you know, like the whole problem about, well, what am I going to do about kids plagiarizing the writing assignments or enhanced by, by having an AI agent or an AI assistant in the classroom with you as a teacher or as a student?

Yeah, yeah. I think AI, interestingly enough, AI or language learning was one of the earlier application areas for AI. Even things like, you know, Rosetta Stone, back when it was one of the earlier teaching softw, it had some things that at the time we would have called AI. And so duolingo and all those. AI works really well into it in one way. So language learning, I think is one of the most well trodden areas for AI in education on the vocabulary side and on the kind of the rote memorization side, because it lends itself really well to the kinds of things that AI thrives on. AI thrives or historically has thrived on very objective questions and objective kind of things and kind of direct mapping between question and answer kind of things. And so it's thrived on language learning, it's thrived on math and physics kind of things where there, you know, there is a problem and a solution and it's all correct. It's had more trouble with things like the social sciences and writing where there's a bit more subjectivity. And so on the very, on the straightforward side of language learning, I think AI has a lot of potential on the real dynamic learning to be fluent, learning to actually have a conversation and participate in a culture that speaks a particular language. That's where AI has not made as many inroads because it's very big challenge. It's the kind of thing that AI might even not be suited to doing. AI is great for teaching you how to say, where is the library? It's not great at teaching you to navigate in a world that speaks a language and find the library based on clues in another language that you're seeing, you know, around you that's changing a bit. I think that within the more recent language models and things like that, there's some nice room where now you could have a much more organic conversation with an AI agent in another language. So it used to be the case, you know, if I'm learning English and I'm using an AI agent, it asks me a question, I put in my answer and it's going to tell me whether my answer is right or wrong and things like that. If my answer is not right, maybe it continues the conversation, but it only continues the conversation if I put in a response that it's expecting. And so it's kind of either got a pattern that's looking for me to play the game correctly and answer correctly, or maybe the more sophisticated ones would have kind of like a tree of if I ask the AI agent where is the library? And it responds it's by the music store and it knows maybe two questions I might ask where is the music store? Or something else. I don't have a good example, but that's the way, yeah, that's the way they used to work. Nowadays I think it's possibly much more dynamic. You could go into ChatGPT right now and have a conversation with it in English and actually have it be kind of a back and forth organic thing.

Yeah, pretty close. Just very quickly to you. I just thought of the idea, what are you planning in terms of keeping an eye out for the odd student that might be a good kid, but it was just, it was a freaky weekend and his assignment is due and he was supposed to write this snippet of code and just this one time, just this one time goes to his chat GPT4 license and ask Chad GPT4 to pump it out. How, how, how do you think you're going to keep that at bay in your world?

No, it's a really tough one. I think the first thing we have to do is we have to reconsider or go back to what our learning goals really are. Because in a lot of contexts my goal for certain assignments is not to just write code to solve a thing, but to use code as a reasoning structure over some problem. And so the problem is not write some code that can do blah, it's solve this problem using code. Solve this problem using code can mean using ChatGPT and can mean using the tools available to you. And very similar to how in math, for example, if the question was find the Y intercept of this parabola kind of thing using a calculator is one approach to doing that. But you have to know how to use the tool correctly, how to interpret the results you get out of it and things like that. And so there's still room for a lot of learning there. You're just learning to use a new tool. I think there's been a kind of knee jerk reaction to just banning it. And I understand why, because, you know, a lot of schools I think saw this coming along and realized we don't know yet how to handle this. And so for right now, we're just going to say no to it in general because, you know, and that's very understandable in the context of we need some time to figure this out because this is the biggest change, I mean likely the biggest change education has faced. I'd say it's the most rapid change education has ever faced. I would say it's the biggest overall change. But when you look at things, you know, like when the really sophisticated scientific calculators came out in the 80s, it took a lot of time before those really percolated out data scale up production people had to go out and buy them. They were expensive, they're still expensive for some reason. And so there was a rollout process. And I was doing some research for an article and found what was really interesting. I think it was Connecticut started requiring calculators on standardized tests in 1987, I think, whereas California banned them in 1993. So you had a six year period where one state was requiring them and one state was still banning them. So it was a long rollout period. Never before has something like as impactful as GPT or calculators rolled out to the entire world at the same time.

Right.

And so that's just, that's a seismic shift, not just in the what it's capable of, but how rapidly it became available to everybody. And so I understand a lot of the knee jerk reaction of we just don't want students using it at all until we figure out what the heck we're going to do about it. But I'm hoping we, as we start to move forward, we start to recognize it's a new tool. So it's a matter of teaching students how to use it correctly. Understanding what a student's success on an assignment means about their understanding in terms of the content and in terms of how well they work with the tool. And just in looking at exactly that way of this is a new tool for students to use and so what do we want them to do? The fundamental skills won't go away. I think very similar to how, you know, my daughter's in second grade. They're doing, you know, their practice on addition and subtraction, multiplication. Needless to say, they don't use a calculator at that point. But once they've shown they understand the principles, there's no reason why they shouldn't use.

Exactly.

Yeah. That just gets them the answer much faster. And we see that, you know, in the entire history of education, you know, when calculators came out, students in calculus, you know, back in the 80s, did not do the kind of calculus problems we expect students nowadays to do. It's not because calculus, advanced calculus, I don't think is advanced, and at least not at the level that would impact high school in hundreds of years. But it was. Now that we. You have a calculator at your disposal. That thing that was going to take you 30 minutes to do by hand can be done in two minutes with a calculator. And now I can expect you to do even more because you have a shortcut on that step. And so that's where I'm. I'm hoping we find the ways these large language models can be done in that same kind of way.

One of the things that I like to do with my students, because of course, I'm a language teacher, I'm not a math teacher, I don't think I could even trust myself to teach addition properly because I'd probably make addition mistakes. But to show my students that, at least from the language standpoint, there are still advantages to that, and then tie it into. The whole thing about calculators is that I asked them to do this addition thing just on. Not as part of a class or anything, but like, okay, here, here's a row of addition that you have to do. These are two digit numbers. Here's 10 of them. And you have to add them using a calculator. And you know, they got punch away on their calculator. And then I say, okay, now watch me do it on my. On my phone. As I say it to Siri, I'm going to do it twice as fast as you because your fingers cannot move as quickly as I can say it. But the whole idea that language still plays a part in this, even though I recognize that the tool is faster. The tool of being able to speak as opposed to type is something that is still. Is still really good in there. I want to encourage everyone to, to bring in their questions to Dr. Joyner, both here in the chat, in the zoom chat, and in Facebook, because I'm going to be opening up the cameras and the, and the, the microphones very soon.

And I'll give viewers a warning to see. No timing. I actually will have to leave at 11 because I have to check out of my hotel, in case you can't ask about that. I know. I guess we didn't mention this. I'm actually into. So I'm from Georgia Tech. I'm in Tokyo right now. And so I've been here for the past week visiting Jeffrey Cross at Tokyo Tech and talking about online education and CS education and really making sense of some of the interesting trends we see. I got to meet with a bunch of our students who were here in Tokyo on Tuesday night, which is a.

Lot of fun just because you might be leaving in a bit of a hurry at the end. What are you up to after this? Like in terms of presentations or books or any research that you're going to be putting out that you'd like us to keep an eye on? You have anything coming up?

No, absolutely. So I've got two books out right now. So our dean, Charles Isbel and I wrote a book, we wrote it back in 2020, it came out in 2021, called the Distributed Classroom. That essentially is a look at what should we keep from the pandemic as we move back to some sense of normalcy. Can we keep some of the advantages to online education that we developed that and proposes a model for doing that, for leveraging the stuff we're already doing for greater impact and access? Then about a year later, about last fall, I had a book called Teaching at Scale, which is essentially kind of a manual of how to teach at scale effectively. So my largest class, for example, has 1500 students. And it's a traditional kind of class. Students still do projects, they write code, they write essays, they do tests, they ask questions on forums, they get their questions answered. They come to office hours or kind of help desk hours with the TAs. And so it's just kind of a guide to how you. You do that at scale. I'm working on a new one. I don't have a full title yet, but it'll be on educational AI and really about, you know, I think there's been a challenge in education and AI historically, where educators very often don't know how AI works and don't know its potential, which is kind of why we see this reaction to GPT. It just seems like a wizard who is able to do students homework for them. And so some understanding of how it works is helpful. But on the other side, it's Also the case that educational AI has really been held back by the fact that the vast majority of people who work on it are AI professionals who kind of think they know how education works because we were all students at one point. And so we all think we know how people learn because we learned ourselves. And they don't have the expertise in education to build things that leverage what we know about learning. So the goal of that book is going to be to try and be a middle ground between those communities. It won't teach educators enough AI to be able to go out and build an AI on their own, but it'll have enough on AI to help educators be in the conversation and communicate things to engineers about why certain things will and won't work. And then for engineers, it tries to give enough education so that when they're building something, some of just the very the things we take for granted can get injected into the design of such systems because they actually are aware of things. Education is an interesting one for engineers especially, I think, because in almost every other task that we as engineers and AI designers are trying to do, we're usually trying to make things easier on the user. And that's usually the goal is make it easier for the user to do what they want to do. That's not the goal in education. And I see a lot of tools in education in AI. In education, where the AI, instead of teaching the student, kind of replaces the student, and it's a case of, yes, the student was able to solve that problem 10 times faster with your AI agent, but if you took away the AI agent, they wouldn't be able to solve it at all. It does too much. So it's not a learning agent, it's a doing partner, which is a fundamental difference.

It's not a bad thing, though, because that's like saying, yeah, you drove here, but if you didn't have your car, it would take you three hours to walk here. What's your point?

Yeah, I think that's exactly right. It depends on the task and depends on what your learning goals are. If your goal is for a student to be able to find the wider depth of a parabola, then the fact that they're able to do it 10 times faster with a calculator is a good thing. If your goal is for them to be able to structure their thoughts on some debate topic and come up with an argument for why they fall on one side of a debate and they use an AI agent to do most of it, and they can't do it on their own and can't explain why that essay worked. They haven't demonstrated the goal. So it's a case of what do you want them to be able to do with the tool versus what do you want them? What portion of their own reasoning do you want to be in that process?

I don't see any questions yet from anyone, although now everyone can open up their microphones or their, and their cameras. Although I would like you to raise your hands if you could if people start asking a lot of questions. So while we're waiting for any more questions to come up, you made me think of something that when you were talking about the, the fact that the, the AIs are coming up really fast and everywhere around the world. I think I was reading an article and unfortunately I can't cite it specifically, but somehow with OpenAI's help, a major university in California might have been Stanford, somehow took the code and created basically the equivalent of what OpenAI had maybe a couple of months ago when they started up chat GPT for less than $600. For 600 bucks they took open AI's code and now they just have to train it and it'll, you know, it. So basically, you know, if you have the hardware and about six hundred or thousand dollars, you could have a chat GPT sitting in your basement too. And one of the questions that we were talking about is how, or actually two of them. I want to tie both of them together so we can cover them if we can. The, the problem of making AI or advanced technology available equitably to all in the world. I mean, ChatGPT is going to affect the world. It rolled out across the entire planet at the same time. But then we still have the divide between north and south, even within the, you know, the, the wealthy north. We have digital divides, we have financial divides. And what does this mean for being a human learner coming up in this age? If you're in the north or if you're in the south, there's so many ways to look at this. How do you, how do you sort of see all of that coming together?

Yeah, absolutely. I think it's a key thing that it is really nice that GPT launched in the way it did, that it's just available to everybody and there's no traditional kind of constraint built into it. There's a free version, everything like that. But that still doesn't, you know, that's a minimum, necessary kind of step. If we, you know, it's very tempting for people to say, well, it's available to everybody. And therefore the digital divide is solved. Which ignores the fact that depending on your prior experience with technology, you're more suited to be able to take advantage of it. If you have parents who are very technologically savvy, they likely can expose you to it more quickly. And so we've seen that across lots of innovations. There's a really fantastic book that I recommend for anybody who's working in this area. It's called Failure to Disrupt by Justin Reich at mit. And it's a look at why technologies and education do not have the impact that they potentially could. And this is one of the big reasons why that happens, is that we can't just put technology in people's hands and assume it's going to be universally accepted. Different people have different levels of readiness for is the case where I'm more optimistic about the potential good of things like GPT in these models is that it's the principles that underlie. All these things are public knowledge. They're in academic papers everywhere. So the reason why Stanford was able to do it, as far as I know, wasn't because, you know, OpenAI gave them something secret or came to an agreement. It's because the principles were already publicly known. And so it's not like the Google search algorithm, for example. It's not like the recipe for, you know, Coca Cola. It's not like KFC's six herbs and spices kind of thing. It's not a closely guarded secret that allows some companies to maintain a competitive advantage. The principles are all out there. It reminds me a little bit more of what was the story, the story about the. I think it's the first person to run a sub 4 minute mile and you plot the, the world record times for mile times and you see for a long time they didn't break through four minutes and then someone did. And then right after that a bunch of other people were able to too because suddenly the potential was there. People were more aware of what could be done. I think that's really what we're seeing with GPT is that once OpenAI released this and people are able to see what it's capable of, it kind of stirs up some enthusiasm to do more work in this area. But there's no secret sauce to it. You know, the principles are known now. There's things that you can be better and worse at. So what a lot of people don't know is that the math and the algorithms that underlie GPT are published and owned. It costs, as far as I know, millions of dollars to train the model using those known principles. And that's mostly in compute time. The best way to understand a little bit more about how it works is that, you know, it's a pattern machine, so it's detecting patterns just like the many AI agents have done in the past. But in the case of GPT, it's detecting patterns where its data set is millions and millions and millions and millions and millions of pages of articles and books and blog posts and lots of stuff. And it's being trained on thousands and thousands and thousands of computers for thousands and thousands and thousands of hours at a time. And so it's just a scale kind of thing. And so then that all cost a whole lot of money. And so, sorry, I don't remember how I got on this. The idea, though, is that the knowledge is common. The resources necessary to marshal it can still be a challenge. And like when Stanford builds it in someone's basement kind of thing, you're able to build kind of mini versions of it that aren't as sophisticated, but they don't have to be as sophisticated because GPT is really targeting a wide range of knowledge and skills and tasks. And if you're coming into it, as one of the projects I'm working on is, and saying, I just want to build a GPT that can just answer questions about my syllabus, I don't need to train it on the massive quantities of data out there. You leverage some of that data, you leverage the training that's already been done, because that's kind of what gives it more of a natural language understanding. But you're able to kind of really target it at a more narrow skill, which can cut the cost significantly.

What I thought would be interesting, as I saw that, and you reminded me as we were speaking, is that wouldn't it be really cool if just like open source software, Stanford builds theirs? And it's specific to. Stanford has a great graduate school of business. So this is all about business, business communications, answering emails, writing emails, and then, I don't know, Cornell writes something that's specific to a database on astronomy, and these two can talk to each other. So if you're accessing the Cornell version of an LLM and the Cornell database knows, hey, I don't know that much about that, but I know Stanford does. And immediately through an accessible network, actually these distributed nodes of knowledge that were programmed to do certain things, like maybe somebody will do something on learning French and somebody will do something on learning Esperanto, and they know where these databases are. And they can go and get that. That's not beyond imagination, right? That could actually.

Yeah, it's kind of a new Internet. I mean that can already exist in sources now it's just a matter and that's, that's why this, working in this area is exciting and also terrifying at the same time. And terrifying not kind of the existential kind of way, just of the. I think we've developed a real kind of fear of trepidation of what we talk about now. Some things I've said today probably won't be accurate a month from now. I don't know what they'll be but the thing, the area is changing so fast. I gave a couple interviews a couple months ago before GPT4 came out for ChatGPT commenting on how to detect a little bit about if students are over relying on GPT and just noting once you've had to answer the same question a few times it's pretty easy to pick up on patterns in how GPT answers GPT4. I mean it's still true for GPT4 but the kinds of patterns are now far different. And so you know, a lot of what we say becomes outdated by the time we say it. One of the things that you know, people have noted about GPT is you know, if you want to avoid GPT, you know, completely rendering all your assignments to be, to be obsolete. Ask about recent events because GPT has no knowledge. It's trained through 2021. I think yesterday they released GPT plugins and GPT plugins give it the ability to interact with more recent sources.

There you go.

So all that kind of, you know, and you know, it's, that's why it's a little bit terrifying. Just as teachers, you know, we, we put a lot of work into our assignments and we're, we're operating individually and so it might take a couple months to really retool an assignment and by the time we retooled it now what we were trying to avoid. So yeah.

Have you had a chance to do any work with GPT4?

I'm actually using it a good bit for designing my educational AI class.

And how do you.

Yeah, I would define it. It's been interesting, it's been a little reassuring in a lot of places because it's really. My comparison is it's a real, it's at the level of a really hard working teaching assistant, not a really smart teaching assistant, A really hard working teaching assistant that I can trust to go out and comb large volumes of information and Bring back some raw stuff to me. But there's a whole lot I still have to do to kind of massage it into real things.

Is it still prone to making stuff up?

I think it's less prone to making stuff up than it was, but it still has that in it. It has no knowledge of truth. And so some of the things I know, the earlier one got wrong, the more recent one has gotten correct. So that's been an observation. There was a funny one where I asked it for some of the most significant initiatives in intelligent tutoring systems, which is one of the topics I talk about in my course. And that's something it does a pretty good job at. It's good at generating a large volume of examples for you that you then have to go through yourself. So I'll ask it things like, what's a better way to say this? You know, say this sentence and It'll give me 15 options, 13 of which are awful, but two of them are good. And the process of getting 15 options from it and choosing the one I like is far more efficient than it would have been for me to just sit there until I was. Stroke of brilliance.

Okay, I'm going to stop you because I don't want you to get into trouble with the front desk clerk. It is 1057. Last chance for anybody. I'm going to check the Facebook comments stream. Nothing. There was nothing in the. The Zoom chat stream. Last chance as I start to wrap this up. So I'm going to start wrap this up. Dave, I. I wish you had been able to stay. I think people eventually would have had. I think people are just stunned with your information. They're just kind of like, oh. And they can't process it fast enough to actually come up with the wrong question. But I want to thank you. That was a really fascinating morning, and I would like to welcome everyone to turn on their cameras and maybe turn on their microphones just to. To let see your. Your happy faces and. Or maybe your stunned faces as. As we say goodbye. Oh, there's Jerry, there's. And Julie and, and Catherine and. Oh, Jeffrey. Hey, Jeffrey. So thanks very much. You probably have a little bit of a grace period. If you drop in there, like 1105 or something. They're not gonna, like, charge you another night just right there. But thank you again. I hope I meet you one day.

I'm.

I'm really shocked that your program is only $7,000. That makes me go, gee, I wonder if I could do that. But then remember, you're doing computer science you're not doing language education.

I mean, we, we also have some onboarding courses. We even have a little curriculum that says, you know, if you want to do the masters, but you don't have the CS background necessary to get in, here's what we recommend you do. And they're, they're MOOCs, they're on EDX. And so everyone, the goal is to make it so anyone has a pathway to, to get this kind of education. It's just a matter of building in the different entry points.

It does a bad job with empty. It starts making stuff up. I know people like that.

Yeah.

And you're looking at one of them. I just make stuff up as I go along. So I'm not much better than chat.

GPT3, I mean, I was about to say is that one of the best skills to have is to be able to convincingly make stuff up. The problem with GPT is that then you have people repeat it and then the person is held responsible for it.

But then we have that problem with like, people in general. I mean, that's, that's what memes are all about. Listen, thank you very much. Don't get into trouble with the front desk. Everybody, let's, let's wave good morning and have a good afternoon. David, Jordan, thanks for joining us. And everyone, thank you for joining us. Thanks, David.

Yep. See you later.

Okay, bye.