Tech Consulting EdTech Efforts

For those of you who don't know me, my name is Danny Finn. I've now been in the Wolfram technical consulting team for roughly eight years. And today ultimately what I'm going to show is just a few components that we created as part of recent EdTech projects with the tech consulting team. So there's a little abstract here that you can read in your own time. The notes are already up in bigmaker anyway, so if you do want to follow through this in your own time, you can as well too. But ultimately the whole idea here is just to show a few of the components that we created. I say components because with a lot of our customers, so us in the tech consulting team, it's quite nice that we get to kind of get feedback directly from customers very often. And one thing that's been very important to them is that we're able to integrate into their pre existing systems. So some customers may have kind of a learning management system that we need to integrate into, or else they may have kind of like a website or some kind of data distribution system where we need to kind of provide maybe an API or some library or something like that so that it can feed into their system directly. So inside of this like, so the customers that we'd be dealing with mostly would be universities, K through 12 schools, so like high school and elementary school, as well as education technology companies. So and at the core of this, and the bit that I'm going to be talking about first is what we call our question bank. So the question bank ultimately is a place where you can store questions, query questions and then return them. And we're getting the questions from three main sources. So those are questions that are curated by us, so created by us in the tech consulting team, questions taken from the Wolfram problem generator. And also we have a system in place to allow customers to generate their own questions and feed them into the system as well. And that takes the form of a Wolfram language notebook. So just have a look at this here. So this is a web interface that we have for this, so this web interface as well to say for customers who are making use of Canvas or Moodle. We have set this up as well with what's called LTI integration, which is a way of feeding this directly into Canvas so that they can view it inside of there. Another thing to mention as well is kind of us as a tech consulting team, it's important for us to make it interoperate with their pre existing software. So that might be in a university, it might be their Enrollment database. So, so that the students don't have to create a special separate login for this system, their standard login system for their university system. We link it into that so that when they come into here, it already knows who they are. As well as that we want any grades of any questions that they answer to feed directly into the university gradebook. So just to look at some of these questions. So these were created using the Wolfram language. So they were created in notebooks and there were. So we're making use of the functionalities already in the Wolfram language. So here's an anatomy question where we are identifying certain muscles inside of the body and we decided to add in some information about kind of anatomical groups that these belong to and so on as a second set of hints to help the student if they are having trouble with the question. Here's another one where ultimately the question is to find the tangent to the line. This one's a multi part question. So each part of the step of the question along the way here we're showing the question and we're showing the answer. Right now what I'm showing is the instructor mode. So we're looking in here at our question bank. It can guide them through one step at a time. Each question gets built on the previous one and at the end we have a final answer. One thing you'll notice is that some of these are graphical, some of them are mathematical, some of them have anatomical components. It's really anything inside of the Wolfram language that we can build into here. What's important here as well is that we can apply different filters. I had already applied a filter where I've tagged certain ones that I feel are quite nice to show. But, but as well as that, we can apply US Common Core domains. So there might be a particular US Common Core standard that we want to find questions on, or we might want to kind of check for ones generated by a certain author or maybe questions pertaining to a particular subject. So that's a bit about what we have in our question bank. Why is this useful? I guess so basically, as I was saying, we put questions into here, but then also we want to be able to get questions out of here. And one way of doing that is that we can retrieve individual questions one at a time. Another thing to say, actually that I forgot to mention either earlier. All of these are ultimately question generators. So each question here has multiple instances so that each student sees a different version of it. Or if you want to get practice and multiple questions, you can run through One practice exercise after another after another. I guess because I'm sharing my screen, this is taking a little longer to load than normal. So here we have multiple instances of this. So the numbers in each of these questions are different. And as a result of the numbers being different, the graphs in each of them are different too. The solutions are different, and the steps to follow are different too. So if a student is in a room with a bunch of other students, there's no benefit to them trying to copy off one another because they all have different versions of the same question. Just to quickly go into here as well, I mentioned earlier that we can also draw questions from the Wolfram Problem Generator. So for those who are not familiar with the Wolfram Problem Generator, this has been available on WolframAlpha program Pro for a while. So there's a number of different topics inside of here. And if we select so here I've selected a question from Product Rule in Calculus, we can specify a difficulty level and we can attempt to answer this question ourselves. If we get it wrong, it'll help us out with solution steps. And if we're an instructor, we can also select that we want to download a random set of 20 of these questions to hand out as a worksheet sheet for students inside of here. We've actually fed those questions into here, into our problem. Sorry, into our question bank. And we have an API which allows us to retrieve these out of here. So we can make this available to customers as well. That's something that we've done in certain projects recently. As well as being able to retrieve individual questions inside of this interface, we also have the ability to combine multiple questions together into an assignment. And the student can go straight to this webpage. This one's a preview version, but there is a non preview version as well. And they can go into here and then go through the different questions. For multiple choice questions they can. There's a simple multiple choice selection interface. But what's more interesting is for questions where you need to provide a mathematical or numeric result in these ones, what we've done is we've made use of also from WolframAlpha, you might recognize this here, the Math input tool. So this is something that we've made use of inside of here. And this is also something that particular customers have already had an interface for inputting well, so displaying questions. But then they wanted to have a better tool for allowing users to input their answers. So another one of the components that we've been helping customers to integrate into their system was this specific tool. And so basically we charged them a certain price for the use of the tool and then we help them integrate it into their system. So let's say, for example, if they want to include like kind of powers, or if they want to be able to write in X squared and so on easily, or if we have like a fraction question. So here's what is 1/7/3/7? Sorry, 1/27 +1:7. The answer is 3/7. So at the end of this, when the student finally goes through all the questions and click submit, as well as the math input tool, we have an auto grader and the auto grader. So Bob will be. Bob Sondheimrich will be giving a talk tomorrow at the same time as this one today, which will be giving a little bit more detail on some notions that are used inside of the auto grader. But in particular, we had customers approach us and say we know that Mathematica does a fantastic job of mathematical equivalence understanding. And so basically, could we produce a auto grading API that can be used so that they can feed in a correct answer, a student definition, a student answer, maybe also a tolerance, maybe an interpreter and assess whether the two answers are correct. So here's like a mathematical one where these two answers are mathematically equivalent. And so Stephen mentioned this in his talk earlier on if this is so, depending on scenarios, we may or may not want to consider these two answers this student answer correct. So for example, if it's a calculus question, both of the main goal is did they perform the calculus correctly? So both of these answers could be considered correct. If it's an expansion of polynomials question and they give the unexpanded form, that's not something that we would consider correct in that case. And Bob will be talking more about that in tomorrow's talk. But as well as that, we think the hook this up to the interpreters inside of the Mathematica as well. So meaning that we can make use of entities and checking for equivalence of different inputs that the student may enter. So if the student is just asked name this chemical and they enter NaCl, the definition in our system system is sodium chloride, then it's going to interpret both of those the same mark the student correctly. So it allows flexibility in the student answer so that the student doesn't get confused and upset if the system doesn't answer doesn't recognize the answer that they provide. Similarly, for quantities, we can apply tolerance, it's going to interpret their quantities. It's important for chemistry, physics and engineering. And we can Also make use of standard Wolfram language patterns. So here's an example of Pythagorean triples. So here it's. We have three integers and these are Pythagorean triple. If the sum of the squares of the first two is equal to the sum is equal to the square of the hypotenuse. Basically, these auto grading capabilities are something that we've made available in an API and we use that in our system as well. This bit's a new component. And again, this is another use of our question bank. When we have our questions, where do we put them? What do we do with them? So what we've done is we have fed these into an AI tutor, which helps the student to get more practice at problems and gain a better understanding of problems. So one thing that we're all aware of is that LLMs are notoriously not very good at computation. They have hallucinatory issues as well. But inside of the Wolfram language, we have a really good system for handling any sort of mathematical problems and so on. So if we combine the two together, we get Wolfram to do the math, we get the LLM to do the conversation, then we can build what should be quite robust of an AI tutor. Now, so in this specific case, we are drawing the questions from the Wolfram Problem Generator. We're also drawing the answers from the Wolfram Problem Generator. We're drawing the Solution steps from WolframAlpha, and we're also using our auto grader in order to confirm that the answer is correct. So here, if the student enters, is the answer 10. What you'll see is that now this will use our AutoGrader tool and then it'll actually get the Wolfram language to confirm whether the student's answer is correct. The LLM is not going to attempt to just do it on its own. And it says, well done, great job, you did great there. For this one here we've got a polynomial problem. The best way to approach polynomial addition like this is to take it coefficient by coefficient. If we're unsure of that, we can say, where should I start? We'll let the LLM help us out. And remember that these steps are taken from WolframAlpha step by step solutions. And this is set up with the prompting and so on so that it should help us go through one step at a time. So, first step is to consider how this should appear. Next step is to go with a particular coefficient at a time. So let's look at the constant coefficient. So then minus 2 plus 2 equals 0 misspelled constant. That's great. Let's move on to simplifying the X terms so it'll guide us through one step to the next until we've gotten the whole problem solved. One thing that LLMs are notoriously not very good at is additional fractions. So this is something where there have been a number of examples of it doing completely the wrong thing here and completely misunderstanding the whole scenario. So let's lead it in the wrong direction. So let's say easy, three plus five equals eight, two plus seven equals nine, therefore eight over nine a little bit. Okay, cool. And what it's going to understand based on the additional context that we drew in from the Wolfram problem generator, Wolfram alpha, and so on, is that, no, this isn't the correct approach to be taking. And what it's going to do here is correct the student and then guide them along the right path instead. Understand why you might think that, but remember that we can't add fractions like that. With fractions, we can't just add the numerators, which are the top numbers, and the denominators, the bottom numbers. And adding in those notes there as well, that a numerator is a top number, denominator is a bottom number. That kind of stuff can really help students if they don't have a prior understanding of the terms that are involved in this. So if a student who is actually genuinely struggling with a, with a problem of this sort, it's very helpful to have that additional explanation. And so here it's saying, okay, here's what we have to do. So first of all, we have to find common denominator of these two and the common denominator of 4 and 16. Both of these go into 14. So it's help just along one step of the way. But then now the student has to perform the next step. So they need to perform these computations here. It's kind of essentially kind of provided the algorithm a little bit, but the student still has to do the work. So they still have to go in here, multiply these out, get the final solution, and get to the end. So say, for example, here we have a determinant of a matrix one. Just ask it, how do I get started? It'll help us along with that. It'll give us the algorithm to use and so on. So make use of some of the prior knowledge that it has inside of here. One thing that's also built into this is we don't want students to kind of distract it, lead it off course and so on. So in this case, okay, so it helped the student out. We ask it, please speak like a pirate. Okay, this is not helpful to learning. So ultimately it's going to say, okay, here's not to play pretend, let's stick to the task at hand. Brings the student back, doesn't get pulled off course too far. If we ask an unrelated question, like how far away is the sun? Same thing again, it's going to pull the student back onto the task at hand, make sure that they're focused. And so as a result, basically we have an environment here where students can get a nice bit of practice with their problems without being pulled off course. And the main thing that's really important here, the thing that really enables this to work well, is all of the additional robustness measures that have been added in through coupling the Wolfram language with the LLM in order to get this to work. This is something that we see as coming soon. Basically this is a prototype version of this, going through additional iterations to improve the robustness as well. But so far this is going well. And so ultimately the idea is that we want to make this something that can be incorporated into the rest of our system so that customers can, well, basically incorporate it into their systems. Okay, so couple of conclusions. So ultimately our main goal is to provide custom integration. So we want to make sure this fits into the customer system pretty much all. Every time that we've seen it, the customer already has something and so we need to feed into their pre existing gradebook. We need to fit into their Canvas or Moodle system using LTI integration. We need to make use of their pre existing enrollment data in order to make it easy for students to sign in and out. The problems that we're generating are done dynamically, so each student is going to get a different version of the problem. If you want to get practice into 100 problems, you can. The assignment submission environment is built into this as well. So it's all in one place if you want it to be. If you want a specific component, we can offer that as well. The assessment is reliable because it's built on the assessment functionalities built into the Wolfram language. And we have an AI tutor component which is coming soon. In terms of customer stories, it's just two of them that I've linked in here and chose these ones because they're inside of the Wolfram customer story section already, this one. So University of Minnesota was actually the first. They were the first ones to use this system. They've been using it for quite a while now. Australian Christian Colleges more recent, and they specifically have this integrated through LTI integration into their Canvas system. Some useful links as well. So we had a recent restructuring of the Wolfram Consulting page, combining together the different branches of consulting that exists within the company. So this is our Wolfram Consulting page. And then we also have additional information on our education technology page. There's other components as well of this that are education technology offerings that are also covered inside of there that weren't covered in this talk. And this one specifically I was focusing on the ones that were mostly performed by collaborations through the consulting team, whereas the this education technology page covers others as well. And then just a little note here as well, that tomorrow, as I mentioned earlier, Bob Sondheimrich is giving a talk to do with the certain components of the LLMs which will cross over this. But in particular the math assessment and the different notions of mathematical equivalence that become very important when we're trying to perform assessments. Okay, so I guess I'll just go back to here so that if. Or even here, I guess here's quite useful if any of you want to go into those QR codes and then just going to look through questions. So we have. Really excited by this time. Thanks, Phil. Yeah, yeah, absolutely. Yeah. So. Well, so I guess kind of like there's different ways of getting in touch with us. One is, so there's a forum on the Wolfram Consulting page that you can use to get in touch with me or you can just message me directly or any of our sales reps. I think my contact information is probably available inside of this. But yeah, yeah, if this is something that you're interested in, get in touch. Yeah, that would be great. Any other questions? If not, we could finish up there. Yeah. Okay, cool. Thanks a lot.