Chris Baldwin built TruthByte, a true/false trivia game that crowdsources questions for AI model evaluation. The frontend is WebAssembly compiled from Zig, the backend runs on AWS Lambda and DynamoDB. Players can submit their own questions to contribute to future AI training datasets.
Chapters
Last month at Open Source North, I attended a talk about measuring generative AI. One of the presenters showed a test set that he uses for model evaluation after it's been trained. Is asparagus a food? That's an evaluation question? That's when it clicked for me. Anybody can start to shape AI. We're in this massive wave of adoption, and there's a lot of people that feel left out. But through model evaluation, anybody can participate. That idea led me to build TruthByte, a true/false trivia game. For the engineers out there, the frontend runs in WebAssembly, written in Zig and JavaScript. The backend is entirely serverless: Lambdas, API Gateway, DynamoDB. Well, right away I want to change the color because it's available. Let's pick a good one. I kind of like this blue. That's nice. All right, so let's kick it off with arcade mode. So right there, had to ask DynamoDB, like, hey, give me some questions. So has any Major League Baseball team gone undefeated? I'm gonna guess that's a no. Is the World Health Organization a government organization? I don't think so. Has a woman ever won CMA? I think so. Uh, passport card? Yeah, it is. Uh, The Hobbit was not part of the Lord of the Rings trilogy. Uh, is soy sauce and Worcestershire sauce the same thing? Obviously not. Uh, can a baseball player catch a ball in the stands? I think so. 5 out of 7. Okay, so not perfect, but pretty good. Cool. So let's continue. There's categories mode, which every question is tagged with some categories and difficulty. So we can select, say, just some science questions. Can we have multiple foreign keys in a table? I— do you know, this is really challenging me right now. This is a challenge. I'm gonna guess the answer is yes. Uh, is there a bird faster than a cheetah? Yeah, there are birds faster than cheetahs. Are age spots the same as liver spots? Uh, no. Can a cow get up after being tipped? Yeah, I would imagine. Maybe. I, I think so. Is the red panda related to a panda? No. Is a king snake immune to rattlesnake venom? Sure, it's a king snake. Is Harry Potter and the Escape from Gringotts a roller coaster ride? I think it is. 6 out of 7. Hey, that's pretty good for science. I don't know how all of those questions are necessarily science. But I think you get the idea. All right, and then the last mode is Daily Mode. Daily Mode is my personal favorite because I think that dailies are a great way to get people to participate regularly. I know I am a sucker for Wordle and Connections, so I thought, why not put Daily Mode in the game. And I don't actually know the answer to any of these, so I'm just going to quickly blow through them. And I got an F, right? I guessed. I, I should get an F. Uh, 5 out of 10 questions correct is statistically what I should get for guessing every time. Right? So that's the gameplay modes. The last thing to really show off is submit. And submit is, on the down low, the most important thing available. So does my Lambda run on my server? And I'm going to tag this with AWS. And the answer is no. I'm going to send the question, and I can keep submitting questions or I can go back to the game, and that's it! That's all I've got so far. That's TruthByte. This whole project is hosted on my GitHub page. Chris D Baldwin is my username, and it is TruthByte, so you should be able to find it pretty easily. It'll be linked in the description below, of course. So, Thank you to AWS for having this challenge, this Lambda hackathon. Yeah, I think the documentation is very good. Again, it's somewhat obvious that I used AI here, but like, let's, let's talk about that maybe. I don't know, like, how did I build it? Is that even worth discussing? 'Cause the how is like pretty interesting, but it's not really the what. So I think we'll leave the how out of it. The what is that here is Truthbyte. This is an open source, run it in your own AWS account, serverless website that allows you to play trivia questions, and gather questions from experts in any category for LLM evaluation in the future. That's it.
