In this vlog Daniel Bourke, co‑founder and machine‑learning engineer at the small startup Nutrify, walks viewers through a typical workday in his home‑based office. He introduces the team (head iOS engineer Josh, labeler Joey, and guest "Pizza Dad"), shows the office layout, and explains the company’s mission to help users learn about whole foods via an iOS app. The video details his daily routine – morning tea and breakfast, a brief planning session, exercise “snacks” throughout the day, and a structured to‑do list. Daniel spends an hour manually labeling food images, then moves to model training using a train.py script that fine‑tunes a Hugging Face/Timm model on custom data, tracks experiments with Weights & Biases, and runs the job on a Titan RTX while a newer RTX 4090 handles large‑scale text inference (NER over ~50 M rows). He encounters a dimensionality bug, debugs it, and discusses the broader workflow: data collection → labeling → training → evaluation → deployment. Lunch is a home‑cooked meal with the team, reinforcing the startup’s casual culture. Daniel outlines the product roadmap: moving from single‑label image classification to multi‑object detection, building an automatic data‑labeling engine inspired by Tesla’s self‑driving data pipeline, and eventually merging image and text models into a generative VLM. He emphasizes the importance of experiment tracking, reading (Designing Machine Learning Systems, Apple’s Overton paper), and the analogy of model training to muscle building. The vlog ends with a night‑time prediction/evaluation run and a reminder that real‑world testing is essential.
Chapters
Come on, I'll give you a quick 1-minute intro to the office. Welcome to Nutrify HQ. I'm Daniel Bourke, machine learning engineer and co-founder of Nutrify, which is an iOS application to help you learn about whole foods. This is Josh.
Hello.
He's our head iOS engineer, so anything you can feel and touch on the iOS app on Nutrify, Josh has probably built that. Here's our planning station. Not sure if there's confidential stuff on there, but we've got 2 whiteboards. This is our data labeling station. So the whole idea with machine learning is that you draw boxes around food and you try to get a model to replicate that. Our model is image classification right now, but we want to upgrade it to object detection. And this is my office. I'm going to put on my hoodie because it's been getting a bit cold lately. I have two computers upstairs. I'll show you them later. We've got an article, "What We Learned from a Year of Building LLMs." Shout out to Omar. This is my technical bookshelf, and one of my favourite books, Designing Machine Learning Systems by Chip Killen. Probably the most important book for a machine learning engineer to read. Anyway, that's been a quick intro. Welcome to a day in the life of a machine learning engineer. Nutrify's office is a converted lounge room in my house, and my brother lives about 1 kilometre down the road, so we usually start between 7 and 8 AM. I often get up and make my girlfriend and I tea, and cook breakfast for the 3 of us. We eat quite simply, mostly whole foods, not too much. This morning's breakfast was 3 rice cakes and 3 eggs with a pinch of salt. We have our morning meetings outside in the sun, and here we catch up on what we did on the weekend and discuss what we'd like to do over the next couple of days. After breakfast, around 8:15-ish, we get into it. The first thing I usually do is write down the top 3 to 5 things I'd like to get done in the day. On a notepad, and for the first half an hour to an hour or so, I read an article on machine learning or a technical paper. To break things up during the day, I do some form of exercise snack, which may be bodyweight movements every 45 minutes or so, or a 10 to 15 minute walk every couple of hours. Today was 5 pull-ups, 12 push-ups, 15 squats every 45 minutes. It takes about a minute or two and adds up nicely by the end of the day. From around 9 to 10:30, I spent just over an hour manually labeling food images for the new Nutrafi Food Vision model. I like to spend about 1 hour per week manually labeling data so I know it's of high quality and I'm familiar with the dataset we're working with. After data labeling, it was time for another exercise snack. This time I swapped the squats for squat jumps. Round 2 of movement done. Model training time. So this is how I train models. I got Nutrify, Food Vision. I got a script called train.py. Can add some notes. That's about the experiment that I'm running. So I labeled about 300, maybe 400 manual images. There's the model name. This is from the Timm library on Hugging Face. Train the body means I want to fine-tune the entire model. And 50 epochs. So the WB note stands for Weights and Biases. All this information here, if we hit the magic key, as long as I've got no typos, this is going to upload. So GCP is connected, Weights and Biases logged. So this is— Weights and Biases is going to track all the metadata, the model artifact, uh, metrics, all that sort of jazz. There we go. View Project Run here. So that'll appear down on my MacBook and training is going to happen on this PC here. So there we go. Downloading artifact, all the files, and we'll see how this runs in a few minutes. So it turns out that this training run actually failed. Okay, I think I found the issue. Dimensionality issue here. When I calculate the test accuracy, dimension out of range, expected to be in range of this. So there's a dimension issue. I don't think I need this torch argmax. Maybe we'll go find out. Oh my goodness, these roasted Sweet potatoes and change your life. But at least it's simple. So we got lunchtime at Nutrify HQ. Naturally, being a food ad, we like to cook our own food. So today we've got some ground beef, a little bit of butter, seasoning. We've got roasted sweet potato, guacamole, and roasted broccolini stalks. And we have two special guests today. We have Joey, who's none other than our residential food extraordinaire labeler. Joey's helping us turn Nutrify from image classification to object detection. And we have none other than Pizza Dad. Oh, hello, hello, hello, hello. Good day to you all. Pizza Dad, you may have recognized him from some of my courses. Helps with computer vision teaching. But today you're sandwich dad. So while the food's been eaten, models are training. Bon appétit, family. Thank you. Cheers. Tuck in, tuck in. Cheers. During the afternoon session, we had Joey on the food box labeling, Josh working through the iOS app, and me refining the data pipeline. If I could recommend a read that most mimics what I do day to day, it would have to be the 2019 Apple machine learning paper called "Overton." You see that person? That's me. Someone who monitors the data, the models, and sees which part of the data needs to be upgraded in order to improve the production system. And as I've said before, We're trying to go from single foods per image to multi-foods. So my main goal now is building an automatic data labeling engine to identify and label multiple foods in images. In other words, copying Tesla's data engine for self-driving cars, but for food images. We still rolling? Are we in 4K? It's been a brilliant afternoon at NutriBio HQ. So, uh, fun thing is, as a machine learning engineer, is there's not actually that much model building. A lot of it is once you build a model, you kind of just refine the dataset. And so this is my workspace for Weights and Biases where I track all of the experimentation. And as you can see, each one of these runs is a mini experiment. So this is the one that's running right now upstairs. I'll go show you that in a second, but there's the loss curves that are going down, which is good. Accuracy curves going up, exactly what we want. And there's the model. I've got all the hyperparameters saved here. So basically, if you're running machine learning model experiments, you pretty much want to be using something like Weights and Biases. There's a lot of experiment trackers out there. I just started using Weights and Biases and I love it. The model is from Hugging Face, but we fine-tune it on our custom food image data. Off to Josh. Josh, what are you doing? I've just been chilling. No, I'm working on text detection. What do you mean? There's just— Daniel changed the names of all the foods so, so I can better interact with these models.
So now it's just ruined a lot of matching that I had.
Image matching.
That's what I'm doing. Text and images.
So we have two models that run in Nutrify. If you take a photo of something, there's the Food Vision model which will identify what food is in the image, and that works on image classification. But some things aren't image classifiable, they have text. So we also run a text identifier over the top, and eventually we'll probably look into something like combine those two into one model. So right now there's two models, eventually combine that in something like a generative AI VLM type thing, but that's future. We come over here, speaking of model upgrades, our in-house grocery boxer, Joey. Joey, what are you working on?
Mate, I'm just going through about 10,000 images of food, doing the manual labor here of trying to source out what is individually seen on here so the model can pick up instead of just one food at a time, it'll do object detection and figure out that, hey, for this example, we've got fries, tomato chutney or salsa, We've got a chicken parmi and some rocket on the screen. So instead of picking up just one, it will be able to detect everything on the page at once.
Multi-food. Multi-food. Now, come upstairs. I'll show you where Nutrify's brain lives. This is really cool. You know what we realized today is that this house has solar panels, and solar gets translated into electricity. And so you could say that the computer here It's been powered by the same energy as food because sunlight hits a plant, grows food, circle of life, like the Lion King. But Nutrify's Food Vision Model is training on here. You know those loss curves I showed you on Weights and Biases before? That's what's happening here. So sunlight into model weights. How cool is that? And so we've got some training logs here. Um, testing epoch 7, training epoch 8. I could optimize this by maybe only evaluating every 5 epochs or so like that, but I like seeing training and testing results. And so we've got the Titan RTX. This computer is about 4— uh, NVIDIA Titan RTX, that is. This computer is about 4 years old, maybe closer to 5 now, but it still runs like a gem. All of Nutrafolly's models, uh, that have run on device are trained here, and If you look down here, we've got an even better deep learning PC. This is quite new. It has an NVIDIA RTX 4090, and so this is the newer generation. All the specs have been bumped up compared to my original deep learning PC, and you might be wondering, why don't I train the models on here? Well, I've already tried that, and the really cool thing is that Neutralize Food Vision model It takes about 22 hours to train on there in the current training scheme that we have. It takes about 10 hours on here. And so, Daniel, why are you training on the Titan RTX but not the RTX 4090? Well, I'm going to get to that eventually, but the 4090 is currently doing the very important work of inference. So it's going over an extremely large text dataset of about 50 million rows that are sort of sentences related to food, about a billion words maybe. And running a named entity recognition— I almost forgot, I know it's just NER— NER model over all of those rows of text to extract the food terms. And why are we doing that? Well, we want to create an inverted list of term occurrence. So say the word apple occurs 10,000 times, we want to make sure Nutrify takes care of the top 10,000 most popular foods as judged by the internet, because that's what we want it to be, a food app. And so this is a really cool thing about having two computers is that one can be training a model and one can be performing inference to enhance a dataset. And eventually we want to follow this pattern for every process. We want one to be training a model at all times and another one to be performing inference to upgrade the dataset. And we keep that flywheel going. So data in, label, feed that to the model training, occur, reoccur all the time. Revolutions. You follow? Revolutions. Keep the clients on the Ferris wheel, and it goes. The park is open 24/7, 365. Every decade, every goddamn century. That's it. Now, I gotta go back downstairs and monitor some more training. There's a timer. I'll see you soon. Machine learning is a lot like building muscle. You want to make gains? Sets and reps. You want your model to be better? Run 1,000 experiments. A lot of them will fail or crash, but That's how you figure out the ones that work, like Dazzling Dream 1348— 1384. Okay, let's round out this video. So what's happening is we've got a model training, as always, finishing the day, model training overnight. So we'll leave that run out, and then what's going to happen after that, very important step in any machine learning workflow, is to predict and evaluate. So That's the last prediction one. And in here is where you'll see I go through all of the predictions and see how the model did on each of our different classes and make sure that it works okay in the real world or on our test images. Because otherwise, if we ship a model that doesn't work, that's not a great experience. But what you'll see is throughout the video, we've been testing it ourselves, and I use NutriAI every day, so I know how it works on real-life images of food. And I actually found a really cool term today in the article I was reading about LLMs, which is genshi genbutsu. Go and see for yourself. That's from Toyota's production system apparently. And I translated this. Genshi genbutsu is real things, real places. Same as Nutrify, using the app in everyday scenarios that mimic actual use cases. Repeat after me: the real test set is real life. Peace. Are you waiting for the beep? Yep. Okay, when it beeps, should I go? Yeah. Goodbye, Josh. Thank you for today. Another beautiful day. How would you describe working at a startup, Josh? Fun.
