Sparsh Mehta explains testing in freight marketplaces, where A/B tests don't work for core growth. Start with a hypothesis, run a manual pilot to find operational friction, then use human-vs-machine experiments to validate before automating. Cheap learning beats premature engineering.
Yeah, Dimitri, I know I was messaging you. Did you wanna come on?
Thanks, Mike. I had a question for you about how to do kind of experimentation in these kind of very complex operationally heavy marketplaces. Because that's something as an outsider seems much harder compared to like a typical tech company where you just run your A/B test and go to town. So how did you kind of do experiments? How did you use them to validate some of these growth levers that you were pulling? And what do you think is the role of that?
Yeah, it's a really great question. And even in operationally complex environments, right, you can do more traditional A/B testing for the parts of your business that are more software-led, your landing page, things like that, which you absolutely should do. But to answer your question, I'll focus more on the operationally complex experimentation that you would be doing. I go back to the model we had at Convoy on that growth team, which was sort of first proved that if you have a hypothesis, right? For example, like at Convoy, one of our hypotheses as we were building that spot auction business was with, if we have with the leverage of our data, right? Because of all the companies in the freight business, we have the most structured data. So knowing that we have that asset and speed of execution, right? Because again, other freight traditional brokers don't have that urgency or don't have the ability to use software to have that speed of execution, right? If we do those things right, we have a right to win in this like decades-old spot auction business. That will be a competitive advantage. That was the hypothesis. Right now, truly to test that, there's a lot of operational components because you have to find truckers, you have to provide the bids in each TMS system for a shipper that is unique and separate. You have to service the shipments or no one's going to give you the next shipment and do it successfully, right? So there's all the operationally complex reasons why it might not work. And so for that reason, you don't want to invest too early in the software, right? until you've ironed out all those kinks. So what we did was like put, like throw a person at the problem, right? Put someone smart, a program manager behind the problem, one person spend all their day just doing it. And as long as they're a good generalist, right? Will be able to execute the job, but also extract the learnings and then guide your product and engineering teams on a roadmap if it works to scalably solve the problem. And you can do, once you start building the software, do A/B tests there. Like at one point we reached a point where we're like, okay, this human-led thing is working and we're already winning because of the asset of our data. Do we even need to spend engineering resources automating this? Right? Because there's a trade-off to that decision. So, for some time we did man with machine where we had a person bidding on a treatment group and then a control group where The early software and algorithm we built was betting and there was a feedback loop where the, you know, when the machine won, we'd try and figure out why. When the man won, we'd try and figure out why and help iterate the system to a point where ultimately it was purely software-led. We automated the entire system. There's still the operational components of servicing it, which we figured out in the process. But yeah, that's just one example. Obviously every situation is unique, but I'd say like find a cheap, inexpensive way to learn quickly and don't build too soon because when it's operationally complex, there's a lot to learn still.
