Analyzing Everything with Byrne Hobart and Patrick McKenzie

This week on Upstream, we're releasing an episode of Complex Systems, a show on the Turpentine Network hosted by Patrick McKenzie, a polymath with interests including software and finance. In the episode ahead, he talks with Byrne Hobart about the restaurant industry, in-app purchases, AI, and more. Check out Complex Systems in the description for more like this. Please enjoy the episode.

Heigh-ho, everybody. My name is Patrick McKenzie, better known as Patio11 on the internets, and I'm here with my buddy, Byrne Hobart.

Hey, everyone.

Byrne is the first repeat guest that we've had on this program, but both of us tend to write and talk about many similar things in our respective beats around the intersection of finance and technology. And every time we have a conversation, I feel like, oh, a lot of people would love if they could be flies on the wall here. So we're just going to have a casual fly-on-the-wall chat for Something that's been on burn in my respective minds recently.

Absolutely.

So kicking things off, you write a newsletter called The Diff, which everyone listening to this should subscribe to, pitch out of the way. The one thing you mentioned recently was about the economics for video games that are monetized by having a split free currency/real money currency and how the pricing grid works for those real money currencies. Do you want to chat a little bit about how that pricing grid is for people who have not been sucked into Zynga before and And we'll go off from there.

Yeah. So with the caveat that I have never purchased strawberries for my FarmVille farm or a cool hat for my Counter-Strike character, but I'm aware that this exists and you can't miss it if you look at the financial statements of some companies in the gaming space. So yeah, there's a model for games where they are free to play, but they are generally not free to win. So you can download the game, play around with it. That's what the vast majority of people do. What they'll typically do is they will structure the game to give you various kinds of virtual resources. And some of those are very cheap and you get them all the time. And then some of them tend to be resources where your access to different parts of the game will be gated by the availability of these resources. And you often end up in a situation where you can get to the next level, you can get the next hit of dopamine if you either spend a bunch of time grinding or take out your credit card or, you know, you usually have your payment information saved, have had it saved for a very long time. But yeah, you can make an in-app purchase and then proceed to have more fun. But when you make these purchases, in many cases you are not directly exchanging dollars for virtual goods, even though the virtual goods are what you're using. You're usually exchanging dollars, often not always, but usually you're exchanging dollars for some kind of in-game currency and then you're spending that currency on those goods. So actually what a lot of these games have is kind of the equivalent of the currency exchange stores you might see in airports and in Times Square or something where the, they are not there to make a really competitive market. Market in, you know, EUR/USD or something. They are, they are there to amortize the cost of high rent by really taking a large margin on those currency transactions. And, um, in, in real-world currency, you know, you sort of have a sense of what these currency units do and what they buy. So, um, so this, you know, the chunking thing doesn't really apply. But in, in in-game currencies, um, they are— they give you a set of specific transactions you can make. And so you're, you're exchanging typically like dollars for magical gems and You can spend, um, it's almost never a round number. It's usually $0.99 for a certain number of gems or $4.99 for more gems or $9.99 for even more gems. And as you might assume, there is a bulk discount. But one of the things that they do in this pricing grid, like if you, if you actually price it out, and this is actually one of these cases as a side note, it's one of these tiny, tiny improvements that AI has made to my life. So if I had been writing the post on this 3 years ago, what I would have done is I would have had Excel And I would have had my browser window and I would have been typing numbers into this grid because they were in some kind of graphic. But because I had ChatGPT, I took a screenshot, dragged it into ChatGPT, said, please convert this into a table, and it did it just fine. And it also did the next obvious thing, which is give me the GEM/USD exchange rate at each price point. And what you find is that you do get a bulk discount, but the bulk discount isn't really linear and it's sort of looks more compelling for the low dollar amounts. And then for the higher dollar amounts, your savings tend to trickle off. But it's also, they generally, they do not do really round numbers and they certainly aren't going to give you a gems per dollars exchange rate. And so you can kind of think of different customer behaviors where my guess is the first transaction is almost always $0.99. It's almost always that someone has one little thing in the game where they, if they had the 50 gems, this would not be a problem. They don't have the 50 gems. It is a problem. They want to get to the next thing. They get their 50 gems and that at some point someone does that perhaps a couple times and they realize they should get a bulk discount. So they calculate what their bulk discount is and start spending $5 or $10 at a time. And then my guess is once you're at the point where you are periodically interrupting your gaming time in order to make these purchases, you just decide, okay, I'm going to buy a very large amount, like $50 or $100 worth of gems. And that raises the question, like if you, if you read anecdotes about these games or if you happen to have had access to some kind of anonymized dataset of consumer spending, you might notice that there is a pretty strong power law, that there are people who play these games occasionally and spend a little bit of money. There are people who have a, what I would describe as, like, I don't know these people, but I've done the SQL queries. I would describe it as a problem. And it seems like there are people who are spending hundreds of thousands of dollars a month on these casual games. Those people, like those huge whales, are pretty rare, but they're spending a lot. It kind of raises the question of, okay, Why doesn't the game give you a chance to buy $1,000 worth of gems at a click, or $5,000 or $10,000? Like, surely the conversion rate on one big hefty transaction is going to be better than the conversion rate on trying to consistently get them to make a small transaction over and over again. And my guess is, when they show that pricing page, they really don't want people thinking about the possibility that they will buy $5,000 worth of gems at a time. And so at some point, Bigger discounts for larger purchases actually scare enough people off from making that first purchase. And we have to assume that all this stuff is A/B tested endlessly. And, you know, when I, when I say probably the first purchase is $0.99, that's not just a behavioral claim about how someone would start spending money on a casual game, on a digital service. That's also a game design feature that when they set up these barriers within the game, they probably set them up so that the easy thing that they probably do some kind of titrating thing where it's like, you have to. grind for a minute to get your 50 gems, and then you have to grind for 2 minutes to get the next 50 gems, and then for 10 minutes to grind to get the next 50 gems, and so on. And at some point, it's just easier to spend some money, but they probably want to make sure that that first transaction is most naturally $0.99.

So I would short the estimate that the first transaction is likely $0.99, unless you have data to the contrary.

I don't.

My point of view on this is largely informed by spending a few years as a consultant for software pricing pages, which is importantly adjacent to, but not the way that pricing pages are done in mobile gaming. But to keep up with the state of the art, because Zynga et al. were employing a lot of very smart people to do a huge amount of testing on this sort of thing, I listened to the conference talks, read the PowerPoints, et cetera, et cetera. And so I have like a little bit of accidental knowledge here. One bit of accidental knowledge, and to be clear, I like a thing that Tom Clancy said once where someone asked him if he felt fulfilled in life in his chosen profession. And he said, I entertain people, it's an honest living. And I think video games broadly entertain people, it's an honest living. There's a spectrum and many of the mobile games which are designed to be Skinner's boxes from hell are on the less salubrious end of that spectrum. And then you can look at things like, say, I've used many of these products in anger, as it were. Currently play Magic: The Gathering after a 30-year absence, and it has a strikingly similar monetization model in its Magic: The Gathering Arena product. And that is used by, you know, a lot of smart engineers in, in, you know, Seattle, San Francisco, etc., etc., who are trying to recapture nostalgia for their childhood, probably have $200 a month average value per user and are no worse off for having, you know, a hobby. It's golf clubs, but updated for the millennial set. which I used to excuse the following. If you listen to a presentation from the CEO of Kongregate that she gave at a number of game conferences, including Gamasutra's conference back in the day, and you can find this if you Google for Kongregate CEO presentation whales, they talk extensively about the monetization of these gaming and these games and how the monetization skews towards the higher end of the spectrum. Finger to the wind, I don't have active recall of all the numbers, but the vast majority of players will never pay a dime. Of payers, you know, the 2%, most will make one transaction and then stop. Some people do not stop at one transaction and, as Bernd mentioned, spend thousands of dollars a month or more on these products. One reason that there is not a $1,000 a month thing on the landing page is that there are humans assigned to those players. Their interaction with the gaming company is no longer simply of a person typing things into their mobile phone, but they've been assigned someone who is very analogous to a casino host. And for people who are not familiar with that bit of the less salubrious economy, if you are in the habit of going to Las Vegas and renting a room for a few days and then pushing a lot of money in on your loyalty card into games where the casino has a great edge, such as slots, et cetera, someone will make your acquaintance and buy you dinner. It's called comp in the industry, but they'll buy you dinner and they'll have a conversation about like, what brings you to Vegas? Are you in— In town for a conference, et cetera, et cetera. Do you come here often? Do you stay at our place often? Do you stay anywhere else? And basically build a profile of you which will get saved into the CRM system. There's actually a wonderful blog post from before big data was big data, saying that MGM had an unheard of amount of information on its customers. They measured it in terabytes. This is from like the early 2000s when terabytes were still cool. Anyhow, the casino host. Their job is to bring you back, to capture more of your share of wallet with that particular casino relative to every other casino in Vegas, and to facilitate your interactions with the casino and bring you back more frequently to spend more. The casino host for, without loss of generality, Zynga, is pretty much doing the same thing. It's an inefficient way to buy a friend, but effectively buying a friend who shares your passionate interest in whatever the recent guild war that you're engaged in is, and will do things like facilitating transactions for, oh, you think you need $10,000 to win the next stage of the guild war? We can definitely take a wire for that and, you know, quote you the number of gems that that buys, et cetera, et cetera. And like many VIP systems in the world, there are things like, you know, you get a preview of upcoming content releases, you get some very titrated but real access to the developers, et cetera, et cetera. And that's very common in the gaming industry. The people who are external to the company, but important to the success of the game, like gaming journalists, well-known players on YouTube, streamers, etc., will often get invited to events under the COVID of NDA for NDA, a handshake, or nothing at all in advance of a game launch to kind of build buzz and help get the word out there. But those folks are not typically directly monetized. These folks are being very directly, aggressively monetized. Anyhow, a little thing about the world that most people wouldn't know. I think it's interesting though, because this is one of those things where you, I, and most people listening to this podcast think we understand the artifact because we can download the artifact on our phones and look at it and make assumptions on how the industry economics must work based on the artifact presented to us. This is an intuition that people repeatedly have about industries. I have flown in an airplane, therefore I know how airlines must make their money. I have been in a hotel room, therefore I know how hotels make their money. I have received medical treatment, and therefore I I understand the breakdown of medical expenses. And these are, generally speaking, poor intuitions to have because of the way that those costs, while they feel to unsophisticated customers, are distributed like relatively evenly with, you know, a couple of levers associated with them, like maybe you'll get a slightly bigger hotel room, are not in any way distributed evenly. There are whales in hotel rooms. There are whales in airline travel. There are whales in many— well, there are are certainly whales in medical treatment. Most would keenly wish to not be in that category, but they also keenly wish to not die. And so they are getting their money's worth, as it were. Anyhow, got on my soapbox for a moment.

I think that's true. I think there's a whole set of companies or the whole set of products where the typical person who interacts with them is not necessarily the median interaction, or the median dollar spent is not spent by the median spender of dollars. And depending on what the power law looks like in that industry, you can have just very, very different economics, or you can have like economics that only makes sense in light of the existence of these really, really heavy spenders. And, you know, I think about with airlines, I sort of ended up doing more business travel right around the same time that part of my job was to read and interpret airline financial statements. It was a really interesting experience to sort of understand that I had not understood how this business worked. And, you know, like some of the intuitions that I had as just a leisure traveler, like I didn't really understand why airlines other than Southwest could ever make money. Like I did. I did know that there was this general competitive dynamic where you have these capital assets, they last for a really long time, you have a lot of fixed costs. And so if you could sell that additional seat for $5, you would. And if Southwest has a lower cost structure than a lot of its competitors, as it did many years ago, then, then of course they, they make all the money. And then it turns out that it's, it isn't just that, and that not every customer is mostly sorting by cost and not every customer's economic relationship with the airline ends when they step off the plane once they've gotten home. And then they could restart that economic relationship with Spirit or Frontier or anybody else, whoever the low bidder is. And so sometimes the intuitions you get are just— they're going to be wrong because you are not the person who drives the economics of that business. And I think that probably works the other way too, where it may be the case that the really hardcore players of these casual games don't really recognize just how weird that behavior is and may I may assume that a lot of other people do this too. My favorite anecdote on just like, you don't, you are very unlikely to understand an industry if you are just the typical consumer interacting with that industry is this brief note from, I think it was his ghostwritten autobiography where Arnold Schwarzenegger is talking about how he and his bodybuilder buddies would always, they love to go to the gym and then go to an all-you-can-eat buffet. And one of their ongoing topics of conversation at the all-you-can-eat buffet was how could these places possibly stay in business if everyone sits down and just eats 2 pounds of protein?

Yep.

So for those of you who haven't read your substantial writing on this topic, something that you've covered many times is that the airlines work because there is a lucrative class of business travelers who are not spending their own money. They're spending money from a budget at work, and they are willing to, subjectively speaking, pay richly for maximal convenience to their travel schedule or maximal convenience for business necessity, and then bank on essentially their personal accounts and loyalty status with that airline, which they are able to put away towards the annual vacation to Hawaii or similar. And one of the points that you made recently was that this makes it difficult for low-cost airlines like Southwest, which have— even if you get a collection of desirable routes in your network such that you're able to capture business travel, if you cannot both capture business travel and also get that person to somewhere nice over Christmas. You will not solve for the job to be done of like the core business spenders in America who can spend $100,000++ on air flights per year. The existence of that statistic, by the way, blows the minds of many people and it falls pretty straightforwardly out of the math of like what happens if you really do go home every weekend from New York to San Francisco or similar, which As folks who've known consultants might attest, is not that infrequent of a schedule when summed over the economy. Anyhow, I genuinely do think there are industries where the sort of hyper-consuming outlier does not understand that the rest of the world is hyper-consuming outliers. I'm stealing that word, by the way, from an essay that I read once, which explained the market structure of the Japanese anime, manga, and cultural industries. And I wish I remembered how to Google for that essay because it was fantastic. But it said essentially that a number of the reasons for the very niche proclivities in parts of the anime world are they might not be very popular per se, but they are popular with a certain strain of hyper-consuming outlier who will, in addition to watching all the episodes, getting them all on DVD, individually, they will buy a lot of the merch associated with that anime, such as plastic figures of the main characters who are almost invariably scantily clad young women, and pay $300+ for those figurines, of which a generous licensing fee goes back to the production company. And so some hyper-consuming outliers in some industries realize that they are disordered or realize that they are, if not disordered, a little bit different than the average member of the population. If you can trust the CEO at Kongregate, who definitely has an incentive to talk her book here, but I would believe it from knowing many people who play various competitive video games. Apparently, the whales for casual games are, you know, we picked up the phone and called them and they seem like regular people. It's, you know, doctors, dentists, small business owners, engineers, et cetera. How can they spend tens of thousands of dollars a year? Because they're well off. Why do they like this game? I don't know. She makes an extended analogy to, I like non-competitive figure skating and I spend tens of thousands of dollars around that because it's my thing. Deal with it. And she said, to the extent that you think that our customers are disordered, a lot of them are like, I just really care about wars between my clan and someone else's clan more than you do. Deal with it. And well, I think there are parts of it that are self-serving, and I disbelieve the illusion for some games that target like particular player populations. I think there's probably more than More than zero truth to that bit of, I don't know, informal, economically motivated sociological work.

No, I think that is probably true. People have hobbies, high-income people have hobbies, and even if the hobby doesn't actually cost a lot of money, the opportunity cost of that hobby is really high. So there was a tweet a couple of months ago where someone asked, why isn't there really, really expensive custom software sold for tens of thousands of dollars to one person who wants exactly that software? And my claim was, it does exist.

It's called Arch.

And it costs, like, if you are a, you know, mid-career software engineer at a big tech company and one of your hobbies is getting your Linux configuration on your home computer exactly perfect, it probably, the opportunity cost of that is getting into the 6 figures if you've been doing that for a long time. And it's fun. But if you described that, if the way that you did a hobby like that was, You worked slightly longer hours and you pushed a little bit more aggressively for a raise from time to time, and you actually dropped 6 figures on making sure that the hotkeys are exactly the way you want them and that all the drivers are doing what they're supposed to do. And, you know, that you can see your smart camera feed and get alerts if something shows up. Like, if they paid for that, that would look weird. But if they pay for it in time instead of in money, It seems, it seems normal, normal within that world.

I have a slightly different point of view on this one, actually, because I think that custom bespoke software with one user absolutely does exist in the world. And I believe this because I have taken people out to dinner in Tokyo who are in the business of producing software with that description. And, you know, without giving away, like, trade secrets for anybody, it's been publicly reported in a number of profiles of lifestyles of the rich and famous that some, you know, without loss of generality, tech billionaires have very complicated smart home setups. And those smart home setups are not something where it's just like, okay, like plug in an Ubuntu CD at the end of the day and you're done. They require, you know, a, what is effectively a small team of engineers to do an initial implementation and then ongoing maintenance work. And, you know, at the end of the day, that small team of engineers has an address and a C corporation and sends out invoices. And those invoices cost the things that, you know, in line with what a tech billionaire expects a team of engineers to cost employed in perpetuity. And while it probably strikes a lot of people in the economy that it is ludicrous to keep a team that has probably invoices you $1 to $2 million per year on the payroll indefinitely just to maintain your smart home automation, anyone who has real estate pays, you know, some amount of money for ongoing groundskeeping work, et cetera, et cetera. And this is just that number scaled up. And potentially, you know, this individual might think, well, I and people who are close to me and who I care very dearly about the opinions of spend a lot more time in the parts of the home that are subject to automation than we do looking at the green, green grass outside. But it will be green, green grass. And whatever the number is that gets that accomplished, I will pay that number, or someone at my family office will pay that number. I will never have been confronted with this decision after initially greenlighting that this is the guy. Okay. Yeah. Give him what he wants.

No, that's, that's fair. And I guess there, there are little pockets of the software industry that, that have that kind of dynamic where there's, there's a small number of people who need it and they have very, very particular needs and they can absolutely afford to pay a lot for that. So there's a, you know, at, at hedge funds, for example, there could be, you know, a team of 5 people who are trading some particular thing like the volatility of electricity prices or something, something like that. And so they have particular information they need displayed and they need that displayed reliably and they're making very, very expensive decisions if they make the the wrong decision on the basis of what showed up on their screen. And so it does make sense for them to basically have someone working full-time on just building and maintaining the software just for them. And that person is partly paid to build high-performance software that rapidly charts the right things and displays the right headlines and that kind of thing. And they're also being paid a lot to get yelled at because if something breaks, then the The time between when it breaks and when it's fixed is time that money is not being made and potentially time that money is being lost. So it's good to have someone feel like it was worth it to get screamed at for that kind of mistake. And, you know, a lot of those jobs can be incredibly high stress, but it's also, it is kind of cool. Like, it is actually a craft rather than a manufacturing position in software. You are not stamping out, you know, building something that they're going to be 100 million copies of, and a lot of people are going to use it in roughly the same way. No, it's, it's something where you actually see the end product and it is exactly what one specific person or one small group of people really, really wanted.

We'll continue our interview in a moment after a word from our sponsors. Hey everyone, Erik here. You know, at Turpentine, we're always looking for ways to streamline our work. That's why we use Notion for all our internal documents. It's been a game changer for us. And I want to tell you why. Notion is this amazing all-in-one workspace that combines your notes, docs, and projects. But what's really cool is their new Notion AI. It's like having multiple AI tools built right in. You can search, generate, analyze, and chat, all without leaving Notion. What sets Notion AI apart is how it personalizes responses based on your content. It's not just another generic chatbot; it understands the context of your work. Plus, Plus, it can search across thousands of docs in seconds, even connecting to tools like Slack and Google Docs. And get this, Notion is used by over half of the Fortune 500 companies. Teams using it send less emails, cancel more meetings, and save time searching for their work. It's designed to keep everyone on the same page while protecting your privacy. Wanna give it a try? Head over to notion.com/upstreampod. That's all lowercase. Notion.com/upstreampod.

Awesome.

You can try Notion for free, and when you use our link, you're supporting our show. Give the powerful, easy-to-use Notion AI a shot today at notion.com/upstreampod. Hey everyone, Erik here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lanahan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has frontend engineers excelling in TypeScript, React, and Next.js ready to onboard to your team today. For backend, Squad engineers are experts at Node.js, Python, Java, and a range of other languages and frameworks. While it may cost more than the freelancer on Upwork billing you for 40 hours, but working only 2, Squad offers premium quality at a fraction of the typical cost without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent. Increase your velocity without amping up burn. Visit choosquad.com and mention Turpentine to skip the waitlist.

It's interesting. We go between those polls over time because, uh, Way back in the day, it was literally the case that all software was an individual person taking out the punch cards and writing one program at a time. Those largely did not survive for very long times. And then the way that software was delivered evolved over the years, to put it mildly. But I remember in the early days of the internet, back when having a website was still like a bold new thing, every website used to be clawed out of the ether by an individual engineer or an individual middle school student like myself, deciding to take it upon themselves to create a website for a particular organization. And after it became obvious to businesses that having an internet presence was very valuable, some rather small businesses ended up contracting out bespoke software development work for the first and only time in the existence of those businesses' history. This was the dominant way that, say, local businesses got their websites made for— local restaurants, for example, got their websites made for the first few years of the internet. And then we had the technological substrate change. There was the introduction of WordPress. And so you would pay a designer to make a custom WordPress theme for you if you were maximally desirous of having that happen. And the cost of the website would decrease down from— there were restaurants that were shelling out tens of thousands of dollars for websites back in the day. And then it became more of a pay $500 for a designer to customize WordPress and perhaps another $500 to get it up and running for you. And perhaps $50 a month to host it and listen to your panicked emails about why isn't the website up. And then that has largely, but not entirely been supplanted by large platforms, which give people a far less custom-made experience, but are leveraging software that can be amortized over millions of businesses, whether those are platforms that have built-in distribution like the DoorDashes and et cetera of the world, or whether they are large software businesses, quote unquote website builders, which have, you know, an encoded method for stamping out unique looking but fundamentally identical websites onto the world. So a fun little business to be around the edges of.

Yeah.

And it is also, you know, we, I guess we lose, we do lose something if they're not These bespoke custom-made things. There used to be this perennial complaint about restaurant websites that if you just really wanted a flash-based experience that would show you pictures of very happy people dining at the restaurant, that you would absolutely want to go to their website. But if you actually wanted to know what the menu was, what the hours were, how to get a reservation, et cetera, their website was just not actually the place where that information was stored. And so it has been nice that just Like your universal basic restaurant website now has a place where you click to see the menu and a place where you click to see the hours and the location. And often it does seem like one of the, one of the forces there is that the group that is most incentivized to create really clean, usable websites for restaurants is the service that's going to do the delivery, whether that's the service that just handles the e-commerce side of it or the, does the actual fulfillment. So in some sense, you know, the, the world of restaurant web design has gotten really, really good because Now DoorDash and Uber Eats are A/B testing which designs work at massive scale, and they've learned a lot. And you actually, like with them, because they control a lot more of the transaction, you get to see little details about what actually matters. Like they will, DoorDash will sometimes offer a discount if you upload a picture of your food. And so they've basically outsourced the fine dining photography part of the restaurant media presence to to the end consumer of that food, which, you know, is a lot more possible now that we all have really excellent cameras in our pockets at all times. And, you know, I've always found stuff like that really interesting. The little tweak to the transaction in order to get the customer to do something that is value-added for the seller, but very hard for them to scale.

I've also noted some interesting transactions where, uh, people might not appreciate this until you've attempted to do a large data gathering operation at material scale, but when you are dealing with Businesses where the owner or account representative might be a difficult person to get on the telephone, either because their hand is on a fryer or because they are not operationally involved in the business. They have staff to do that, but they might still be the only ones that can give you an authoritative answer with respect to what the menu is. It is like pulling teeth to get some signal on, okay, you sell spaghetti for $12.99, but what is the spaghetti really? And one of the things that the ordering platforms have recently started doing is producing AI-based descriptions of spaghetti for those 4 customers in the United States of America who just not understand what spaghetti looks like. And I say that laughingly, but I have arbitrarily high confidence that it can be A/B tested whether adding 2 sentences of copy to the word spaghetti increases the sales of spaghetti. I would not bet against that at most odds offered to me. And then there's a interesting sort of like escalation ladder you can go to where if we get minimal information from the customer, we can impute details of what they offer and put in that AI placeholder and then ask our customers to act as a distributed intelligence gathering network to take a photo of the spaghetti as it is delivered. And then if you want, you can loop it right back to, hello ChatGPT, this is a photo of spaghetti. Does it have a meatball on it? We expect meatballs, but we're not sure. Please tell us, and do that at material scale without needing a team of people to tag photos internally, which, by the way, is absolutely a thing that tech companies have done internally. One of the genesis for Amazon Turk as a product, for those of you who haven't used Amazon Turk professionally, the name harkens back to the Amazon Mechanical Turk, a machine that could play chess with an arbitrary level of artificial intelligence back before electricity was a thing. And the— spoiler alert— the large language model that learned how to play chess was a human who was sitting inside the chess box. And so Amazon found itself routinely having a business problem where we need arbitrarily large burstable amounts of the smallest amount of human labor that can actually be called labor, and the thinnest slice of cognition that is still cognition. And we need to throw it at a bunch of problems, and the mix of those problems will change every day. And so Amazon built a tool for internal use and then was like, hey, we are not the only business in capitalism that has exactly this problem. And they did the typical Amazon thing of like, let's externalize the service and turn a cost center into a revenue center. It's a fun thing to play with. I don't know. There's many things to speculate about in the future with regards to how LLMs get used for most workloads. But one of the things that a lot of philosophers on that subject haven't done is look at the things that Like real businesses are offering real money for it right now, for like one penny to identify if the following photo is in the right, you know, physical orientation because we didn't want to hire a team of vision researchers to write an algorithm to do that. That job, thankfully, is going to be automated by a for loop in LLM in the very, very near future. And so just like progress against certain monetary prizes is an implicit signal that we haven't had major breakthroughs in cryptography in the last short while, the front page of Amazon Turk tasks is implicitly a signal that this is above the bar for commercially available large language models at the moment.

Yeah.

And I guess there's, there's the temporary period, which I don't know how temporary it is, but there is that period where the Mechanical Turk behaviors that used to be more of a cost of goods sold turn into capital expenditures for the buyer. Because now you can, when you think of that category of problem where you do need some minimum quantity of intelligence, you really don't need much and you don't want to pay a ton for it, you can think of that as less, we're going to continuously pay to solve this problem, and more as we're going to buy the training data that we need. We're going to produce the training data that we need to automate the solution of this problem.

Mm-hmm.

So, I guess my suspicion would be that for Mechanical Turk, that over time, more and more of the tasks will be reflective of that. And so you can actually use that as a sort of a way to look maybe 6 months in the future at what AI will be capable of. And I guess you have to, you have to discern a little bit, you know, both from thinking about that use case and also thinking about how, you know, like trying to use an LLM or trying to use some other AI tool to solve it yourself. Um, you have to think about that a little bit to figure out which of those are just kind of permanent. But there, there is a large set of tasks where if you can describe it clearly, you know, an intern could do it or Mechanical Turk could do it or ChatGPT could do it. And over time, I think the set of things where you, you don't just, uh, you don't just put it into ChatGPT, it's a little bit harder than that, that, that set does dwindle and the way that it dwindles is that people get the training data from, from things like Mechanical Turk. And I know there, there are plenty of other services that also do that kind of training for, for exactly the same purpose. I do want to rewind slightly on the, um, like back to the delivery apps business, because it's, it's still, I think it's still a really fascinating business. Um, and there's lots of fun to think about. So if you think about someone starts a restaurant, they have their one location there, they are probably, you know, hopefully good at some combination of cooking and just hospitality generally, just giving people a nice experience at dinner. And running a successful restaurant does involve a lot of other competencies, some of which you can perfectly outsource. So you're not setting up your own electricity and plumbing. Those things are provided by third-party service providers, and the pricing is very transparent. It's very easy to understand. But there are some things like designing the menu where you just wouldn't— you'd have to be very fortunate to be a really good chef who also knows exactly how to price things on the menu and how to arrange your special offers and in what order do you put things on the wine list to maximize the amount of money someone is willing to spend. Larger chains can afford to invest a lot of money in that. And I'd like to go to some of the larger, not super cheap chains sometimes and just look at the menu and think to myself that when, when the jalapeño poppers special got put on this menu, it was like, that was what someone bragged about when he came home and his significant other asked, how was work today? He gets to say, finally, after months of fighting for it, I got the price of the jalapeño poppers down to $13.49. And yes, they are zesty. And like that stuff, you know, it doesn't really make sense to think about that really hard if you have a single location and if people like the staff, you're actually able to make good food that people enjoy. You, you don't have to care that much about those details. But if you are someone like Red Lobster, you have to care very, very deeply about all these little nuances and all the cross elasticities of different items on the menu and whether or not there's something on the menu where most people won't buy it, but a party of 6 might choose to go somewhere else because one person in that party does want an entree that's vegan, but that does not have tofu or whatever the, whatever the weird constraint is. And when, when restaurants move, when they're using larger platforms, the platform can often do some of that analysis for them. This is going to show up a lot more at the DoorDash level than at the level of whoever is doing the QR code-based menu app. But over time, I think they will, they will be collecting enough data that they can start making more recommendations to restaurants on things like, you know, you, you really should be charging more for this. You, you probably don't really need this on the menu at all. And, you know, some of that does require knowledge that these, these providers wouldn't have. But increasingly, as the, as the provider of the menu is also someone providing the payments, they can actually look at scale and say, okay, your diner is your own special thing, but also So there are many, many diners. There are enough diners that we have statistical confidence on whether or not this, you know, whether there should be 2 hash browns or 3 when, when someone orders this particular special.

I think that in some ways the platforms here, or B2B SaaS largely, is recapitulating the role of the franchisor in a franchisor/franchisee arrangement where historically you didn't necessarily have to be a genius at site selection because McDonald's had a department of people to do that for you, and you were being selected to do quite a bit of restaurant managerial labor and provide capital on behalf of McDonald's Inc. And the history of franchising is long and wonky, but part of what you are buying in return for your franchise fee and the commitment to only buy your groceries at McDonald's in the future with the hidden embedded margin for McDonald's Incorporated is you were buying their ability to employ the best expertise on the market for a very long time where you could not afford the best expertise on the market. You might not even understand you needed expertise in particular fields. And even if you were able to afford it, you could only afford it in bursts, but could not continuously have someone in charge of menu development, whereas McDonald's corporate definitely can. And so this function gets redirected into platforms and B2B SaaS companies, platforms in the fashion that we've been describing, and B2B SaaS companies, because they're largely in charge of encoding a method for doing a business process. And usually, like, version 1.0 of the application is, well, you probably have some method for doing, I don't know, travel expense reporting at your company. We have a jumped-up spreadsheet that will do that in a web application, but you won't have to share Excel files over email anymore. But increasingly, the company will become experts at the problem domain and kind of eat around the edges of, you know, if you were doing this seriously, you'd probably have someone who's in charge of negotiating corporate rates with the various vendors that you use the most. Oh wait, we are doing this seriously. We have someone who is in charge of the rates. And now part of what you're paying for the SaaS price every month or every year is essentially hiring those humans on your behalf without having to employ them. them directly. And I think, by the way, that once you start spooling out those sort of nuances, the take rates of platforms look very different. And because when people compare, you know, a 15%, 20%, 30% take rate to, like, say, credit card fees and think that, oh, at the margin, the only thing the platform is providing is a customer and then they're absorbing the payment expenses, then the take rates look very large. But if you compare them to franchise fees, uh, the take rates like, uh, suddenly look quite normal. And they don't charge you, you know, opening a McDonald's. McDonald's will tell you upfront, yeah, you will need $3 million upfront cash on the barrel to do this, of which we will accept up to $2 million of bank financing, but $1 million cash on the barrel has to come from you. And so capitalism has found a way to employ experts without requiring you to put down $1 million when you open a DoorDash account. And I think that's kind of interesting and fun in a lot of ways.

It is.

Yeah.

I think, I think there's a, there's an interesting balance sheet story there where, yeah, you can, you can do a sort of capital light version of the franchising model. And because the capital's already there, right? The restaurant already existed. So you don't need that incremental investment. You just need to add the sort of layer of franchising on top of it. But there's also an interesting dynamic on the P&L side where if there are external service providers and they know that they get pretty decent margins from dealing with some of these small companies, not— they are— the margin picture for that, for dealing with the small companies It is a pain to get that restaurant on board. It is just not 10 times harder to get a 10-location franchise to sign an agreement than it is to get a single location to sign an agreement. But once you have them on board, you can charge them a lot and they, they just can't be that price sensitive and it's just not worth their time to be that price sensitive. But once—

It tends to be a very sticky product. Yes.

But once that relationship is being routed through a platform, the platform can be in a position where they take away $10 of the supplier's gross margin. That gets them enough pricing power to raise their prices by $1. That's pure margin for them. And it's eating into a business that because of those sales and marketing costs and just because of the operational expense of dealing with lots of fragmented customers, it's a fundamentally lower margin business. So yeah, you do end up with a lot of intermediaries, I think, getting nervous about what their gross margins look like in the long run because there's now someone very, very are implicitly negotiating on behalf of their customers. And I could see that going in a lot of different directions. Like maybe DoorDash decides at some point that the right strategic move for them is to acquire a big food distributor so that they actually own that piece of the relationship. And then what they get to say is, okay, we can think about these cross-elasticities and maybe if we cut the price of tomatoes for our customers, it actually leads to so much incremental spaghetti and pizza consumption that we actually capture more of the upside Than the downside, and I think one last one last point to make on the the whole question of A/B testing menus and just you know systematically analyzing every little bit of the unit economics in order to eke out more margin. I suspect that this explains a phenomenon that you you pointed out in one of your coming back to America tweets about portion sizes. Because I bet when you A/B test it, very few people are annoyed that they left 300 calories on the plate, but some set of customers is pretty annoyed that they were still slightly hungry or could still have eaten another bite. After they finish their burrito. And the marginal cost of those ingredients is really, really low. Your labor cost for making a, you know, a burrito with 100 more calories is basically zero. So if you A/B test it, you actually A/B test your way into bigger portions, maybe more food waste, probably also people just eating a whole lot more when they sit down at one of these restaurants.

I read a review of Chipotle recently that made exactly that point. And I remember reading the review, being a little bit confused because Disclaimer, I was a longtime Chipotle shareholder. Worked out kind of well. But Chipotle is Chipotle in the same fashion that McDonald's is McDonald's. What is the theory of the mind of a user entering text into a text box about McDonald's? What social or emotional need does that fill for them? And the user's review was, they were not generous with their portion of rice on my burrito. 3 stars. And I thought, you know, if this is just going into a pool of reviews which are aggregated across all Chipotle organizations, et cetera, et cetera, like that. And then I thought, well, possibly this is like the anthropology of using apps where if you ask someone to leave a review, some percentage of people will answer that question, just like I am compelled to use any text box put in front of me with a question above it. It does seem shockingly to me as someone whose stomach was resized by 20 years in Japan that there are people who do really want that marginal 300 calories. One question I don't know, because I spent 20 years in Japan, do we have an institution of, between like the mom-and-pop restaurant on one end of the scale and large franchise chains, franchised or corporate-owned chains on the other end of the scale, is there an institution of restaurant groups where there is a brain trust that owns like 12 or 15 locations, often in a variety of concepts, and they provide design, menus, pricing, And maybe like clientele selection and real estate selection, but they don't individually operate each restaurant. Oh, it occurs to me, Lettuce Entertain You in Chicago is kind of that. They have a, they have a Japan concept and a China concept and an India concept and one-offs of all of them, as far as I know.

So the, the Danny Meyer complex of non-Shake Shack restaurants might, might sort of qualify where it is a bunch of different concepts and they do In that case, as far as I know, it is common ownership, but they aren't, they aren't all part of the same thing. They're owned by the same, owned and managed by the same broader organization, but they are, they're of different, different genres. I don't actually know how much, how much that kind of thing exists, although I do get the sense that there is this ecosystem of restaurant experts who are, they're familiar with some particular category of concepts, some general, general aspect of that business, and they will drop in when, when there is A restaurant idea that is turning into something that might scale nationally. And, you know, if you look at the, if you read the Cava S-1, the IPO prospectus, and look at the biographies of some of the managers, it is clear that there are people who have, have done this kind of thing before, and they are just always on the hunt for a concept that has maybe 3 locations and they're all very busy. And it's, you know, 3 locations in one city and you sort of do the math on Cavas per capita in the Washington, D.C. area. and do the math on how big Washington, D.C. is compared to the rest of the country, and suddenly you have some pretty exciting numbers. So I think that that is probably the closest thing to that. And it may, like, maybe that actually says something good about the American consumer and just our taste in general, that there isn't kind of the— there isn't— the skills don't transfer all that well. And that if you are taking a concept that works, like it's Mediterranean fast casual, it's Mediterranean, like slightly Premium fast food that you probably need to spend all of your time thinking about that exact concept and not trying to think of okay what are what are the other food categories that could also be put in this bucket and you'll probably drive yourself crazy if you're trying to do Mediterranean fast casual over here and also trying to do Indian fast casual over here and Korean fried chicken fast casual over there so maybe yeah maybe that is where where it starts to break down and then I guess like the the Union Square Hospitality Group said Daniel. Meyer's set of restaurants. Like, they are all just really, really well done. It is like, it is basically the McDonald's equivalent for upper middle class people in New York where, you know, it will be like a generically good experience. It may be, you know, one of the better meals you had, but probably not. It's probably just like a very, very good meal and the service will be impeccable and you will get exactly what you expect. And so that, I guess that gets into another piece of just restaurant economics of part of what you're selling is just certainty. And you offer a lot of certainty to the people driving a minivan full of kids home. They know the kids, their blood sugar is plummeting, it will continue to plummet. And so the job to be done is just get them, get something, get some food in them before things get really, really bad. And then the other category, like I think about this a lot with just restaurants to have a business meal at. I used to just obsess over this when I was younger, trying to figure out like, what am I signaling if I choose a particular restaurant? Like, does a chain say something? If it's not a chain, do I actually know if it's a good restaurant or greasy spoon restaurant? And Yelp was just a lifesaver there because they recognized that as a category that people searched for and they had landing pages for it and everything. But that is, I think that That is what companies are trying to solve for when they do that. And I guess with business meals, part of what you're trying to solve for is you do actually want to slightly signal that you are a prosperous person who doesn't have to worry about money. So you do want a place that slightly overcharges given the quality of the food because you're going to be picking up the check and you need to just casually not look at the numbers and just pay things. Or I guess it depends on what kind of business you're doing. I suspect that if you are having a chat with someone who wants to be a client for your CPA business, you probably carefully scrutinize the receipt and make sure the numbers add up and things. But yeah, it's like thinking about the job to be done. It's not just a calorie delivery service.

Yeah, I used to have an opinionated bit of advice, which I'll repeat for the benefit of anyone who can use it, that if you are in business for yourself, it's good to find a local restaurant where you like the vibe and you like the people and start making a habit of doing as many of your business dinners there as possible and tipping generously. which will typically get you formally or informally added to the list of whales. And then you get the joy of being the person whose consumption is not obvious to everyone else in the restaurant, but there will always be a table available for you. And that's the thing you can either explicitly ask for, or the culture that is hospitality workers and managers will simply arrange for you. And I often think that it coheres with the aesthetics of many people in software and similar to We get to be patrons of the local gastronomic arts by finding a restaurant we like and using some of the relatively abundant software dollars to subsidize the gastronomic arts. Zooming in on another thing you said, one of the jobs to be done of the QSR, quick-service restaurants, fast food industry has historically been decreasing the amount of variance associated with dinner, radically compared to the home experience, and then quite a bit with regards to like the traditional restaurant experience. I think one thing that people don't appreciate is there was a complaint that everything is getting suckier over time. We all went to McDonald's as children. McDonald's is observationally in these days not quite in— they do not deliver precisely the same thing that they did 40 years ago, and they did not 40 years ago deliver precisely the thing They delivered 20 years before that. But partly this is because of the good kind of inflation, inflation of expectations in the United States of America, where we've become a richer nation. Part of becoming wealthy has been that a lot of people in many socioeconomic classes have kind of promoted themselves out of cooking. And for good or ill, there's many people who are enormously attached to the cultural ritual of food preparation. But much of America essentially does not do it anymore. And I think that will be almost normative within our lifetimes. But be that as it may, but rather than McDonald's being a once-a-month treat for middle-class people, it is now a major source of calories for large portions of the population. And partly due to that fact specifically, middle-class people have promoted themselves into a different class of restaurant for, you know, the Shake Shacks and— Chipotles and similar of the world for the fast casual experience, which does not discriminate along some axes, but does discriminate along others, including willingness to pay a few dollars more than McDonald's, rather small but material number of dollars more than McDonald's. And then we've built a gradient on top of that too. Like, you can get Chipotle or you can get Chipotle delivered to your door in 30 minutes. And, you know, there is an opaque But very real fee that is a difference between those 2 numbers. And some people decide, you know, I'm, I'm more of a, like, stay here, continue working, and then get the Chipotle delivered to me by a courier person than I am a take an hour out of my day to walk laboriously down to Chipotle so that my servants can prepare meals for me person. And so the optimistic upshot in this is that the amount of societal wealth that quick-serve restaurants represent is vastly underestimated by most people. We, we live like medieval kings and look at that situation and Oh, those miserable off people at McDonald's who are only, you know, having servants spring to their whims at the touch of a button to bring them their next Diet Coke.

Yeah, like I was, I was very much familiar with that norm growing up where, you know, we, we did go to restaurants, we did get only Chinese or pizza delivered. Those were, those were the delivery categories of food and they, they were kind of a treat. And I do remember just feeling I remember feeling just like slightly emotional, slightly like I had made it the first time that I seamless a meal to my desk because I was working late and the company paid for it. I felt like I'd actually just reached this different category of existence where feeding me was just economically profitable to the organization that employed me and was not like a little giveaway or perk. Like there is this economic relationship where They are basically buying more hours of office time and getting a really good deal. Like, the return on investment from something like the Google cafeteria is just phenomenal because it gets a lot of people instead of they're hungry and so they go home and have dinner. It's they're hungry, so they eat and they talk to their other Google friends and maybe get some ideas, sit back at their desk.

I've seen arguments by people that the tech cafeteria Where the thing that distinguishes the tech cafeteria from a cafeteria in most of the nation is that food is of subjectively higher quality and virtually universally free. And the institution that is the tech cafeteria was reported on many, many times, often by people complaining adjacent to that reporting of these entitled tech workers. They don't even expect to pay for their own food, et cetera, et cetera. And some people have advanced the hypothesis that this was a zero interest rate phenomenon. And then now that money has a cost again, And that salaries will be slashed and people will be forced back into the office and there will be prices on things in the cafeteria or they will discontinue that service. And that is not a reality-entangled point of view at all. The economics are just so absurdly attractive. Every company has a number and few companies disclose it, but finger to the wind, it's probably like $15 per beneficiary per day of what it costs to provide those services, and $15 next to the total compensation of an engineer, even at a place that is not paying Google rates for engineers, is not a lot of money. And cynically, but really, like, recruiters have documents that— this is not an accident, this is a plan. There's a document describing the plan. If the candidate pushes back on salary, then tell them that the value of the free cafeteria food is worth $10,000 a year and say, okay, so you wanted $15,000 more in salary, So I'll give you free cafeteria food and $5,000 more. And many relatively sophisticated professionals do not see that as a line. And the reason you know that they don't see it as a line is because there are documents describing this strategy, because it works all the time when you employ it at scale. Sorry, a little bit of labor activism still left in me. But be that as it may, cafeterias are here to stay forever for the You know, descriptively speaking, the best compensated employees in capitalism. Uh, and, you know, you should just expect the economics of that to continue pushing down as the, uh, borders of the upper middle class in America push up.

Yeah, I think, I think that's right. I think there's like, to the extent that companies did cut back on benefits like that, I think that it's because the way that employees think about free company-provided food is a really good indicator of where the company's culture is right now. And that the, the optimum for the company is that, Yeah, we're paying $15 for you to work several hours, and there is just not a country we could outsource software development to where we would pay less than that per hour, and certainly not for the caliber of people we get. But then you can also— you can do the same pattern that you would do without the employee-subsidized cafeteria of you leave around dinnertime. And what you can do instead is you come to work with an empty lunchbox instead of a full lunchbox, and you load up enough food for yourself and your family, and then you roll out the door. And, um, I think that is, that is the kind of thing where it was, uh, probably more common in a zero interest rates, you know, growth focused world where it was, uh, there was a bigger risk from not having enough headcount than there was from having too much headcount and letting the culture diverge a little bit from what it had been. And I think at mature companies, it's also just, it's harder to have people fully bought into the mission because a lot of the mission has been accomplished by that point. Like Google has organized a lot of the world's useful information. Um, I'm sure there's more to do with that, but there's also like a lot of of the things that people do at Google are not organizing the next piece of information. They are optimizing either the cost of delivering that information or the revenue that you can derive from that, which is important. The reason venture capitalists were very excited to put money into Google was not— I mean, the mission helped, but the exciting thing was realizing this is a product with a lot of usage and figuring there was probably a good business model. I don't know that many— I don't know that the VC— I mean, it would be fascinating to read the original investment memos for the Google Series A, but I don't know how many of them realized, to the extent to which they realized, that Google actually had just probably the best economic model ever, where you have all kinds of these positive feedback loops where you're getting more data, and so you can serve better results and better ads. And the more searchers you have, the more ads you can display. You have higher bid density, so everyone's incentive is to bid roughly what they think their incremental margin is. So not only are you making more money, but you get just really, really good intelligence on on the profitability of different sectors. And sometimes that's irrelevant. Like, Google is not going to open up a law firm to do mesothelioma class actions. But if they noticed that people are paying a lot of money for leads in, I don't know, the office software space, for example, that might on the margin affect their decision to offer such services. So yeah, it is an amazing model. And It is useful for the world to make sure that when you develop an amazing product, that it is extremely profitable to do so, so that people who are just dollar maximizers want to invest in amazing things. But yeah, less inspiring. It is just more likely that you wouldn't feel especially bad as a Googler in 2024 taking advantage of some of the benefits in a way that's clearly not what they were designed to encourage. Whereas in 2004, I suspect that a lot of people would have really frowned on Like blatant attempts to exploit the Google system.

So fun fact, at a variety of large tech companies, the tax department is frequently tearing their hair out about that specific exploitation, less because of what they think that it does to the corporate culture, although there are other people in the company that very much worry about the corporate culture. Tax department specifically worries about what the IRS will think about it because the expense of providing a cafeteria is a business expense if you are providing the meals for consumption at the business's convenience and on the premises only. And if you violate either of those 2 prongs, then it's simply extra compensation that you've provided your employees and you have now not paid Social Security tax, et cetera, et cetera, on that compensation, or it gets disallowed as a business expense and you've underpaid taxes, yada, yada, yada. So jumping off another thing you said, the— I don't know if we've ever seen the original investment memos for companies like Google or Facebook. Facebook, although that would sure make for fascinating reading. But there have been a few oral histories of those early investment rounds published on the internet. One factoid I remember from it was that Ron Conway had to arm twist the VC firm, and I'm forgetting which one that was, wanted to be in the Google Series A before it did the not too atypical thing in VC and wanted that a little less. And they were about to go out of the deal. And Ron Conway called them on the phone and threatened them, saying that if they did not make good on their handshake commitment to invest, Ron Conway would fill the entire round himself, which is one of the canonical Silicon Valley stories of like threatening someone with yourself having a very good time. But one could not have known that it would play out exactly that way on the day that he made that offer slash debt. But a fun bit of Silicon Valley history. that, you know, recedes into lore and memory. And perhaps when one tells the story ex post, it does not match exactly the events of the day, but such is history. So is there anything else that has been on your mind recently, Byrne?

What else has been on my mind recently? I'm going to cheat and look at what I've posted recently and see—

This is one of the best reasons to write things. One of the reasons I have kept a blog for a number of years is that it effectively functions as a publicly visible journal for someone who has occasionally sporadic memory of what I was supposed to be working on this month. It's like, okay, flip back 6 months ago to where I wrote down my priorities.

That's kind of— so for a while I wasn't writing much online. And one of the places I started writing again, my current phase of writing all the time, was Goodreads. And part of the reason for that was I would look at a bookshelf and remember having read all the books on the bookshelf. And I would also feel like I can't, you know, I can't name 5 discrete things I learned from that book, or I can relate the plot to you if it's a novel, but I can't tell you all the details. And I felt like I was— it felt like I was getting a more insightful experience in the process of reading it than whatever I retained. And I know that's a perennial problem that anyone who reads a lot has. But writing on Goodreads, it did— part of what it did was it just, it did force me to actually think about what did I get from this book and what do I agree with? What do I disagree with? I, like, you know, sometimes you will read something and just gradually misremember it until you've almost inverted what the original point was. And it's good to have some kind of reference point. And I think it just, it helps with memory to have not just ingested something, but to try to explain it. So yeah, that is a very, very useful habit in my experience. I highly recommend just having some kind of a written record of what you've been doing and what you were thinking about and when. And it's also, I've written enough that I have forgotten a lot of things that I wrote. So I remember more, but I don't remember everything. And so sometimes it's weird to be looking back at something unrelated and end up finding something just delightful. Like, I will occasionally find things I said about AI in 2020 that I feel like, you know, some of them aged poorly, some of them aged really well. I think at one point it was like in 2020, I had some joking allusion to, you know, I don't think my job's going away. going to be replaced by a computer just yet, but I'm going to be keeping a very close eye on GPT-4. And yeah, it hasn't replaced me yet, but it keeps getting closer. So yeah, it's a good habit.

I definitely endorse the notion of write notes on what you've been reading recently, write notes on what you've been doing, et cetera, et cetera. Even if you don't publish them, they'll be super useful to you in the future. It has been surprisingly useful for me to ask ChatGPT or one's LLM of choice to riff on what you would have written as notes for a particular book or similar. More so, I was at a conference and a fellow attendee approached me and said, would you— I know you read the series Dagger and the Coin because you wrote about it on Twitter and gave it a 2 thumbs up recommendation. It's my favorite of all time. Will you do a conference talk with me about this tomorrow? I said, in principle, yes, but the funny way my memory works, I remember most of the major plot points, themes, The twist, et cetera. Oh, it has one of the best twists in fantasy literature. But I don't even remember the names of the characters anymore because I have the written equivalent of face blindness. And then I was like, oh, but like, this is a solvable problem these days. Even if there's not a vast secondary literature about this relatively obscure fantasy series that everyone listening to this podcast should absolutely read, you can just go to ChatGPT and say, hey, who are the top 10 dramatis personae in in Dagger and the Coin and give me a one-sentence description or, you know, a Twitter-length description of their events in the plot. And it was like, okay, Cythryn, the banker for the— ah, yeah, that's her name. She's the viewpoint character in the, like, second chapter. And then goes from there. Anyhow, so I'm far less worried about being replaced by LLMs and similar, and far more interested in, like, what's the ceiling for capabilities augmentation for that assisting humans who are, knock on wood, The OG intelligence in the universe. Um, OG intelligence in at least the local brain. Yeah, yeah, yeah, yeah. Everyone can make the caveats there.

I, I find it really useful for things like, um, if you, if you know there are multiple examples of some phenomenon happening, but you don't remember what they were, and you want to, like, you know that this has happened, you know, 3 times before, but you don't remember when, um, and you don't remember who it happened to. ChatGPT is a really good intuition pump for that, although you do have to check every single thing it says because it will get things wrong, especially Especially about really obscure topics. And in fact, so I had a weird experience recently where I was writing about buybacks and I knew the general history where they used to be rare and that there used to be these legal questions about can you actually do a buyback or is that market manipulation? The SEC eventually said, okay, here are the times when it is definitely not market manipulation, so just go to town. And I asked ChatGPT about earlier instances of this. And it told me they really didn't happen before like the 1950s. And then I really prodded it and it got me to, it gave me some examples and then I would check those examples and they just didn't actually seem to correspond to real things that had happened. But then I was reading in doing that research, I ended up Googling and I, when I Googled just the concept of buybacks and just random dates in the 1930s and 1920s, I found a blog post that had a table. Table that was showing high and low valuations and dividend yields and price-to-book and things like that for various oil companies in 1932 or something. And I realized that a book that has that table probably has many other similar tables and is probably compiled by someone who is very interested in giving extremely detailed history. And so I bought the book and it was great. It's, I will blank on the name, but Um, it was, uh, basically a history of US financial markets in, uh, from 1929 to, um, I think 1933. And, um, it's, it's the kind of thing where I'm— yeah, okay, I found the title. It's, uh, The Crash and Its Aftermath: A History of Securities Markets in the United States, 1929 to 1933. Um, and it is— most of the book is basically tables like that and explanatory text. But throughout the book, the author does mention numerous times that companies were actually buying back stock and talks about how this was like a maneuver That, so there was, if you know your 1920s and 1930s stock market history, you know that there were these proto mutual funds that were, they would borrow a ton of money and speculate in stocks, and then they often traded at a premium to the value of their holdings. And then Goldman Sachs had 2 of them, and those had cross-linked levered ownership, and then they collapsed. And so if you love complaining about Goldman Sachs, You have to reach pretty far back to find really big scandals, but you do find those scandals. And a couple of those organizations, they were trading at a large premium in 1929, but by 1931 or so, they were generally trading at a discount. And so some of them were actually pretty aggressively buying back their own stock. And so it was kind of exciting to see that this did happen. It's been written down, but also it's forgotten. lore. You can't— even ChatGPT, like ChatGPT mostly denies that this ever happened, but it did. It does actually exist. So, you know, you get to feel a little bit like, like Gandalf in the early scenes of The Fellowship of the Ring where he's riding off to Minas Tirith and digging through the archives and looking at these ancient books that no one has looked at for hundreds of years. You get to do a little bit of that if there's something that's not in the ChatGPT corpus. It also makes me, makes me slightly more I'm slightly more optimistic about the claim that these LLMs are not actually trained on just huge volumes of pirated works, because this might have been pirated by somebody. It doesn't seem to exist on the usual suspects, but it's definitely not in the training data. So yeah, you do want to be somewhat cautious, but I think as just a way to remind yourself of different events that you know happened and you're forgetting some of the details, it's great. It's the tip of your tongue machine.

Yeah.

If you think about the writing process, um, you know, writing, if you know what you're going to say and you have the facts on hand and whatever you're referencing is in a tab and you don't have hundreds of tabs open at the moment, um, the writing process can be really, really fast. It's, you know, not, not necessarily as fast as speaking, but it gets pretty close. But sometimes what throws you out of that flow state is that you have a question you know you want to answer and there's there's, you have the meta question of, will I get the answer in 2 minutes or 2 hours? And that can be, it's just frustrating to have to skip into a different level of abstraction and think about optimizing the writing process instead of what is the next word going to be. And by, I think ChatGPT helps keep you in a flow state. Um, I also noticed that with, with coding with, uh, with an LLM is that the way that it keeps you in a flow state is you always know what the last problem you were trying to solve is, both in natural language terms and in terms of code. So you can always, it's a little bit faster to just jump right back into where you were. Whereas if you remember the name of the file you're working on and you just open it, you are at the top of a file and you have to look through a bunch of things and try to figure out, okay, where did I get stuck? And then, you know, what are the linked dependencies to other things that got me stuck?

Yeah, also just as a, you know, pro user tip for other users of these things, it is one of the first tools where you can ask the tool directly, hey, what should I do next? And even a wrong answer to that question is much better than no answer to that question if you're doing the staring at a blank screen problem, or if you've lost your train of thought, or context switching between 7 windows to chase down that, uh, uh, the anecdote that you wanted from the congressional testimony in 2012 has blown out your own context window for, uh, what you were doing. Just like, it, it, it's effectively having an infinite patient coworker that you can bother just to have someone to be able to bother and not be like stuck in your own brain about And so, as someone who is aspirationally patient and has been a coworker before, I think one of the undermeasured productivity increases that we'll get from LLMs is simply less people tapping a coworker on the shoulder to function as a proto-LLM for things that computers are now perfectly capable of doing, and saving both the 1 minute of that interruption, but also like the vast context-switching costs that engineers and similar creative workers go through when their train of thought gets derailed and they lose 30 minutes of state built up in their head.

Yeah, yeah, it's definitely good for that. It's sort of, it's like code comments, but for your thought process where you have these periodic check-ins and you periodically sort of explain to yourself what you're doing. And like, you know, I think that the phenomenon, it's a little bit rarer with Google searches, but it does happen sometimes with ChatGPT that I'll, as I'm typing in the core question, I will realize I know the answer and not have to ask the question. So it's been good for that. I've actually, on the topic of chatbots, one of the things that's been very interesting to me is watching my kids use them because to my kids, LLMs have, of course, always existed. My 6-year-old once looked over my shoulder while I was Googling something and she asked, is that ChatGPT? And I've never heard something that would terrify viewers more than that. But it's—

no, it's much worse and it has ads in it.

It's interesting to look at how they interact with it. And like my, one of my kids, we're going to have a babysitter and hadn't had one in a while. And she asked ChatGPT what to do with a babysitter and it gave her a couple suggestions. And then she said, that's not enough. And asked Please tell me 100 things that I could do with a babysitter, and it gave her a numbered list of 100 activities that a kid could do with their babysitter. And then she responded to that with, "That is way too many things." So I think it is just a lot more natural to to view them as kind of kind of humanoid. No one no one really talks to Excel in the same way. You don't think of Excel as a buddy, but the the chatbots do. have something of a personality. Um, I was very appreciative of Anthropic's willingness to, uh, give, give us a chatbot that didn't just have a personality, but actually had a special interest in the Golden Gate Bridge and could steer any conversation in that direction. Um, it, uh, it felt, I felt, you know, represented.

That, uh, was a, a wild art piece, uh, in a lot of ways. Uh, but, uh, not just an art piece. We fundamentally don't understand how this technology works under the hood. It's been, what's the phrase, discovered more than it's been invented. And so some big picture questions, do we work like that under the hood? Is that why folks like us end up being a little bit too interested in credit card points, et cetera, et cetera?

Right. Is there just one overgrown neuron somewhere that has lots of knock-on effects? Yeah, it's, I think that does get to a kind of deep question about how you think about these models, because one view is that they are, they're getting very good at simulating something in the way that an increasingly detailed image simulates a real thing, where at some point it is sufficiently detailed that it's like the main thing that's salient is still it's 2-dimensional and adding more detail does not actually make it feel more real. But in another sense, they just, they do a lot A lot of their sort of, you're going to humanize them and call it mental shortcuts. A lot of their mental shortcuts are very endearingly human. So there was a, when the New York Times sued OpenAI and they had this example of OpenAI being able to reproduce paywalled articles, but it turned out that those articles had also been copied elsewhere and were just in broader corpuses. One of the interesting things about it was that the articles, when they were generated, they would be almost word for word, and they would use years correctly. But, um, in one, at one point in one of the articles, it was about, um, Uber and taxi medallions and things like that. There was a sentence about how much, I think it was how much money taxi drivers had borrowed against their medallions and how much that number had changed and default rates or something like that. And, uh, the LLM just skipped it. And that felt incredibly relatable to me because you can anchor a lot of things to years. Like a year is 2008 means something to a lot of people, but a number like $357 million is just not, it's not a token in the same way. It doesn't have the same salience. And if you are trying to relate some anecdote, you might remember the year in which a law got passed because you remember, oh, that was the year I moved to this place or got this job or met this girl or whatever. But you're not going to remember that number. So you might just drop that, that particular sentence from your, your attempt to relate this story to somebody else. So that did make me feel like they are replicating, obviously it's a different implementation, but they're replicating some things that our brains do. Or maybe our way of thinking has been so influenced by language and by writing that we've actually manipulated ourselves into having brains that are more similar to LLMs and that are more similar to taking a sequence of tokens and predicting the next token.

Yeah, it is somewhat convenient that we already had a word for hallucination before we saw the concept demonstrated by non-human intelligence. But be that as it may, I do think whether one is a 6-year-old or a professional user of LLMs, understanding that it's a tool that you can argue with, which is not a mode of interaction with many tools, perhaps for programmers, but it's largely a one-sided argument when you're screaming at the compiler saying, why won't you work? The phrase, no, you're hallucinating, try again. has worked it into a prompt for me many, many times. And similarly, when I'm using it in sort of search engine mode for the tip of the tongue, who was that trader who blew up? Was it a British bank? Not sterling, but it was definitely from Singapore and vis-à-vis futures or options, probably in like Tokyo or Osaka. And then—

It's going to tell you Nikkei-san.

That's the sort of thing that it is just preternaturally good at getting. But frequently it'll say, oh yeah, you know, that time in India, blah, blah, blah. It's like, no, the salient bit of information was this. And oh, Nissen, yeah, you know, Barings Bank. And then you have to Google for the Wikipedia article to verify the particulars because Wikipedia is much more likely to be, well, it's less likely to make catastrophic mistakes than ChatGPT is in my experience for those heavily detail-oriented things.

Yes.

And oddly enough, Google Search used to be very good for tip-of-the-tongue searches, particularly in certain domains. Like, if you described a movie plot, it would often nail the exact movie the first time. It's been much less good for tip-of-the-tongue searches, particularly tip-of-the-tongue searches that are seeking a particular document. I'm finding it difficult to even find particular documents by the name of them these days. My usual go-to example from this, which keeps disappearing from the internet in the obvious places, is the shorting the mezzanine tranches of the—

Subprime-backed CDOs.

Thank you. For a famous document that was covered in Big Short as the Jenga scene, but there's an actual PDF. It's amazing. And every few years, whatever combination of university webpages and Scribd uploaders, et cetera, that had it previously decay into bitrot, but there's always someone who has it. And Google has been unable to find that one recently, which had the times we were living in.

Yeah, that's, that's like one of my use cases for Google, which Google like is, is bad at for good reasons, is when I read a story about some event and I want to compare it to another similar event. And Google wants to weight the recency very highly because it's clearly a query about a query that is similar to queries people are doing for this recent event. So it will, you know, if it's something like this company got bought and I want to read about when they first got funded or this company raised their Series C and I want to read about the Series B, um, it will, Google will often very helpfully correct me and, you know, know that I surely meant to look for the most recent thing. And I think that actually makes sense if like the, if the median user who's doing a query like that, like the reason you would do that query is you heard about this thing and you heard about this event and you want to learn more about it. So of course they should show you the most recent instance. But I think that assuming that use case is a little bit— it's annoying. And then LLMs have almost the opposite problem where they're very atemporal. And I guess the RLHF actually adds some temporality to them where they're very bad at telling you what the median person would've thought at some point in the past on some issue. They sort of tell you What someone today would say they would have thought if it were 1950. But there is sort of an endless present for LLMs, except to the extent that years provide— the years are tokens that tell you something about the distribution of the next set of tokens. And because of that, you can get them to tell you about these historical events and to not overweight the recent past, which is also outside of their training window. I guess that what that amounts to is that there's a nice synergy between the two, where Google does have the actual document written by a human being. You know, if it's a— it's often a PDF of a scanned document that somebody printed out. You can tell it's— you can assume it's real. And then the LLM knows about the existence of this thing and can basically give you enough information to do a query where you put it in quotes and tell Google, no, I do not correct this. I am the correct one right now. I am not the median user. I am the weird user who has this special use case.

Yeah.

So yeah, using them together, I do think that actually ChatGPT has improved my Google efficiency.

Yeah, I would definitely agree with that. I think one of the reasons why people like us complain that Google Search has declined in quality over the last couple of years is we are experiencing our transition into middle-aged and extremely senior users in the Google ecosystem, where back in like 2006, we were the whales that were being optimized for in one way or another. And as Google has grown their business and the median user who gets weighted for no longer looks or searches like us, we are increasingly like— I think if you had a marketing meeting at Google these days and started to talk about binomial operators, the Google marketing department would shoot you and then throw your corpse out of the room. And so Binary operators, not binomial. Middle age, it's a trip. But for perhaps a brief window in time, we are the folks that LLM product teams are optimizing for. Or are we? Who knows? Looking under the covers of these things is somewhat difficult.

I guess there's a— it might be a feature of LLMs that they can optimize for many different kinds of users. In fact, they almost can't help but optimize for different kinds of users. There was a wonderful tweet recently where someone was interacting with the chatbot built into their smart, I think their health monitoring watch, smartwatch or something like that. And they said, please build me a to-do app in React. And the chatbot said, let's focus on fitness and exercise. And the user said, no. And the chatbot responded with React. So, yeah, you can, you can, you know, they, they are sufficiently general purpose that you can bully them into producing the JavaScripts that you want. They are implicitly customized for whatever use case you happen to have, unless it's a use case that conflicts with their rules about what kinds of content they can and can't return. So customized for everything except, you know, instructions on building explosives.

Well, on that happy note, I think that's as good a point as anywhere to end the conversation. Bern, for, uh, the relatively few people here who haven't encountered you before, where can we find you on the internet?

Sure. So my full name, Byrne Hobart, B-Y-R-N-E H-O-B-A-R-T, is also my Twitter handle. That is a good way to know some of my thoughts. And then probably the best, the best way to read me regularly would be at thediff.co, T-H-E-D-I-F-F.co. It's a newsletter. There's a free version. If you enjoyed this, I invite you to sign up. And you will, if you read the archives, you will see things that we riffed on in this conversation, talked about in more detail, but with fewer corrections. And then you will also get more thoughts on AI, financial markets. We actually didn't talk very much about, I guess we talked a little bit about financial markets, but yeah, lots on financial markets, how those work, what is new and different in them, and company profiles. much, much more.

All right, thanks very much for joining me. Absolutely.

Upstream with Erik Torenberg is a show from Turpentine, the podcast network behind Moment of Zen and Cognitive Revolution. If you like the episode, please leave a review in the Apple Store. Hey everyone, Erik here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries, from our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers like Econ 102 with Noah Smith. We're launching new shows every week, and we're looking for industry Leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's erik@turpentine.co, and let's partner together.