In this presentation, Robert Ta argues that AI adoption without standardization leads to chaos and limited business value. Citing a BCG study, he notes that fewer than 10% of organizations can point to a generative AI deployment delivering sustained, measurable value, while 56% of CEOs saw no revenue or cost impact from AI in the past year and 60% lack a formal AI security policy, exposing IP to leakage. He identifies a common emerging enterprise architecture pattern among leading firms such as Stripe, Uber, and DoorDash: an internal, API‑first AI gateway that abstracts all model calls. This gateway enables cross‑functional policies—rate limiting, authentication, model routing, cost tracking, and a knowledge/context layer—while allowing a diversified model portfolio (self‑hosted open‑source models for low‑cost tasks, frontier models for complex orchestration, and specialized fine‑tuned models). Drawing on Jeff Bezos’s “services‑only‑talk‑through‑APIs” memo, Ta frames the gateway as the product that decouples teams, protects IP, and provides predictable spend forecasting for engineering and AI executives. The talk concludes that building this internal AI layer is the meta‑solution for organizations seeking to standardize AI, control costs, and secure their AI assets.
Chapters
If you're an AI builder or tech executive, you're probably looking at AI adoption. And what we're finding is AI adoption without standardization creates more problems than it solves. And every engineering org is really adopting AI tools. Few of them have answered the structural questions that determine if it compounds or if it just creates chaos. This points to a study done by BCG. Less than 10% of orgs can point to a GenAI deployment delivering measurable sustained value at scale. 56% of CEOs report AI produced neither revenue growth nor cost reduction in the past 12 months. And 60% of orgs have no formal AI security policy governing employee usage of public LLM APIs. So your IP might just be leaking out the door. The organizations pulling ahead— Stripe, Uber, DoorDash— at least to our research based on the public information out there, they do one thing really well. They invest in infrastructure around their harnesses and a knowledge layer or context layer. Those are very different things than just worrying about who foots the bill for my tokens. A few things here from those companies, but they've essentially seen this emerging pattern out from public information for enterprise architecture, where if you're a developer with Cloud Code or with Cursor, your CI pipeline might call for adversarial code reviews or whatever, or you have an internal app, a product feature that calls an LLM model via API. What we're seeing is these enterprises are putting in the screws and tightening them around all model calls go through abstracted layer where we can inject rate limiting, model routing, cost tracking, et cetera. So the whole idea is if I have 100% of work needing to be done, maybe 20% of that is just some unit tests. Maybe I'd throw that into a self-hosted open source model like DeepSakura Llama. Maybe for planning or orchestration across many repos in a highly subjective, complex domain set of problems, I have a frontier intelligence model like Opus or Fable 5, or, you know, in the future Mythos, whatever comes out. Maybe that does the orchestration, the planning, and then maybe I have specialized models. Maybe I create them for my own super unique case. But either way, we're taking the casino out of the— you spend your tokens and we're saying, let's put something there that optimizes what you get out of the casino. And I think that's really interesting as an emerging pattern. It makes a lot of sense. And to me, from my enterprise architecture background, I've always seen API first. As a thing. The API is the product. So route all calls through APIs. Jeff Bezos very famously sees it, this is how AWS, Amazon Web Services, built its empire. He famously sent out a memo saying, "All teams will have their own services and you only talk through your services." And there are well-defined schemas and contracts that back them. And that's how they accrued IP and value was that decoupling was, no, you can't have humans with tribal knowledge in their head about how to glue together systems. The systems must be intelligible. We need to dogfood that. There needs to be a real developer experience in, in integrating systems. So API first was where that got traction, that idea. And now I'm seeing the same pattern of let's, to control this better and apply cross-functional policy like rate limiting, authentication, model routing, cost tracking, let's just have an internal AI gateway and all calls to AI stuff, throw it into there and we'll figure it out. This is the emerging architecture we see for solving the problem of I don't want a token max out of my budget, out of my mind. I want to have predictable forecast spend for my engineering org as an AI tech executive or engineering executive. And as a builder, if you're tasked with trying to figure out this problem for your engineering leaders, this is the meta that we're seeing emerge.
