Sridhar Ramaswamy, Snowflake's SVP of AI, explains how the company's Horizon project uses AI-powered search over metadata to help users find data across hundreds of thousands of tables without exposing the data itself. Natural-language search lets people discover datasets and request access through automated permission systems like future grants.
Chapters
I'd like to get into some of these applications in a bit more detail. So maybe we'll start with search, since that's your forte now. I speak to an awful lot of chief data officers and the one thing every single one of them complains that is the data across their organization is stuck in silos. They have all this data, no one really knows where it is they can't access. Feels like AI and search is going to help with this. Can you talk me through how it's going to help?
100%. So, you know, some of our larger deployments of Snowflake have a hundred thousand tables. That's nutty. If you're like, oh, I want information about this specific topic, where should I look? It's really, really hard. Usually all of these devolve into a giant Slack channel in which you're like, hey, I'd like information about this project. Like, does somebody know something? It sort of comes down to it. And tools like Google don't really help because they don't have the kind of deep context into specific data sets, what are the semantics of it, and so on. So we have an. At Snowflake, we have an ambitious effort called Horizon, which basically makes sharing, creating of shares, sharing data within an enterprise just like a whole lot easier. You can attach, you know, we help you figure out semantics. We help you figure out, for example, is this column email addresses? Is this column other kind of pii? Of course, you can also put information about tables, about schemas. And we have this effort to make it really easy for you to search through the data sets again in natural language and get to the data. Of course, access control is a big, big deal. And no company is going to say, in the name of making data easily visible, I'm going to make everything visible. That is also a disaster. But what we're doing are clever techniques by which you can search over the metadata and figure out, ooh, there is this data set, but I actually don't have access to it. How do I request the owner of the data to provide me with access? Because my request is a legitimate request. So we think about the life cycle of data discovery and then how that subsequently drives data sharing. And I think this is the kind of stuff that is going to be helping a whole lot. And then other aspects of AI that we will get into then will make it easy for people to be able to quickly query that data. Part of the objective of snowpilot, the copilot effort within Snowflake, is that it should be able to Use things like the previous queries, the context from the experts on any particular schema to help future people write SQL in an easier fashion. But it all starts with having the data in the right place, having metadata attached to it, and making it super easy to discover and share data in a controlled way. And that's what we are doing with Verizon.
That's really interesting, the idea that even if a data set needs to be kept private, you can still make the metadata public, or at least slightly more.
Visible across your organization, the metadata searchable within the enterprise. In other words, you can separate out the two. Searching over the privilege to search over metadata is different from the privilege to actually be able to run stuff on it. And again, we are in the business of providing data owners, enterprises, our customers, with the right tools. We think this is an interesting differential. By the way we obsess about these details. We even have something called a future grant where we basically say, I want to give access to this particular schema, let's say about like revenue data from Europe to Richie, but I also want to give the same access for all future tables that I'm going to create in the schema. Because these things are living, breathing things and, you know, as new things come on, you want to keep that access. Again, that's a choice that business owners can make.
Okay, so data access management is one of those things where it feels like it's no one's idea of a fun time. So maybe you don't want all this stuff automated so people aren't having to mess about with some of the technical details.
Yeah, I think, but it's a matter of providing the right level of abstraction. You know, just saying everything is open is clearly not going to work. On the other hand, people are realizing that saying everything is closed doesn't really work either. So it becomes a question of what's the right level of abstraction that you offer to the administrators, to the business data owners, so that they can responsibly manage how data is shared. I always think of process as like, you know, it should be just enough, not too much friction, but not too little, you know, friction. Everything is open either. You know, that's like the magical Goldilocks situation that we try to get our customers into.
