Spatial Computing Catalyst #11: Inside Sketchfab, the Search Engine for 3D things

It's an honor meeting you. And who are you?

I'm Albon, the co founder and CEO of Sketchfab.

And Sketchfab is the center of the 3D world I call it. Right? It's.

Yeah, it's a platform to publish and find 3D content on the Internet. Our core product is an embeddable 3D player you can plug in on any website, e commerce, blog, social, to add an interactive 3D view of an object or a scene.

And since I was last here, I don't know, two, three years ago maybe. When did you start?

We started in 2012 back in France. So it's been been seven years now.

Seven years, yeah. It's like going to be an overnight success story like all the others. Right.

We started, we were way too early and challenged us to still be here today and here we are.

Yeah. What has been happening since I was here last in terms of your stats, you know, you. You're seeing uploads of people building entire like villages now and uploading them to SketchFab.

So we recently passed 2 million users and 3 million 3D files. So we're the largest library of 3D files on the Internet. And I mean in terms of general trends, 3D capture definitely accelerating. More and more people using photogrammetry software or real time 3D capture as well. And then traffic is growing. We passed billions use powered by sketchfabs.

Billion.

Yeah.

I think when I was here last time I was in the hundreds of thousands. So you're on an exponential growth curve.

Yeah, it's nice.

It's going up over time. And I bet the polygon density of your models is going up over time as well, right?

Not really because I mean as we've become kind of the de facto platform, our core users have learned to kind of optimize their content for it, like for the browser in general. Of course we keep improving the pipeline to better support larger models, but otherwise it's been mostly about educating users that a model of 100,000 polygons is going to run better than 1 million polygons. They might as well do that if they can. And we've also improved our integrations with other software. So typically on the photogrammetry side, which is usually very polygon intensive, like our integrations with software like RealityCapture or MetaShape, they automatically offer a decimation resolution that we know is going to work well for sketchfab and so they send us content is ready to go. That being said, we're working on a new pipeline completely separated It's a long R and D project funded by the Frame French government, actually, which is called Massive by sketchfab, which is a pipeline that accepts unlimited file size and you upload a file that can be 10 gigabytes, billions of polygons, and we tile it into small tiles and we generate a level of details and then we only show you in the browser the part you're looking at. And we hope to. Yeah, we announced it a few months ago and we'll have more news around this at Siggraphs this year.

Wow. So we're about 22 months before I think we're going to be able to buy an Apple pair of glasses. So the exponents, I predict are going to keep going for your business and keep going up in pretty crazy jumps. Every from here on you survive the desert or the dry period. And now you're going to see some really interesting things happening in the industry. That's my thinking. Does that agree with you?

Yes. I mean, you know, we've been around for a long time, seven years. And every year or every other year there is a new Trend related to 3D. And so, you know, start 2012 and then in 2014 there was a VR hype and then the 3D 3D printing hype, sorry with MakerBot, and then 2016 the VR hype and then 2018 the AR hype. And what we do is kind of at the crossroads of all those trends and we try to not be too dependent on either of those trends. And so while AR and better AR glasses are definitely going to provide better ways to consume our content, the bulk of our traffic and use cases, I mean, we have a ton of usage just of the regular web with regular free assets on regular e commerce websites. And I mean, we'll be ready whenever we need to be ready for the full next computing wave of spatial computing. But in the meantime, we're making sure to build a robust business and platform for use cases that have been here for a while.

Yeah, that's what people like me do. We take you two or three years into the future and try to predict what's going to go on. But this summer we're seeing games like Minecraft Earth and Harry Potter's new game from niantec. Right. That's going to bring mobile AR into a new usage. And we're seeing a lot of apps from 711 and Nike and all sorts of different commerce companies who are turning their products into 3D and starting to think in 3D. Right?

Yeah, I mean, we're I guess we're playing on two levels here, because as more and more brands are investing in digitizing their products and their catalog, more and more potentially sees a need for sketchfab as part of their needs and workflows. And then as we've reached kind of critical mass of content with 3 million 3D assets, we've also started exposing those assets to be reusable in other projects. And so, for example, we're integrated in Spark ar, which is Facebook's AR platform. We power their AR libraries. You can search sketchfab within Spark AR to easily bring in assets to build.

Tell me what Spark AR is, because several of my friends have said that's a pretty significant new platform. Tell me what it does and how a user would use sketchfab in it. Do you agree with them that that's a pretty, pretty significant new platform?

Yeah, I mean, Spark AR was released by Facebook a year ago, and it was pretty much mostly under the radar for a year until they opened it to Instagram. And so now you need that to post an AR filter on Instagram, which of course gives you a massive reach. And so it's essentially a software. It's a 3D software. It's not for everyone. It's a special software. And then to build filters and to bring in assets. Of course, you can bring 2D assets or videos, but then if you want to bring in 3D assets, either you can import your own or there is a sketchfab search bar within the product, so you can search whatever you want, a dog mask or your own assets. Typically, I do a lot of 3D capture and I can search for, search my own stuff, my own sketchup stuff from within the Facebook software, then bring it into their AR editing environment to generate custom behaviors and triggers, and then post that to the Instagram or Facebook camera.

Very cool. You do a lot of 3D scanning. Tell me about what you're doing with 3D scanning and what you're seeing others do with it. How is the usage of scanning things in 3D, like volumetric photography or whatnot? How is that changing and how is your usage of it changing as well?

Yeah, so for me, I love 3D scanning. I scan mostly three things. My kids. I make 3D portraits of my kids, and I think they exist in 3D in the real world. And it's amazing. I have three portraits of my son when he was three days old, and I can revisit zoos in VR or AR. It's just amazing. I like to scan vehicles in my neighborhood. And motorbikes. And there are a lot of. I live in Brooklyn, there are a lot of very old cars that are not made anymore. And I just find fascinating objects that I want to preserve in something better than a photographies. And then I like to scan food just because it's a cool thing. Just like on Instagram you like to post photos of food. And I think in 3D it's even nicer. I guess over the years as I've learned more and more about how to use these tools, you always want better quality and so you need to feed the software with more pictures, higher res pictures. And I used to only do scans with like 50 or so images and now I use typically 200, 300 images. And so it's much longer, but the results are much nicer. And once you've reached a certain threshold, you want to keep pushing the bar in terms of the overall community. I think more and more people are hearing and discovering about photogrammetry just for sketchfab. It's a sheer volume of content coming from those workflows. They realize it's available to today, it's possible today, it's available to anyone, it's easy to use and you might as well just get started. There are more and more new photogrammetry software on the market. I think traditional 3D people are also realizing it's good for them, not hurting them. Just like traditional photographers can use Photoshop to be better photographers, traditional 3D artists can use 3D scans to work faster or get references and stuff like that. And then the other big trend is of course around real time free capture coming to our smartphones. Latest iPhones all have a depth sensor and so that's what scandi is built upon, which means anyone with an iPhone X has a native built in 3D camera in their pocket. It hasn't been really exposed by Apple that much because technology today, they only use it for the face and looking and stuff like that. And certainly front facing. And so of course to do good free capture we wanted back facing as well. But it's coming from their acquisition of PrimeSense back in 2013. They were the makers of the Kinect. So they've spent the past six years bringing this tech into the iPhone. And next step is easy free capture for anyone. And then twiller knock a lot. Amazing workflows, digitize the real world, publish it on the web on sketchfabs and consume it in AR in a completely different part of the world.

Yeah, I hear we're going to get a back facing 3D scanner this year on the phone. And then when the 5G iPhone comes out next year, we're going to get a whole bunch of new things, including. I'm hearing that the glasses are going to come next year, but we might not be able to buy them until 2021. But they'll announce them with the 5G iPhone, is what I'm hearing. But which makes sense, right? 5G brings us all sorts of new capabilities, but these sensors are becoming key. And Scandi. You use Scandi on your. How do you scan your bikes? Because you do beautiful scans.

Yes, I mostly use photogrammetry because, as I mentioned, once. Once you reach a certain quality level, you want to stay at that level. I use Scandi. So we're native Skand. Sketchfab is natively integrated with Scandi. So when you create. I can show you. Actually, when you create a scan with Scandi, where is my scandi? Sorry. There you go. When you create a scan with Scandi. So this is my daughter, for example. And then there is a sketchfab icon. And then in one click, I can. I can put my title description, and then I can upload to sketchfab title. And then within 10 seconds to scan your baby.

You just hold it around and scan it like this, right?

Yeah. And takes a few seconds and then generates a 3D file. And then you can.

When do you do the photogrammetry one? Do you do that? Because people. I know there's a few people who understand what photogrammetry is, but most of my audience probably has never done a photogrammetry scan. Explain the process.

Yep. So photogrammetry is just taking a lot of pictures around an object at different levels. And so typically, check, check, check, check, check all around at one level, and then another one at higher level, and then another one at a lower level. You don't need. You could do with just 10 pictures. But of course, the more pictures, the better the results. And then photogrammetry is just the process of stitching those images together.

And what tool do you use?

I mostly use metashape by agisoft, which is one of the few professional photogrammetry software available for Mac.

And how much does that cost?

I have no idea. I think they have a standard edition, which is like below 200 bucks, and then the Pro Professional Edition is like 1000. They also just started doing drone photogrammetry, which is really cool, which is the same process, but taking photos with a drone, which means you can fly your drone over a building and then just generate a ton of pictures and Then stitch them together to recreate that in 3D.

That's cool. That's cool. A lot of fun. And I'm noticing there's a lot more people who are learning these techniques and playing, playing with them and capturing different things. And a friend of mine who's really advanced photographer is taking that and putting that into an AI and teaching the AI how to, how to build generative generate new buildings or new cities out of the scans he's built. Right. Because he's built so many scans he's captured.

We're seeing more and more people doing like machine learning around our content. Where's the largest data set to virtual thing or the largest Data set of 3D hydrants, for example. So if you want to teach an autonomous car, what is an hydrant on the road? Like what's the best way to do that? I remember when the first. So now we have a marketplace to sell where you can buy and sell the assets. And the first item I sold from my 3D scan collection was a 3D scan of chocolate croissant parachute which I put on sale for $399, which is pretty much the price of the actual thing. I wasn't really expecting it to sell and then someone bought it within a few days and I looked at his website to understand why would he buy this thing. And he's doing a food app that gives like nutrition advice on food items. And so to do that it needs to teach this app about a lot of food plates and ingredients. And so he's buying a lot of food on sketchfab to be able to automatically recognize what is a chocolate croissant.

This is one of the thesis of my book is augmented reality is going to bring many new kinds of jobs. And that sounds like a new kind of job to me. Right. Scanning food in for 3D for training in AI, what food actually looks like.

Yeah, I mean there are so many things we can't even foresee. But yeah, there's this other company in New York, they are working with a lot of restaurants. So they digitize all their food items from their menu. And then they just did a deal with the Burger Bear Burger where you can see your burger in AR before ordering from their app.

What are you seeing happening in education? I'm seeing quite a few teachers using 3D objects on Merge cubes and other things to.

Yeah, I guess on our end we're using more and more like learning management systems in schools. I mean again, we're larger data sets of parts of the body of Dinosaurs or current animals and anything really. And yeah, we see people building curriculums around that. We have a ton of museums on the platform. Almost a thousand museums from the Louvre to the British Museum to the Met, and again, amazing learning material. Those museums always try to enrich the content with like 3D annotations and audio and things like that to really document the 3D assets. And it's been a great resource for a lot of people.

People, yeah. Back to the AI because I have a feeling you're thinking about this. You know, Autodesk showed me you can put two buildings in and then generate a thousand buildings in between those buildings, or even some ones that didn't exist in between those buildings. Are you thinking about how to use AI to train the system on new things and maybe be able to generate new kinds of objects from the AI that would look.

I wish I could say yes, but to be honest, I mean, we're a tiny team, we're 30 people and our plane is completely full with all the work around our web based renderer and making sure it's the best on the market and so everything you can do with the assets themselves, our goal is to keep it. Same thing with capture, like Warnup, really investing on capture because so many people are and are packed and our promise is to partner with all the ways to create content and be their best partner in terms of publishing and then partner with all the places to consume content and be the best source of content. If we start marching on either of those two sides, we'll do shittier jobs, send the others who are only doing that and then. Yeah, I mean, our goal is really to be central hub and let content be created somewhere, let content be consumed somewhere else and then just be the home on the Internet.

On Friday, I'm going to Betaworks to see synthetic characters. Right. They're starting to create singers that are completely AI generated. They look like a human, but they are not human at all. Right. Are you thinking about where that world might go and how they might store their personalities or even their polygons in sketchfab and share them with other people?

Yeah, I mean, I guess I am a bit. My angle has been more around. I can see a future where we all have our 3D portraits stored on a platform like Sketchfab. And over time. And so over time you can see a 3D view from birth to death, essentially and AOI tables and learn a ton of things around this. I think. Well, this is more interesting for me and has more use cases and the virtual beings Is. Well, I guess it's different. We are seeing some of those trends with the face filters in ar, like in Snap and Facebook. Of course it gives new ideas when you see those use cases getting a lot of traction. But on sketchfab itself, we have so many free portraits and free captures of actual humans showing gaps. That also gives a lot of.

Yeah, like deep. Deep fakes are one thing that I'm noticing. I mean, the Dali Museum ingested all the videos that Dali had recorded over the years of himself or other people. Right. Speaking at conferences or talking to the media. And then they trained an AI to be a virtual Dali. And when you go in the museum, a virtual Dali is talking to you. Right. Do you see that kind of stuff being stored in sketchfab in the future?

Yeah, I mean, well, definitely the. I don't know if it's going to be used for. Well, I guess it is going to be used for fake news and stuff.

Because if you have a 3D model of somebody, you can make them say anything or do anything.

We released animation support maybe three or four years ago so you can upload an animated file and they announced it. I published an animation of a 3D scan of myself which I rigged using Mixamo dancing like Gangnam Style. Like that. It's entirely computer generated and this got a ton of views. And a lot of people still think that I was actually recorded dancing this way in volumetric video. Like, no, this was just regular. In no way I'm able to dance this well.

So you're seeing a lot of people play with animation techniques and playing with AI and playing, feeding these things, making them animated different ways.

Yeah, and seeing new tools like Radical that is able to extract the motion from a 2D video like you did in front of a 2D video, and is able to extract 3D motion from the videos that you can rig a 3D character.

Yeah, we're seeing that on Friday.

Yeah, it's really interesting.

In fact, we put it. We're putting up a video of Radicle. It'll all get confused if I start talking about timelines because this video will be up in a week. But we have a video up of Radical and that's pretty amazing technology. Right. And so we're going to be able to train our sketchfab polygons to do all sorts of crazy stuff.

And I've seen two ways, like publish from Radical to sketchfabs and import measures into Radical.

How many different viewers do you have to make now? And do you see that number going?

Keep Going down. What do you mean?

Like, I just got an Oculus Quest and there's a browser out of the.

Books, same viewer, so.

Same viewer, yeah. And you can get, you know, Firefox that plays VR. And I got.

Yeah, I mean, Apple has.

Facebook has certain things.

So we have just one viewer. And that's one of the perks of the web. Well, at least the goal is that it's going to run everywhere, to be entirely cross platform, initially just for 3D, but eventually also for VR and AR. And so with WebVR now evolving to the WebXR specs, the same SketchFab URL is going to run across all devices. And so I got my Quest a bit early, of course, and sketchfab worked out of the box. There are some custom things we have to do. The first one is typically the controller, because all the headsets have their own controllers. And so we need to be able to recognize each device to show you the actual controllers you have in your hands. And so that's one custom thing. And then they don't all have the same. Same power because the Quest is more of a mobile device. And right now we have some work to do to optimize the performances for each device. We already do a lot of this optimization. Typically, when you upload a model, we automatically generate a number of texture resolutions. And depending on the hardware you're using, we're going to show you like a lower resolution on mobile, higher resolution resolution on desktop, and then yet another resolution on VR. And then we eventually want to do the same thing with geometry, and so automatically adapt the geometry level to the hardware you're using.

That's interesting because as 5G comes along, we're going to be able to view very high resolution or high polygon models. Right. And if I'm still on 3G or 4G on a bad WI FI link or something, all of a sudden I have to take fewer polygon models in my glasses of the future. And you guys are thinking through that, how to deliver a good experience no matter where it is.

Yeah, it's already in the works. And the goal is to be able to automatically target for each platform, to automatically identify the platform. Then we also have ways to bypass this. Just like on YouTube, you can move from low resolution to resolution.

Yeah. Are you mostly staying focused on being the database of this new 3D web? Are you going to start thinking about tools at all to make the polygons do fun things?

Yeah, I mean, we're not mostly focused on being the database. We're initially Mostly focused on being the best web based renderer and one of the byproducts of that is becoming the largest database. But it's a byproduct and so we invest a lot in terms of like on rendering and performances, loading speed and so on. And so keep adding new material, supports and so on. And then what's interesting with all the new viewers and all the new formats, you know, like USDC and GLTF and GLB and so on, is that in order to build sketchfab, we've already done a ton of work like unification work. We Support more than 50 3D file formats and then we spent seven years building our own converter. So we convert all those files into our own format which is optimized for the web. And now we're able to convert back to gltf. And so a lot of people are already using us just for this bit of code which is a transcoding pipeline. And again it's, it's a byproduct that now we have and could license it as a standalone product. We haven't really built it into a standalone product yet, but we have a lot of requests from large organizations. Typically all the places where you can upload a file they only support one format. But with our pipeline they could add 50 formats we support and then just use our transcoding service. And it's interesting because the same things happened in the video world and now as a byproduct we have this product for the 3D world and then we also have a pretty robust offering tool that people never see. If you're not using sketchfile, you can't really know about it, but we have an editing layer where you can add 3D annotations, spatial audio, you can define your VR scenes like set up, scale and so on. And then we have a ton of post process effects. So we like almost like Instagram for the 3D world where you can add sharpness, saturation, depth of fields with automated fake depth of field. It's really nice. You can add a virtual ground and drop a shadow on it to stage the model. And a ton of stuff that you can do within SketchUp.

More coming, I bet. As you guys come up with new ideas. What would you tell the industry right now that you're seeing or that they should be thinking about as this world goes more and more 3D every month?

Well, I talk a lot about photogrammetry obviously because it's like a longing way to create content. And then I also say, I also keep reminding people that 3D on the web offers more reach than pure VR or pure AR. And so like brands, they want to jump in on like the most futuristic experience, but as a result they're going to build very like custom standalone steps or kind of require to download an app or to have a headset or whatever. I think it's great. But I think step one should be three yards away, which is 100% rich. And then if you want to add on top of that VR, AR, which you can do with the assets you've made for any web based 3D experience, you can and you should. That's the order of things that make sense to me.

Yeah, if you walk in a shopping mall, you're not going to load an app for the shopping mall, except in weird situations. But web ar, you start up a AR app and start going around the mall and interacting with 3D objects. Right, cool. What else is going on here? Hiring? Are you raising new funds? Are you? What's the business like?

I mean, we are mostly. So we spend spend the first six years of the company mostly focusing on A building the best product, the best 3D player on the market, and B, reaching critical mass of content creators and content. And over the past years, over the past years, maybe the past 18, 18 months, we've seen more and more companies adopting the platform. And so now we're kind of doubling down on that in order to build the business and so exposing more features for companies typically refining our viewer API to build very advanced things on top of sketchfab, like product configurators so that you can load a boat and pick your color and your seating option and all that. So we're seeing really great traction around this, more commercial applications. And so we keep investing on that. And then now that we have a ton of content, we spend a lot of the past making sure all content created would be published on sketchfab. And now that it's there, we're looking at step two, which is distributing that content. And so we released our download API a year ago, which is what Facebook integration is about. So you can search sketchfab from there, but also from Unity, Unreal Torch, many other AR and VR applications. And then we released our store, which is a marketplace where you can buy and sell assets. And again, until very recently, we're only working on making sure content would land on sketchfab. And so it's fairly new that we're looking at this second part of using sketchfab assets outside of sketchfab. So yeah, our main focus is while still building the best 3D player, which are both the creators and the businesses. Extending the feature set for APIs, either Upload API or Viewer API, and then Building new integrations on the consumption side so we can search sketchfab in other applications.

Very cool. Well, thanks so much for spending some time with me.

You're welcome.

We love sketchfab. Marcelo was excited to come here and see his good sketchfab. You use sketchfab, right?

I do. I do use it and it's pretty cool. You know the way that you guys integrate with other apps.

Right.

Like I use an app called Lotus Ludus.

Yeah.

For presentations.

Yeah.

They let you integrate your models really quickly. Right.

Just.

Right. And you can interact with them. That's a really powerful thing. I think integrations is key. And it's. Get Trust is great for that. You know, 3D visualization for video games, like I'm seeing. We created video games back in the day too.

Right.

And having characters that you can get on sketchfab and then you can buy them out. And I wanted to ask you about that because, you know, there's people that gave them for free, there's people that charge for the models, and that's fine. Have you thought about Netflix business model? In a way, yes. I'm talking about bitter game developers.

Yeah.

We use a lot of models. Right. And then you can have like monthly fee.

Yeah, there's. There's gonna be all sorts of memes that are gonna get created with.

Content is key.

How many. How many different objects are in sketch? 3 million objects. Right. So imagine you're creating an animated film or something. Yeah, something. Some new. It's not even fun. We can't even think about old terms. We have to think about. We're building an immersive environment that we in interact with robot rolling around here.

And that's why I'm asking the Netflix market.

Yeah. Eventually we'd want to have a subscription offering as well. But yet another thing to build. We're making three different things. One is like the platform for content creators, one is like technology SaaS solution for businesses, and then one is the Marketplace. And those three things could be three separate companies all requiring like a team of 30 people who are doing all that at once. And so, yeah, typically we would like to do many more stuff on the store. It's mostly been a lack of bandwidth, but baby steps.

No, I get it, I get it. But this is great. I really like the detail on the models.

Like, you can tell it's interesting because a lot of people are impressed by the quality of the content, but we have quality content. Not only, but it's because we have a quality viewer. And over the years, at the beginning, great artists wouldn't want to put their content on sketchfab because our renderer wouldn't do justice to their work. And they were like, I'm only going to put my stuff here if it looks as good as inside 3Ds max. And so a lot of other companies are building viewers and platforms and they don't understand what secret source to get great content. Well, get a great render and then be good at integrations that you make the workflow seamless for the creators.