The Next Era of Fashion Design with AI & 3D | Dr. Huamin Wang

Hello. Hello, everyone. My name is Boming Wang. I'm the chief scientist at Style3D. And it's my great pleasure to be here. This is actually my first time at PI, but I can see passion and enthusiasm from everybody about technology. So I'm very happy to share my experience and my understanding of the future of fashion design and in my view. The future will be defined by AI plus 3D together. So first, I would like to briefly introduce myself and our team. So I'm the chief scientist, of course. But before that, I was actually a professor at Ohio State for 11 years. And I graduated from Georgia Tech and Stanford. And I do research exactly in AI and 3D. So this is exactly the field I'm working on. We actually have a very strong research team. We have about 20 people. Most of them are PhDs or masters, and we file patents and we publish research papers every year. So as you can see, we are highly recognized by the research community. For example, this year we have 5 papers being accepted to ACM SIGGRAPH. In case you don't know, SIGGRAPH is the top conference in 3D graphics and AI technology. So you can see how strong we are in terms of research. So to talk about AI and 3D together, I would like to first talk about AI and 3D, but separately. So I think we started the journey of adopting 3D technology by the fashion business for many, many years already, right? But I think it's okay, but I don't really think this is a success because when we started developing 3D technology for fashion, we saw that 3D garments can be very useful just like physical garments can be used everywhere by designers, by manufacturers, by even retailers, online business. But the truth we found today is that Most of the time, 99% of the time, 3D garments are only used by tech designers for evaluating patterns. And this is very unfortunate because that quite limits the application, the usability of 3D garments. So we look at this problem. Why does this happen? And there are two reasons. For designers, it's difficult to use 3D garments because simply they don't know how to do 3D modeling. To do 3D modeling, you have to know how do you model 3D objects. Especially, you need patterns to do 3D modeling of digital garments. And today, a lot of designers, they are not familiar with pattern making. So that becomes the first issue. The second issue is the realism. 3D garments, they are OK. They are good for pattern evaluation. But if you look closely, you will see there are lots of details missing. There are lots of details. Incorrect, right? So to fix those realism, it's a very, very difficult problem. I've been working this for many years, 15 years probably, and I know how difficult this problem is. And even if you know how to do it, you have to spend a lot of time. You also have to hire people with very strong knowledge about 3D modeling. So both problem exists that makes 3D garments difficult to be used by designers and retailers and online business. So that's the situation of 3D. Now what about AI? We found that AI is quite the opposite. We have seen AI being very useful by designers to visualize their ideas, to come up with inspirations. So this is okay, right? There are lots of AI models being able to do that. We have also seen that retailers, online business, being able to use AI models to generate images, videos, and even virtual try-on using image-based solutions. So we have seen AI models being useful on both sides. Now one question is, because we're, we're using images, we're using AI models, for designers and for retailers. Can we go from design to retail directly? Now if that's the case, that basically means that we need AI models to generate images for retailers based on the images generated by AI models repeatedly. And if that's the case, then we made the first problem. It's called AI hallucination. It basically means that when you use AI models to generate images multiple times, you're going to see inconsistency. And that's the issue because the images generated by retailers may not be the same image, may not be the same garment designed by the designers, right? So that's the first issue. The second issue is that if you're going to just work everything on images, that means you don't have 3D garments, that means you don't need patterns, but that also means that you're losing the connection with manufacturer. So these are the problems we found very critical to the AI solutions today. So if you look at this page, this slide, and the previous slide, you see that AI and 3D, they have their own strengths and their limitations, but they're all complementary to each other. So in our vision, we think that the future will be AI and 3D together. They will coexist for the business. And we think that the whole pipeline will be like this. For designer, they come up with inspiration, and it goes to AI-empowered 3D modeling and simulation, which can be connected to manufacturing. And then it can go to retail for image video generation. And to meet this objective, we need two important components. The first component is using AI to make resources, especially patterns, more easier to create and more accessible by designers. So that's the first challenge we have to address. The second challenge is realism. We want to make images much more realistic for 3D garments. So when you render images, of a 3D garment, you want to make it realistic. So today I'm going to talk about the middle part, so from pattern making to 3D modeling, and finally image synthesis. How can we make these three components easier? So let's start with pattern making. So to do pattern making by AI, the first thing came to our mind is to just to ask ChatGPT how to make a pattern. OK? And actually, ChatGPT will give you an answer. And here's— I have a picture here. So if you ask ChatGPT, how do you make a pattern? It gives you a bunch of descriptions, instructions to make a pattern. But these instructions are written in natural language, in our language. So you can't really execute them to make a pattern. So we found that, yes, ChatGPT can describe the procedural, but it can't really make a pattern. The second experiment we did with ChatGPT is to draw pattern as an image, because we know that ChatGPT has this image generation feature, right? So we asked ChatGPT, can you draw me a pattern as an image? And yes, it draws images. But as you can see, they're just images, which means that you can not modify those images directly, just like a regular pattern. And to make things even worse, if you look at the rightmost image, it's not even complete. It's cropped. So that's what happens to ChatGPT because ChatGPT has no idea about what it is actually doing. It just tries to create some images that looks like a pattern, but it doesn't have— doesn't know if it's really a pattern or not. So that means the third problem is not accurate enough. It's not professional enough. Okay, so, so we think that large language model like ChatGPT, it has very great potential because it can think like a human, but it's not professional enough because it's not trained by human. So our solution is going to train large language model like ChatGPT to behave like a pattern maker. So this is our goal. So to do so, we use a large language model with multimodal input, which means that it can take text, can take images, or even sewing patterns to be an input. And it also has a feature, very important feature, called chain of thought. Basically means that it can think like pattern maker. It can give you reasons behind what it's doing. The output of this large model is going to be instructions like before, but instead of writing those instructions as text, as a natural language, we write those instructions as script code. And these script codes can be executed by a pattern CAD software, and this software can eventually turn those script code into sewing pattern. So that's a very natural idea coming out from ChatGPT. You know, we found that they can write instructions, so we just write script code for CAD software. The key question here is how do you train this large language model to be professional and to write script code? So to do so, we use existing script code, for example, like a Garmin code, and we also use a lot of textbook and documentation so it know what previously, what those patterns look like, right? But obviously that's not enough because we know that textbook do not cover everything. There are lots of hidden knowledge by, you know, pattern makers. So this is the third part is very important called reinforcement learning. What does this mean? Basically means that we hire a bunch of pattern makers and we tell those pattern makers to use this language model every day, and the large language model will give pattern makers results, and the pattern makers will give feedbacks. So the pattern maker will basically tell large language model, you're— this is right, this is wrong, and then pattern makers can improve its skill, uh, a large language model can improve its skill to make the, uh, the result better. So this is called reinforcement learning, and we found that it's very, very useful to make the result, the pattern, more professional. So it's just like humans, right? It learns like a human from practice. So we published this paper at this year's top AI conference called CVPR, and this is a screenshot from our system. On the left side, you can see the user can do text prompts or sketch or image. And on the right side, it can create the pattern based on the input. Please just forget about the result here because this was done a year ago, and we have actually improved this a lot. But we found that this actually works. You're very welcome to come to our booth and see a demo after the talk if you're interested. But our plan is that by this time next year, we're going to formally release it. And it's also customizable. So if you have your own data set, if you have your own pattern maker and the pattern data, we can customize this for you. So this is about pattern making. To build the 3D model, to build the 3D garments, pattern is one important resource, but it's not the only resource, right? For example, we also need trims. And in the past, if you want people to model 3D trims, you have to know how to do 3D modeling. And that's something needs to learn. So to solve this problem, we also develop an AI model that can generate 3D trims from images. For example, if you want to make a button with a flower shape, the model can create that. Or if you want to create a buckle with a special shape, you can also use the AI model to create that. So this feature has already been developed, has already been integrated into our software, and hopefully will become available for use pretty soon. And also, we also developed something for fabric finish, For example, if you have some graphic design, or maybe you have a logo you want to try different finish effects, you can use the AI model to do it as well. So what you need to do is to upload a graphic input and a finishing effect, and AI model can generate the corresponding result as an image. And you can apply this to the garments to see different logo or maybe different graphic finishing effects. So these are all available in our software already. So that's about resources, especially Patternmaking. The next question is, how do you actually do 3D modeling, 3D simulation, especially with the help of AI? And why do we need the help of AI? Because today, I think 3D modeling is still a very difficult problem. You need people to do it, and you need people know how to do it, right? And why it's difficult? There are several reasons. First, when you have a pattern, it's difficult to assemble those patterns, those pieces, those panels into a garment. You need to define those stitching relationships, and also you want to, you need to place those panels around the body so that the 3D simulation can simulate the garments correctly. And if you have multiple layers, if you have many, many small pieces, many small panels, this can be very time-consuming, and you may have made some errors. And the second part is about the physics. To do the simulation, you need a lot of simulation parameters. You also need to specify a lot of fabric physical parameters. And these can be very difficult to do it right. So we would like to address these problems also with the help of AI. So to begin with, let's talk about garment assembly. How do you assemble pieces into garment? To solve this problem, we think that, uh, we can start with a foundation model. So what is a foundation model? The foundation model basically means that we come up with an AI model that can take whatever the user provides to it and then generate 2D pattern and 3D shape together. If we're able to do that, then it's very easy to produce a simulation-ready 3D garment because the 3D shape has a lot of information. For example, if two panels are very close to each other, you should just stitch them together, right? So in other words, we can easily find the stitching relationship from 3D shapes, so we can make the result simulation ready. Again, the problem is how do you build the AI model? So we build this AI model based on the data, and we found that it can be used for different— all kinds of different purposes. For example, you can use this for pattern generation. The difference between this model and the large language model I mentioned earlier is that this is based on— completely based on diffusion model, so it may have AI hallucination. On the other hand, if you think about the large language model, it doesn't have AI hallucination. It's purely based on instructions, so that one can make sure it's precise, controllable, while this one is mainly for quick visualization if you don't care so much about the quality. So this is one usage of this foundation model. Um, the next usage of this foundation model is for 3D assets. For example, you may have some 3D point clouds from 3D scan, or you may use AI to generate maybe some, uh, 3D digital avatar with some garments, you can use this foundation model to turn those 3D assets into 3D garments. So this is another use. The third usage of this foundation model, which I think it's very, very important, is to turning sewing patterns into 3D garments automatically. Especially for business, if you already have a lot of existing patterns in the data set, you can use this foundation model to turn those patterns into 3D garments. And let me show you, uh, example here. So here the user can upload a pattern DXF file into the system, and then the foundation model can start doing stitching. So it automatically finds all the stitching, uh, relationships among those panels. And it also finds the initial 3D shape of those pieces. So you can see that it's automatically decided. And once you have all this information needed for simulation, it can just start simulation and it creates a 3D garment from the pattern. So that is great when you have the foundation model. When it works, it's great. But what happens if it doesn't work, right? Because AI always has something unpredictable. So in that case, we studied this problem. Uh, last year we had a paper is that when you only have partially correct information provided by AI, how do you again assemble the garments from patterns? So as you can see here on the left side, we have the pattern and this algorithm will repair what AI did to eventually complete the garment. For simulation. OK, so we have a plan B basically for garment assembly. Now what about plan C, right? So what if the shape is really, really complicated? For example, if it's maybe knot or ties, you can't really use AI to correctly decide the shape. So in that case, we also develop an algorithm allowing user to I won't say fully automatic, but fairly easy to create those knots and tiles based on heuristics and templates. And we published this paper at SIGGRAPH this year. You can see that we can use this algorithm to generate those knots and tiles within a couple minutes, I think. So it's not fully automatic, but still, it's very easy to use. So I hope this convinced you that garment assembly can be done today by AI model. It's improving. I won't say it's still ready. It's not ready yet, but it's improving. And hopefully in a year or maybe 2 years, it will be ready for use. Now, the second problem, which I think is also a very difficult problem, is how do you get all these parameters needed for simulation, right? I think the parameters, we can classify the parameters into two categories. One category is the simulation parameters, for example, grid size, for example, time steps. I'm not sure if you're familiar with those parameters. But if you're not familiar with those parameters, I hope you don't need to know those parameters in the future because we're going to make them fully automatically decided. For example, this paper, this project, what we did is to automatically decide optimal grid size for simulation. And we'll keep doing those research. So in the future, hopefully, all of those simulation parameters can be hidden. So you don't actually have to specify those parameters. So we want to solve this problem. The next problem is the fabric. The fabric has very complicated physical properties. So we also spend a lot of our research efforts trying to precisely measure their fabric material properties. So we develop all kinds of different devices, and we develop AI algorithms to measure those parameters. Now, for people with the hardware measurement device, we do provide those projects, and we do provide those algorithms for that. But we also know that in many cases, these measurement devices are not really accessible for everybody. So we want to also use AI to decide parameters of fabrics without actually doing the tests. So this is an ongoing project we are also trying to explore. And hopefully, I can disclose some of those updates later in the future. OK, so these are about 3D modeling. And suppose we create the 3D garments, the next question is, how do you generate images? How do you make images and videos realistic? So we consider two ways to do it. One way is to do it in 3D. In other words, we want to make 3D garments more realistic. For example, here, let's say on the left side, we have a very basic coat without too many details. And on the right side, using the AI and 3D algorithm, we can generate those wrinkle details. And it can be very easily done. So you don't need the designers to— tech designers to specify all these simulation parameters to create those details. We also want to do it using 2D solutions on the images. For example, on the left side, we have the original rendered image. And on the right side, you can see these enhanced images using our algorithm. And this is also a research paper we published last year on NIPS, which is the top conference in AI. And this is also an available feature now in our software. Is called AI enhancement. And here you can see that this enhancement feature can improve consistency for image generation. For example, if you change the camera view, or if you change the facial expression, if you change the body pose, you can do this very consistently using this feature. So this is basically the pipeline. I talked about from design to manufacturing to retail, you can see that it involves a lot of steps and a lot of components needed to be developed, or we already developed those features. So it's a fairly big system. And if you look even closer, you can see that it actually contains a lot of small AI tools or features. For example, for users to— for designers, we do image to sketch and sketch to image. And we also have all kinds of different features to improve garment design. For example, can generate colorways, can generate fabrics, can do local modifications. And for virtual trial, we can also add avatar to the image can also change the background, change the body pose, change expression, even change hair. So you can see that this is a very big system. And when we have a big system, even though it's very powerful and we are very proud that we developed this system, we realize that for the business, it's not so easy to use because you need people to learn how to use it, right? When people come and go, you have to retrain people. And for business in a very dynamic market, we're changing our business model, we're changing our trend, we may also change our workflow all the time. So the key question is, how can we make this system easier to use by the business? So our solution, I think the thing that we're working on is called the fashion agent, AI agent. What does AI agent mean? Well, it basically, it's like a ChatGPT. You can talk to the ChatGPT and you can be a designer, can be tech designer, can be pattern maker or, you know, whoever in the business. And you can talk to the ChatGPT in different ways. You can do voice, you can do text, you can do image, DXF files, whatever, can take whatever input you would like to do. And what happens to this large model is that behind this large, large language model, we have a bunch of smaller agents responsible for individual work. For example, for design, for pattern making, or maybe for 3D modeling. So this agent, you can think of this agent like your AI employee which already learned how to do this jobs, and all you have to do is talk to this agent, tell agent what to do, and agent will do the work for you. Okay, so this sounds to be a very fancy idea, but again, the question is how do you get this? How do you get this to work? So training is very important. To do it, we use a lot of public data accessible, and we also do customization, and actually it has to be customized for different business. Because every business have its own data, has its own employee, so you don't want to share your employee with others, right? So the same idea, it has to be customized. And it's also augmented in real time using web search, and more importantly, it's reinforced. We also do reinforcement learning. In other words, when you do conversations with this deep model every day, it will improve its skill, it will improve its reasoning to make it work even better. And let me show you our preliminary result of this agent. For example, here on the right side— on the left side, you can see it gives you some hints about what to do. And for example, the user uploads a sketch and then it asks, what you want to do next. And in this case, the user wants to generate a rendered image. And then it generates a rendered image. And it gives you some options. And the user said that I want to change the color. So the agent will change the color. It gives you the option, but you don't have to stick to those options. You can do whatever you want. You can talk to it whatever you would like to talk to, just like talk to a human being, right? And it will understand your instruction and perform this instruction for you. So here it wants to generate a colorway, so it generates a colorway. So let me— Go through a little bit quickly. And here, want to do a try-on of this example. So it tries on to a human image. So this is a prototype of this design agent. We have a bunch of smaller agents, but as you can see, this is going to be the— AI employee that works for you. So I think in the future, today we have a lot of AI and 3D tools, features being developed already. And I'm pretty sure that you have seen a lot of those. We are very proud that we are leading this. We have a lot of those tools developed by us. But I know that in the future, in the long-term future, these features, these tools will be more or less the same. What matters is their synergy that can be provided by a comprehensive system with all of these features, and it's made possible only because you have an agent. So let me quote what Jensen Huang, the CEO of NVIDIA, said at the beginning of the year.

The IT department of every company is going to be the HR department of AI agents in the future. Today they manage and maintain a bunch of software from the IT industry. In the future, they'll maintain, nurture, onboard, and improve a whole bunch of digital agents and provision them to the companies to use. Okay, and so your IT department is gonna become kind of like AI agent HR.

[Speaker] So I think the age of AI agent is here, should start early and hopefully with us because we're leading research in this direction. Thank you. Yeah.