Scientist Stories: Anton Tsitsulin, Unsupervised Graph Learning

So welcome to the first New York City local meetup of the Learning of Graphs conference. You know, I think we are a group of people excited about very similar topics, but also with very diverse interests in terms of applications for, let's say, underlying mathematics and all these things. So I think, you know, the local conference, you all know about it. It has happened, I think, 2 or 3 times in the past, I believe.

So far, or how many times the conference itself, but this is the first— twice, right? So that's correct.

But this is the first NYC meetup, and there have been several other meetups in various big cities. I hope this is going to be an event that will be occurring again in the future, hopefully sometimes here as well. Just like a short word about the host, right? So it is hosted you know, by NJIT, in particular, the computer science department that I happen to be a member of. In case you do not know it, I mean, some of you know it already, that NJIT is in the top 50 public universities in the United States and very active in research. I should say that we're always looking for great faculty. So for those of you who are in the position to be looking for jobs, you know, consider us. And for everyone in the room, you already know that, but for those that are online, you do not know that we are in a very impressive location in Jersey City with great views of the city and New Jersey. So I'd like to say many thanks to the student organizers. You know, like, I've been helping very little with all of what you're going to see here. People responsible for producing this. I'm just going to mention the names: Ali Parviz, Yanqi Yu, Jacqueline Marsh, Yangbang Wang, Yingcheng Wang, and Derek Lin. I'm sorry for mispronouncing some of the names. It always happens. And I'm just going to introduce the first speaker, who would be Anton Tsitsouline. Biography: Anton is a research scientist at Voodoo Research in New York City. His research interests are scalable principled methods for analyzing graph data in an unsupervised manner. He received his PhD from the University of Toronto in 2021. So thank you, Edward.

Go ahead. Thank you, Ioannis. And yeah, it's hard to underemphasize how impressive the location is. So I'm very excited as well to talk about graphs in New York. In majors, actually. So the title of my talk is Large-Scale Unsupervised Graph Learning. And I'm particularly excited about the unsupervised aspect of learning with graphs. So that's what I'm going to talk about. So I'll start with a quick overview of what Google Research NYC is and the teams work. And then I'll try to speak about unsupervised learning in theory and practice. Select interesting theoretical developments that we've been doing and some interesting practical stuff that I think is interesting for graduate students around here. Then I will talk about how we use modern machine learning accelerators for graph machine learning. And then I'll, you know, a short plug for our library for graph machine learning, graph neural networks. So, you know, I figured this is the first— I'm the first speaker, so I have to do this just once. Graphs, in case you don't know, are representations of relationships between entities. And they have like a very different number of edges. There are different node types and they all depict very complex structures of very different things, not just initiatives. And, you know, there are two types of graphs that we love and care about. One is far more prevalent in academia, relational graphs, when the relationship comes from some source of truth, some external source. So for instance, thinking about payments, social networks, citation graphs, you know, hopefully, forward graphs. One thing that our group is interested in a lot is similarity graphs. Similarity graphs are graphs where you have edges that are built from some underlying data similarity or distance function. In that case, we start with, let's say, some collection of embeddings, then we try to find edges across data points. And, you know, it's good to reflect like why we're doing graphs at all, you know, traditional machine learning or large language models. First, they allow us to do computations like complex computations on abstract data and, you know, different kinds of data. So again, we have a graph machine learning model, we can apply it to very, very different sort of stuff. And we can do computation and merge diverse data types. So we could have different node types and edge types. We can kind of like merge different encoders and like merge different information together. That allows us to handle multi-dimensional data, which is important in industrial applications. And, you know, like just to kind of emphasize this even more, processor can be very, very best, you know, like we have, you know, for us it's your department in academia that are citation networks between people or papers, but like in real work we can model very complex systems. So for instance, this imaginary video sharing platform, you know, has some kind of advertisements going on on music videos and then people like can watch, can like, people can sell advertisements, they can be like, you know, skip or skip there can be very, very many different relationships covering objects of very different sizes, modalities, for each node and each set. And graph neural network team is a part of a broader graph mining group in Google Research. We'll have our VP coming here tomorrow to talk, I think, about broader picture stuff. We are often dealing with billions of nodes and trillions of edges. And this is considered large-scale data. So we can do research on combining algorithmic ideas with kind of like machine learning systems and models. And it powers kind of basically every Google product in some shape or form, including like Search, Ads, YouTube, and so on. And there we have some public information available. Blog post link here. And just in case, I'll be sharing the slides, like, links will be here if you're interested. And the Graph Neural Network team is kind of like— the idea is to make it easy to use structured data within networks. And then we do research, we develop a library to express this, we'll talk about this later. We do like some infrastructure work, and then we kind of do custom models. This. So the fun part begins, unsupervised learning in theory and practice. I want to start with kind of a notion of interior optimization first. I think interior optimization is one of the interesting things that we can do, many unsupervised problems, we can solve many unsupervised problems in interior optimization with graph neural networks. I use like a super broad definition that you probably would not agree with, like it includes clustering, assignment selection problems, and many more. And there are like many different applications that are interesting, for instance, in active learning, you know, like cleaning data, like even scalable clustering with side information. And then one common theme about these problems is that they're very hard to solve. You can like prove almost any of those to be like hard, and then like you go and try to solve something simpler, or you can try to attack these problems kind of like magical talent that we all got in the past couple of years and try to apply neural networks in particular to solve those. One example of a hard neural optimization problem is a representative selection. It's a very, you know, if we solve this, that would be amazing. That's a graph problem of basically selecting data points for the machine learning algorithm to optimize its downstream accuracy. It sounds too good to be true. And, you know, the solution is like harder than NP-hard under each hypothesis. And yeah, it's like super impossible to solve very hard. The fun part is that we can try to attempt to solve it with graph neural networks. And the idea is very simple. We try to learn embeddings and representative journeys or representative selection. Let me just explain the problem a bit more. We try to select examples, right? So we want to select kind of a diverse set of representative examples that would kind of span the space of our problem and maximize our classification accuracy. So we try to apply a very simple algorithm, try to optimize the coverage and the learning of representations. Learning is done by contrastive loss, the coverage, and the modification of K-means. And yeah, with some tricks to avoid trivial solutions. This is the illustrative example of how this works. You know, we have a graph that has some attributes depicted here in color. We plot a class attribute in it, and then we have some latent spaces that are selected that are simply things closer to the median and one, you know, and it's about, uh, square training. Um, one interesting part about this work, I think, is that if we don't have a graph, you know, we actually can create a graph. Contrary to, like, many intuitions, building a graph actually helps with this particular problem, verified in many different benchmarks against many different baselines. If we just create a scenario graph and do the same thing as I showed in the previous slide, we can actually achieve a lift in performance that is kind of unexpected for some people. So we lift the accuracy by a significant margin compared to non-end-to-end baselines. And yeah, the second finding is that constructive graph actually helps. This is kind of exciting for many different problems because, you know, we can build graphs and play our graph tools to actually improve on the accuracy of the algorithm systems. There are many, many different baselines there. Like, most of them are developed by theoreticians, so they have a bunch of nice modular algorithms that they analyze for this particular problem without any particular success in coming to the good solution. And then we tried all of them to see that they don't really work as well as we have this split. And these two belong to groups, and then this last one is just like clustering with GNNs. And that's what I'm going to talk next, because we have a method for GNN clustering, for clustering graphs with graph neural networks, which I think is also exciting because it provides provides one, it provides a nice view of graph pooling because, you know, like clustering is the fundamental graph building block. That's one building block that we all love. It allows us to solve very large graph problems. It is just a nice formulation. It's solvable. We have many different losses and so on and so forth. And, you know, pooling is just a weird thing. That we network people do. And, but pooling of graphs is actually like super, super close to clustering, which allows us to build many great intuitions about this problem. In particular, I love spectral clustering because it's like such a nice bridge between some really hard combinatorial optimization problem and something quite easy to solve with linear programs in scalable fashion. And the other nice thing about linear programs is that you can solve them with gradient descent. So that's what we're going to do. This is a definition of modularity. This is probably what you have in your CS224D, in any graph mining class, classic clustering objective that has this linear formulation. And this is kind of a— we notice this linear formulation is kind of a sum of sparse and low-rank matrices. So we can have like— we can pose it as a generalized eigenproblem and solve this in an efficient manner. And well, you know, like we have a hammer, we have a spectral skin, like this linear problem, let's try to hammer down the problem and optimize exactly the same loss. And then since this is a spectral problem and we expect the solution to be orthogonal, we can just use this as an expression of orthogonality constraint on our embeddings. The great thing is that it doesn't work. Because if it did work, then it wouldn't be a paper accepted anywhere. This is not accepted at GMLRS. Happy. It doesn't work because it's slightly too restrictive of a formulation, and then it can stop, and only this part gets optimized. And this is not what we want to optimize. We want to optimize the actual objective of the clustering. So we kind of make a much, much simpler, more relaxed regularization that just tries to preserve the cluster sizes to be kind of balanced. And this now works, right, in this magic. We optimize modularity, and this class regularization thing is just like— allows the problem to converge to the solution that we want. You know, there are some tricks. So yeah, you know, this clustering thing is kind of like— has a very sharp minimum for each particular solution. So we have to kind of like perturb it. So we have to use very high dropout rates. Right now, what I showed is kind of like has matrix A in it for optimization. So we can't just do mini-batch too easily. It's like sensitive to dropout rates and produces like kind of a soft clustering. But we get to threshold results. Yeah, we'll also carry with data sampling, for example, like— This is like a baseline quality feature. This is the baseline quality ground. Then merge them together, beautiful things emerge. And we can leverage the information that is present both and like use kind of cross-correlation to improve the results by like 40% or something on average across datasets. We have like a lot more experiments in the paper because we underwent a couple of revisions. What I want to talk about is not real data. Real data is boring. I want to talk to you about synthetic data actually, because that's one of the focuses we have in our group, that we discovered with this particular paper that we actually love to do synthetic evaluations. And, you know, we, for this particular experiment, we generated a bunch of stochastic block model graphs. And the nice thing about them is that we can vary different things about them. So for instance, vary scenario 1, this variance, strength of the signal in the graph. Scenario 2 is very similar features. You know, 3, 4, 5, or they both— sorry, 3 and 4 is like the variance. Scenario 5 is like the slope of the power law. Sorry, the average degree of synapses is like the slope of the power law. So we can vary all these different things and like very precisely test about what our method is sensitive to. Which, you know, when it works, you know, we can make beautiful charts, but we're on top of other methods. When it doesn't work, it allows us to debug the method and figure out when it doesn't work, what prohibits it from converging well, which is a really great tool because, you know, machine learning is really hard to debug. Graph machine learning, even worse. You know, like, I look at the graph, I look at the gradients, I can't really tell anything about this. When I have a bunch of small synthetic graphs where I can control some properties, I can actually understand what's going on. For instance, like one of the previous versions of the method, I was just— right? So it didn't work when I had like super high degrees in the graph. Okay, let me just try something else, let me control for this. So that's how it came up with this particular thing. So we were so excited about this that we actually wrote a separate paper in KDD about benchmarking on synthetic graphs. And we benchmarked a bunch of things. And the main point of the paper is about how all classical benchmarks are in kind of a very specific subspace, which has very strong signal in the graph and in the features. But this is not what we actually observe in many production machine learning systems. Sometimes people give us very poor graph structure. Maybe there is not much information there. Sometimes people give us absolutely garbage features. And our machine learning tools should be working with that. And we want models that can actually handle all the richness in the data. So we made this. There are many more beautiful illustrations in the paper. And we can study this kind of response curve to the different properties. Which I think is very exciting. So usually, like most of the time, unsupervised learning, including when you do it on graphs, is done through contrastive learning. So just to give you some idea about how this works, you have a graph, and you try to make a permuted version of the graph. In this particular case, I just pretend I shuffle the features. We contrast these examples, and then we use one of the contrastive or symmetric learning process on that. But the problem is that, you know, I live in the industrial world and when I have an unsupervised machine learning problem, I actually don't have labels to speak to, you know. In academia, it was very nice. I was developing an unsupervised learning method. I was looking the validation set of my labels and I was optimizing things. But now, you know, in my job I cannot actually do this anymore. But I still need to perform model selection. You know, we have to optimize hyperparameters. I need to understand if I'm giving, you know, like Google Ads, like an actual working model or not, you know, before they slap me. What am I doing? I need to debug things. So how do I do that without just very unsupervised, actually with snowlings? For that, we have to rely on kind of like weak signals. So there's a paper from last year that explores the idea of getting the signal whether the embeddings are good or not from the data itself, from the embeddings themselves. So the idea is that if we have absolutely random embeddings, these are probably not good embeddings. If we have absolutely identical embeddings, which are also not good embeddings. So we want something in the middle between these two horrible things. And we propose a couple of different metrics, one based on intrinsic dimension, another two based on condition number, which means that If we fit a linear regression to these embeddings, how stable it is numerically. Coherence is related to the analysis, like theoretical analysis of linear regression. Soft clustering is just about how well clustered points are in high-dimensional space. We can probably do more things, but I think it's really important to actually think about unsupervised problems in unsupervised space. Then again, Since I work in industry, I can call unsupervised programs, programs that are actually supervised. The distinction is actually kind of weird. People have been calling self-supervised learning unsupervised since probably Word2Vec is the first offender when it tries to predict the next word. Things are kind of supervised laws, but then you have this kind of signal that is just generated from the data, which I think is still, you know, kind of qualifies as unsupervised because we can still get loads and loads of signal from kind of free without labeling or doing anything human. And for that, one example that we released in the public is this like kind of large-scale graph regression problem to improve general processing units. These are kind of GPUs made by Google, uh, that solve, you know, that are like large-scale machine learning accelerators. Later, um, so we released this huge dataset of very large graphs. So there is like, um, and more than a million graphs, and then they have like many, many nodes. Contrary to the chemical graphs that people study, these are usually quite tiny, and it's like not a classification but a regression problem, which neural networks absolutely hate to solve. So I think that's kind of made it a very interesting benchmark, this model, quite basically. And then we ran a $50K Kaggle contest to have people, graduate students, beat us by a couple of tenths of a percent, which was kind of what was— that was the point of it. It's still kind of interesting. And yeah, we had a paper the day it was introduced to the public. So let me just explain, because I think it's a very good problem. You know, when you write something in your favorite machine learning framework, it gets compiled to some intermediate representation. And then this intermediate representation is translated to, let's say, like assembly for these beautiful chips. And I will focus on the central part. Central part gets the computation graph and then translates it into like translates it to the actual code using some heuristics. So for instance, when you have like, you write something to a dense, it has like a matrix inside of weights, right? But how is this matrix actually represented on device? Like whether it's row column, row major or column major order, it makes a big difference in performance, but it's actually decided by the fire. So like you have all this kind of like tiny flags that influence the performance of your overall program that you absolutely don't control, which is great because we can optimize them with machine learning and in particular with graph machine learning to make our accelerators even faster. And yeah, I'll talk about this kind of like accelerators and how we use them because I think that's also cool and it's kind of like a unique thing that we have at Google. Yeah, so, you know, before all this graph neural network jazz, we have graph embeddings that Joshua Elkington and Brian YU will be speaking about in a couple of hours. The embeddings kind of like are meant to capture the properties of the original graph. And, you know, it's just great to give them to people that do machine learning because it's like People these days, they only deal with embeddings. They absolutely can't deal with graphs. They don't know how to parse. If you do graph neural networks in your pipeline, it just becomes a graph neural network pipeline because that's the major majority of complexity. It's just dealing with all the different size graphs, the kind of visual shapes, stacking, and so on and so forth. You know, we come from a graph to an embedding. You know, we love graphs. We have many algorithms available for them at very large scales. But the problem is that it's hard to integrate the machine learning pipelines. So we transform them into embeddings. Each node gets its own point in a low-dimensional vector space, and then we can use these distances to do the link prediction, classification, everything that we know about. In graph mining, you can do a little bit of interaction in the embedding space. A standard embedding algorithm, Eric Seitel, 2014, people learn to predict context in random walks. If you have a random walk, it tries to predict the green node from the blue node, and then it can be represented in two distinct steps. First, you need to create a corpora of random walks, and then you train a skip-gram model that is in one-to-one word embeddings. Thankfully, Brian is not in the room. You know, like in a slightly transformed fashion, here's a formula for you. You know, you have a path, you know, whether path exists, we try to score 1, so we get the embedding to be close. If there is no path, embedding to be far away from each other across the whole corpus, and we try to optimize that. And, you know, there are a bunch of different machine learning pipeline architectures. You probably don't think about this at all, but we can do different styles of prediction algorithms. In particular, this DeepWalk algorithm is suitable for the batch processing. So In order to scale it to Google scale, we need to figure out how to do scalable random walk sampling and then how to do scalable training of the Bayes based on this random walk sampling, right? The problems with that is that first, sampling a graph cannot fit in a single machine oftentimes. If we have a couple of billion nodes, couple of billion edges, then it becomes hard even with like the machine that people have is this. And then the modeling, the problem is that usually the problem is that your A100 GPU has 80 gigs of RAM, right? And that can only fit, it can only fit, let's say, this can only fit a couple of million points. With these embeddings. So in this particular case, I'm not sure this configuration is actually correct. I think this should be like approximately for 600 million notes. So I think the factor of 4 is— I have that. Okay. So sorry, my confusion was that you can fit the embedding table on the GPU. It's just that you need to fit two embedding tables and the backward gradients for them. That's why there is or x-factor that I just calculated when I made this table, but I forgot about this. So, okay. You know, so for graph representation, graph mining team at Google really loves distributed computation thingy. You know, Google invented MapReduce 20 years back almost. It works well. Thanks, Jeff Dean. There are a couple of different approaches that other people take. Graph compression and graph partitioning for very large-scale graph mining. We don't love graph compression because, you know, if you have a factor X for graph compression, let's say you can compress a graph 10x, then the moment a person comes with an 11x graph to you, you cannot do it. And that happens more often than you can imagine. So whatever your graph compression algorithm, you know, the person comes with an even larger graph and then you can't do it. It's just hard. Partitioning is just very hard to get right. There are very different types of graphs and, you know, it takes like a graph partitioning expert to actually do something. And then you have to— every single person in our team should have been like a graph partitioning person, which was just not possible, right? And then graph partitioning is also very particular. So we have random walk sampling as our target. So it's actually very simple to do MapReduce style job or random sampling, you pass node IDs to their neighbors, and then you just have a join, distributed join with the distributed hash table that's very cheap, easy, and then you have the kind of samples and their adjacencies. Skip that. Yeah, that algorithm is very, very simple. And then what we get in the end is kind of this like kind of histogram. With different lengths that we traveled from each and every node and to each and every neighbor. And then we can plug this histogram back in our objective function and optimize that if we knew how to optimize that. So the accelerating training is done with tensor processing units, which are cool and are available, you know, both in Google Cloud and academia. Google Cloud gives insane amounts of enable people to use their TPUs basically for free for academics. You know, they can apply for grants, win them, do production, and then we're very happy. What is different between like a GPU and a TPU is that, you know, TPUs are organized in this amazing, like I just love that, amazing-looking racks of a couple of thousand chips per pod. And they have very like, like each particular GPU is kind of like small, smaller than a GPU, but they have a high amount of high bandwidth memory per chip. So, and that memory we can use, let me just have a number in here in particular, for like this latest generation, I thought has approximately It's like, it's multiplying, um, 9,000 chips, and then each chip has like around 100 gigabytes. So you can combine them all together with super fast like bandwidth, and you know, if you pay some money to Google, you will have a supercomputer, um, with like 200 terabytes of RAM with very fast communication. That's just insane. You know, we can shop so many graphs. So that's what we kind of did. We, you know, even a smaller, like, subset of the vault has insane amounts of RAM. So you can, like, optimize crazy large embedding tables relatively cheaply so far. When compared across different architectures, we compared with the beefiest GPU machine you can get, 6900s at a time. So no H100s were available, but they have exactly the same amount of RAM. And the competing system was Meta's PyTorch Micra. It produces suboptimal quality embeddings at decently high speed. There is an academic system, Lightning-E, that is a CPU-based system. Based on graph compression, so you cannot do really large graphs in there. The problem is that it produces very low quality results, which is great for speed but not great for the actual production. So we actually have to, you know, to serve Google customers and make everyone happy, we have to have good performance, like good quality in whatever we do. And this is a primary concern. Speed is like slightly secondary. Instant Embedding is one of my older papers. It does CPU embedding at slightly slower scale, again, on CPUs. And then Skew-CPUs are kind of like an old baseline CPU-based system that is based on spinning up large amounts of CPU workers and have them communicate between each other in kind of a raw, open fashion for gradient updates. And it's super slow, confuses 5,000 CPUs, but it achieves good quality. The TPU system though only uses 64 accelerators and then it produces high-quality results. I think I have a picture, the classic T-SNE picture to prove a point. You know, we are very— sorry, what's the interconnection? Yeah, so it's a proprietary system based on a very cool technology. It's optical instead of the classical InfiniBand style connectors. You can find it in the paper description. I think it was introduced in the fourth generation, and that's like 7 Gbps. Yeah, so I'll come to the last part of my talk, this short block for TF-GNN, our graph machine learning library in TensorFlow. So just kind of a recap, we like to think about graph neural networks as message passing operators, and then each node has kind of a state or embedding, and then we pass these embeddings across the edges We have a tiny neural network to transform the state into a message. Then messages are updated and aggregated into something, some new state for multiple steps. And then final node states are read out, and then you do your classification and prediction, clustering, whatever else on your final states. A cool thing about MTNs is that they really alleviate some complexities about having multiple different node and edge types. That means we can model like this very complex system, as I've been showing you, like videos, advertisers, you know, users passing messages to each other, different kind of tiny neural network processors, and then you can easily model very different sets of features. They can do different tasks, no classification, prediction, classification, and like contrasted with those style unsupervised very, very easily. And you know, like one of the blocks, one of the TF-GANs was used for this nice science paper for weather forecasting. And that's exactly like I would like to used on a very large scale as well. The same kind of style of thinking applies. You have a state, and then we try to predict the next state, and that's basically your weather forecast. I mean, the system was pretty much significantly better than, you know, possible physics simulation systems out there back at the time. The MPNs can be scaled to very, very large graphs through sampler sampling. We have a kind of blog post about how to do that specifically, but in general, it's nothing different than what I showed you with default style pipelines with random walk sampling. It's not that hard to transform. I mean, it's hard, it's engineering hard, but conceptually, it's not super hard to transform this random walk skip pipeline into the scalable sampling pipeline for very heterogeneous graphs. And that's the approach we again take to achieve scalability on very, very large size graphs without limiting ourselves to partitioning. Again, partitioning is just again hard when you have very complex graphs. Because you kind of don't know how to partition across different node benchmarks. Yeah, so we're very happy to release TensorFlow GNN 1.0. I didn't realize this is like a— yeah, it's— we're 29th of February. We technically released CFGNN like in February, so it was this month that we released the first stable version. So now we are promising to support you basically forever for graph neural networks. Yeah, there is this potential. But most of all, we're very excited to actually polish and simplify the library for production use cases. And we kind of COVID different architectural layers. So this is serious software engineering. There is this data layer, data exchange, modeling, and kind of orchestration layers for both of them. Reasonably low-code solutions to graph neural networks. And then we handle a bunch of nasty things that happen between graphs and machine learning accelerators, such as TVUs. But first, again, one thing that is pretty useful in industrial settings is some kind of explainability. If we classify anomalous, we probably want to know why we classified that. So we build experience for the first person system. That's the fun part for me. The, you know, TPUs are very annoying to program for because they only accept static shapes at every step of their execution. We handle that for you if you use TF-GNN. And we have like kind of a fun time machine learning -based optimizer for when you have a graph, you sample a graph, you shove these graphs into an accelerator, so we pack them optimally and then with minimal padding into a single batch of examples and learn the optimal padding on the fly from actual data, which I think is very fun. This allows us to train super fast on GPUs and Yeah, we have in-depth follow-ups to see how easy it is, but it's surprising to see I was able to use that. Yeah, so take-home messages. We can model very, very hard systems with Gen, with the graph neural networks and graphs. Embeddings are fun, and we can scale to very, very large graphs. And between Gen and we try to make your lives Yeah, we work best now on very hard graphs for accessibility. Thank you for attention. This is me. You can always reach me. These are links to the papers that I covered, all I talked about in this talk, and more good stuff. Thank you. Thanks, so if you guys have any questions, we should have one or two questions right now before the— What kind of approaches were the best on the GPGraph contest? Ah, on the GPGraph contest, it was like a bunch of prop-based feature engineering. That we didn't think about. I think that gave like a 40% boost from what we initially developed, which is kind of a nice lesson, right, for these things. Like on Quora, everything is preprocessed for you and you're supposed to only use the features that you're given. In real life, you know, you engineer some crazy feature from your data that actually gives you good performance, and that's what you're supposed to do. In terms of architectures, it was graph transformers. Surprise, surprise, once again. The graphs are large though, so it was a bit hard to adapt graph transformers directly, which is one fun part of the competition. So we hope actually to, you know, involves, you know, community more of these datasets about how to scale graph transformers to large address. I'm still not interested in that interaction. Maybe that will go on later, please. And how many comparisons can we think handling? I don't have numbers on top of my head. I'm not a hardware guy. Last I checked, it was like 4-5x faster. Just not the whole thing. Yeah. Yeah. So, and you know, you can only connect 16 cars with it. So you have kind of an upper bound in terms of like the amount of RAM your model can consume. Here we connected to the same— like, we— you can select a configurable slice of a TPU fault system. I think the last architecture is like 3D torus connected, basically, and connect like arbitrarily many chips, basically. So for this rough building experience, we connected 64 with super fast. So I guess in the interest of time, we'll— Thank you. Very good.

So, um, do we want to use the—