Your Brain's Data: What AI Can Actually See

San Francisco has turned into an AI cult. And this is the cult's altar. It's a homemade GPU cluster, by which I mean it's a metal container that's been filled with graphics chips. Historically, such chips have been used to run video games. But in this city, at this moment, the graphics chips are being used to run artificial intelligence software. This altar is fueling a neural network. Most GPU clusters are in large, fancy data centers. But in this city, at this moment, you can find them in basements, next to a water heater, and a few feet from a makeshift gym.

So we're gonna have you come a bit closer.

And you can also apparently find a laboratory and a dude named Ben with a weird contraption on his head.

What does it feel like, Ben? Um, it's kind of— it's fine.—

and another dude named Jonathan trying to read Ben's thoughts. Let me explain.

We're trying to figure out what they want before we actually work with them.

Jonathan Xu is the co-founder and CEO of AllJoint. It's an 18-month-old company backed by the likes of Khosla Ventures and AI legend Jeff Dean. AllJoint wants to measure the electrical signals that flow through our brains and then use AI software to analyze and decode those signals.

Or, as Jonathan puts it: We're a deep tech company trying to decode thoughts, emotions, images, all this high-level semantic stuff from EEG. Now we have, you know, the world's largest dataset for EEG and image and—

You already have the world's largest dataset?

Yeah.

EEG technology that measures the electrical patterns of brain activity has been around for decades, but it's been of somewhat limited use. You simply can't get very clear reads on what neurons are doing when you're measuring from outside the skull.

Our whole intuition is to take transformers and large datasets and decode all the stuff that we thought was not possible from EEG for the first time.

Jonathan's big idea is to use AI to make up for the deficiencies of EEG technology. He's convinced that AI has gotten good enough to find patterns in this data that researchers have traditionally missed.

We've known how to do this for decades, so that part is not new science.

Historically, the way that we have looked at such data is either by eyeballing the raw time series data, frequency analysis, really like simple ways of doing that. So thanks to the advent of machine learning, we're now able to leverage AI to decode much more impressive stuff. Please come in, come in.

His hope is that improvements in AI may soon make it possible to read people's thoughts. To make all this work, AllJoint needs willing participants to train its AI models, which brings us back to Ben.

The study that we're doing today is focused on images, and so it's your brain response to visual stimuli. We're just gonna put this on your head and we'll use some conductive gel to ensure contact between the electrodes and your scalp. And that gel is also completely safe. So yeah, I'm just gonna put this on your head right now.

Do you believe him? That's absurd.

People get paid a modest fee to go all A Clockwork Orange and stare at images for hours.

Basically, we're going to present to you a series of images, and you're going to be tasked with answering if you see a picture of the toy Woody from Toy Story.

A picture of what?

Woody. It's the character from Toy Story.

Oh, okay.

I'm just making sure that it's centered and that we cover the occipital lobe well, so.

Ben, how did you even hear about this? You saw a thing for brain test.

Yeah, I came across it and I was like, "Eh, it's in the city, I live in the city, not too far from my house." Help science and make a little extra money.

Exactly, society house, exactly. Yeah, yeah. So yeah, we're just gonna now apply gel. To the electrodes.

Mm, brain lube.

This cracks me up.

So I'm going to put gel here on the reference electrodes first. Each letter and color corresponds to one on the 10-20 electrode placement map. And so we're focusing right now on the occipital lobe, the visual cortex, because we're doing image reconstruction. I was thinking maybe it might be cool to have this gel be like Actually good for your scalp, for like, you know, you leave it on like a conditioner or something. Cool. And I'll go into the next step.

The pictures go by one after another, and the EEG device collects data on what the person's brain is doing. Sooner or later, the AI software starts to learn what brain patterns correspond to what images.

How much does the helmet cost?

I think $2.2 grand.

And who makes it?

Emotiv.

Anyone could buy one of those?

Yeah, you can just go on the website.

Yeah.

This is our local cluster. It has 7 3090s.

Those are NVIDIA chips?

NVIDIA, yeah. You can just game on them if you wanted.

I just think this is funny. I think it's such a sign of this moment that you have young kids in these hacker houses, startup houses with their own, you know, neural nets just running in the basement. The other side of this is like somebody's Bedroom or like a—

Yeah, that's a bedroom for now.

It's like hardcore startup house.

I love it.

Yeah.

We'll leave Ben to find his Woody.

You've got like the work area, you've got the lab right downstairs. And so this was by design.

Yes. We need like shower facilities for washing gel off, lab space to collect data, somewhere to put our cluster, a coworking space. And most of our employees are also new to the area and will I actually prefer living in a house like this where the commute time is like 1 minute. It's pretty efficient and I really like it.

There are tons of brain interface companies now, including Neuralink, Synchron, and others.

Like the big picture, what would be the end results that would actually make people's lives better?

Yeah, so we want to basically bridge the gap between like your thoughts and the outside world.

So the participants of the study are helping create this database of maps Matching electrical activity to the image. And then the goal is to push the science further where you're able to pluck important things out of somebody's mind, maybe, right?

Yeah. So if we do this and we're successful, right, we're able to basically encode all aspects of your consciousness into like an embedding and basically go from uninterpretable time series brain data into this embedding that is able to describe your conscious state, that unlocks a whole wealth of, like, opportunities.

Researchers have tended to deride the idea of an EEG device being able to do anything close to reading thoughts. But Jonathan and his team think they're already seeing early signs of this technology being able to do new, unexpected things. One big breakthrough has come just from measuring the brain activity of more patients.

We've demonstrated by doubling the training set size to 8 sessions, we get around, like, 75% of the way to lab-grade hardware. So we're working with 2,000 $100,000 hardware and then we're getting 75% of the reconstruction accuracy as like $100K hardware. So now we're just gonna push that more and see when do we meet it and when do we surpass it? What does the exponential curve look like?

Last year, Jonathan was a member of a team that published a major paper on this mind reading approach.

The question that we were trying to answer is, given your brain data, can we figure out what you're seeing?

You're showing a ball, a girl, clothes, chair, and then the brain is kicking off this electrical activity and you're trying to hone in on this pattern matches table, this pattern matches ball.

Yeah, so that's how you train a model, and the model's goal is to take the EEG and predict the image. And then at the end of the day, when the model is trained, you can give it new EEG that it's never seen before, and it's gonna try to generate an image.

The future of where all this technology might go is both awesome and creepy.

There's obviously a ton of potentially beneficial uses for this kind of technology. On the other hand, it gets weird fast, right? 'Cause it's like, oh, you've got a machine that can tell if I'm lying or telling the truth. Do I wanna live in a world where everybody sort of knows all that stuff?

There are very important ethical and moral questions that we care a lot about. We would hate to see this used in interrogation or like surveillance, right? Like you want to augment the autonomy of an individual and improve their lives. There's like some cool things that we're trying soon with emotion decoding. Like let's say we can decode 20 different emotions from you. Just plug that in to like ChatGPT and now you have an emotional token that you can augment it. So you can have the AI know how you're feeling and respond to that. This could be very useful in healthcare domains with personal therapy. Like, you have a lot of emotions that you yourself do not understand. And what if you could map that objectively and have either an AI therapist or a real therapist walk you through your feelings and diagnose and understand that?

If you had emotional token on your 2025 bingo card, please take a bow. It's early days for AllJoint, and the startup is not alone in its quest to try and make new use of this EEG-AI combo. Most of the companies trying to do this work envision a day when people wear a small device on their heads that translates their thoughts into actions. You could, for example, navigate around a computer or smartphone via thoughts alone. And obviously, Jonathan is hoping for the direct transfer of data from person to person. Are you alarmed? Are you excited? The answer should be yes. Never change, San Francisco. Never change.