Ep. 26 - Rolling In The Deep (Research) with Daniel Jacobson | Developers:After Dark

Now, before this, what was the process that someone would need to take to in order to accomplish something like this without the connector?

Good question. I don't know. I don't even know how it would have been possible.

Hello and welcome to another episode of Developers After Dark. I am your host, Nick, with my co host Dennis. We have a repeat guest today which is going to be fantastic because we have some updates on some really, really cool stuff that's happening around the developmental world. But before we get all to that, let me pay a couple of bills and just say for those of you who are watching us for the first time or those of you who have heard of us or watched other episodes, of course, you know, you go to devs@darkpod.com that's going to be the place to watch not only this episode, but other episodes that we've done previously. 25 to be exact. But that's not the only place you can catch us. You can also go to the YouTube channel for the HubSpot developers. That's at HubSpot Developers. Now, don't just go there for us. There are some would say a plethora, some would say a multitude, some would say a bunch of other HubSpot employees, developers from the community, all with tons of content, a wide range, just a wide range of topics. You'll see people that have been on, you know, on this show, Brooke, you'll see Hannah, you'll see, of course, our guest who I'll let him introduce himself if you hadn't seen him the first time. But that's going to be the place to go for just about everything. You will learn new things, you will get updates, things, things that we've launched that we go over. It's just there's so much there. And please, when you're there, like subscribe and just check out everything because it's one of those places you can go and you can spend a lot of time there just learning a multitude of.

So, Nick, you sound like a YouTube influencer. Oh my God.

And subscribe, you know, send money to my Patreon. But no. And then of course too, if you really want to be in the know with everything that's happening in the development world, you're going to go to developers.HubSpot.com changelog and twice for Dennis, that is developers.HubSpot.com/changelog. Not only will you see what has been released as far as things happening in the development world, but upcoming things as well. So with that said, we have a really great episode As I mentioned previously, because we have a return guest, I like to call him the liaison of everything AI that kind of happened in the world. I live actually vicariously through him when it comes to learning about AI. But before we let him introduce himself, I did get to earlier this week talk with Daniel Jacobson here at HubSpot and we got to do a quick little what I like to call one on ones, messing around with a couple of different recording formats here. But it was really, really interesting because what we got to do is we got to talk about the CRM Deep research connector with HubSpot and OpenAI. And it was really, really fantastic because Daniel got to go in and he got to explain what it was, explain why it was beneficial for users. We got to have a really good talk. So we're going to cut, you're going to take a look at that and we'll catch you on the back end of it. Hello everyone out there, all of you Developers After Dark. We have a nice little special sit down with a very special guest and we're going to talk about the HubSpot Deep Research Connector. Now, our guest, and I'm going to let him introduce himself, is Daniel. He is a HubSpotter. So, Daniel, why don't you tell everyone who you are and what you do.

Sure. Hi everyone, my name's Daniel. I am the director of product at HubSpot and I lead our developer product group, the PM team. More specifically, helping to empower all of the developers building on the HubSpot platform.

Excellent. Now this little sit down is pretty special because we're going to talk about the Deep research connector from HubSpot. And knowing that this Developers After Dark is a little bit of a. It's a great name, but it's not, it doesn't exactly catch. Everyone that does listen to our podcast and watches us on YouTube. There are not just developers, but people who are developer adjacent. We talk about like people in SEO and marketing and just HubSpot users in general. And all of them, of course, belong to that family of over 250,000 businesses that rely on HubSpot. So with this, why don't you explain to everybody what this connector does, does so that way they'll know as part of that.

Yeah. So we just recently released a connector into ChatGPT more specifically for Deep Research use cases. And it basically gives the 250,000 businesses built on HubSpot deep research and analytical capabilities powered by a subset of their CRM data. So with this first release, we focused on companies, contacts Deals and tickets and we've proven out and read a bunch of prompts and generated a ton of reports and you can get some really interesting insights as a result of those queries.

Now that must have been a pretty awesome collaboration to have to work on or to be able to work on, because you figure, I believe the figure was over 75% of HubSpot users already use ChatGPT. So I made the analogy, I think in the last official episode we had that it was kind of like peanut butter and jelly, right? These two things will go together very nicely. But it's, it's, I mean, of course that's a very simple way of explaining it, but you look at it from like it and in some of the things that you'll hear is like it's doctorate level research, right? And you have your own customer data and you have those things and you're able, like you said that there's just these deep insights. This is, I mean, it's needless to say, this is pretty massive. I think one of the questions that would come up in my mind, you know, if I first saw this was, I mean, this, something like this must already exist, right?

No.

So in this case, we're the first CRM to integrate with, you know, ChatGPT for any sort of deep research capabilities. And it was really just inventing it as we went. In partnership with engineers from OpenAI, we kind of collaborated on the schema and the spec for how this could integrate with deep research to provide useful and meaningful results, gave feedback on the experience and made sure that it all worked end to end with a bunch of different tests and things like that. So, yeah, this is a first, especially in the CRM world. We're really excited about it. And the thing that I have been excited most about, or maybe delighted most about, is hearing stories, learning what prompts people are using, what surprises that they are experiencing as a result of having this new capability to do such deep research and analysis that previously was elusive or only reserved for huge corporations with massive resources to have teams to do some of these things. We're creating a lot more accessibility around all of this type of research.

Was there anything amongst that that surprised you the most? That one of those examples where you were like, oh, I can't believe someone's using this for this purpose. This is fantastic.

So we run some of our app partner programs on HubSpot and so I've had several, my peers on the product team at different teams at HubSpot basically trying to understand through tickets that come from our app partners. Just like what's good and what could be improved about their part of the product. And so we've had folks looking at how can we improve our UI extension capabilities for our developer platform, how can we make our marketplace experience better, all of those types of things. And so that's kind of cool. It's like, hey, as a pm, you don't think of like PM use cases quite as much. But yeah, there's, there's quite a lot of value in terms of understanding customer experience, customer pain points and doing almost product discovery and customer discovery work through some of these things.

That's amazing. Is there. So let's say, for example, let's say I am a. I'm a manager of a customer service team, right? Because we're talking about tickets and customer service. Would I be able to use this if I wanted to say, you know, maybe I sell XYZ widgets, right? And that's my company's business and I run the customer service team. I'm the manager and I kind of want to see what the. Let's say I'm coming into my busy season, right? Let's say that's in the Midwest. Here it's summertime, so it's, it's getting warm and people are a little bit more active and maybe they're going to buy more XYZ widgets. Maybe that's the time when they do it. Is there. Would this be a way. Are using this. Would this be a really great way for me to be able to like forecast my tickets or to get any type of deep research into the staffing needs that I might need?

Yeah.

So you could run a query to evaluate seasonal trends based off of historical records in terms of ticket volume and types of tickets you might expect to experience. 2. Do some forecasting on, you know, should you expect to see significantly more tickets filed? Does that also result in ticket resolution slowdowns? Like when you have an increased volume of tickets, does that mean responses are worse or slower resolution rate might go down, et cetera. You can basically generate a report that evaluates that from a historical trend point of view to help with forecasting and projection of. Do you need to scale up your support team, do you need to hire seasonal staff, et cetera? Yeah, totally.

Excellent. Now surely with something this powerful, I mean it can only be.

You can.

You have to have like a certain HubSpot tier, right? You can't, not everybody can't just have access to this.

In this case, it is not constrained to HubSpot tier. You do need a ChatGPT SKU that has deep research capabilities. Depending on the version of ChatGPT you're using, you basically get an allotment of deep research queries you're able to run each month. But yeah, it'll work with any HubSpot tier.

Excellent, excellent.

If you don't even have HubSpot, we actually have the ability for you to create a portal in the process of trying to connect HubSpot into ChatGPT. So if you're trying to invoke the connector, you can go ahead and create a portal through that flow and just get started with it. Although you'll want data in your CRM to be able to analyze, so you'll need to put in some data first, of course.

I mean, it seems like this does essentially solve one of probably the two, not the biggest, but the two things that stand out the most as far as problems. When it comes to, for example, you can have all the data in the world, but without being able to have some way to like analyze that data like intelligently, you end up with almost like data paralysis. Right. And then on the, on the flip side of things, you need really good data to be able to kind of pull out these sort of insights. So it really just goes to show like the amazing data you can have within HubSpot and the way you can actually access it and the way it all puts together. And then you have on top of the work you've done with this collaboration with deep research to make those deep insights. I feel like there are just an unlimited amount of problems for users that this can solve.

Yes, I mean, as long as it's constrained to, you know, contacts, companies, deals and tickets, use cases, yes, you can do quite a bit. Data fidelity and the quality of the data in your CRM plays a big role. One of the things that I like most about the experience is that it is connected back in jobspot. So, you know, when generating reports, it embeds deep link citations to the records that it is querying from HubSpot. So for example, you know, say you're trying to evaluate or generate a sales report and get recommendations to improve sales win percentage. So like, you want to, you want to close more deals and you want to win more deals. You can basically have it linked to lost deals directly into HubSpot with explanations as to like, maybe why it was lost and things like that.

Yep.

And so that way you can like see like, hey, is that actually pulling in the right data? Well, it is almost always pulling in the right Data, but you can see like back into the HubSpot CRM, you know, give me the entire history of that deal and give me the full record there and things like that.

Now, before this, what was the process that someone would need to take to in order to accomplish something like this without the connector?

Good question. I don't know. I don't even know how it would have been possible. I suppose you could have built a custom app, querying our CRM objects APIs and integrating it with some LLM deep research API behind the scenes, things like that. It would have been really tricky.

Exactly. I was setting you up there a little bit on that one, but good luck. Exactly right. No, I just got to say that is an amazing amount of work. I imagine this was, this wasn't just like something that you just spun up overnight. This definitely took some time.

It was probably faster than you might have thought. This was like a really quick opportunity that we seized and we were able to do so by building on an infrastructure platform that we have at HubSpot that makes it easy to spin up new services and attribute RPC calls to basically become MCP tools, things like that. Yeah, the entire product development lifecycle phase here was around four weeks from ideation to having functional end to end product and then a few more weeks on marketing, comms, reviews, all of that and getting it launched. So it was pretty quick.

That's amazing. That is absolutely amazing. Well, I thank you for doing that. I appreciate it because I guarantee you are solving quite a few problems for people. And not just solving problems, but you're giving them the insight because that stuff exists, right? It exists in the data. It's just a matter of trying to find the most adequate way to one, find it and then two, put it into a format in which you can create what I would just call actionable items. So this, this really does take. And I mean, and it's not in the examples that an example that I gave it was, you know, it was just essentially we're talking like customer service and customer service tickets, but there's plenty of other stuff and we actually, I'll throw up on the. If you want to take a look, you can always go to. Let me expand this. We can take a look at this here. There's not only we have company news for it, but then there's also an actual landing page for the overall stuff of that. And we'll put that up on the screen in a couple minutes here. But basically like the, not just the customer service tickets, but there are plenty of other kind of examples, like you have sales teams, you have marketing teams, you have customer success like you had mentioned before, that can use this tool and be able to just extrapolate not just meaningful data, but like making sense of it with the deep research. Was there any. I guess the question that I had for you is, was there any problems that you ran into while trying to do this that you kind of had to stop and rethink?

I don't know if we encountered any problems throughout the development life cycle, but there were some things that we really thought about at the forefront that we kind of knew were must haves. The primary one being that we wanted to enforce user level permissions on all requests so that only individuals who have access to read or access that data inside of their HubSpot CRM, that's the only data that they can generate reports on. Within ChatGPT, historically, many of the apps that are created are portal level apps. Typically they're like, you know, set up and installed by an admin and they're doing large scale, maybe data synchronization, use cases, things like that. But with the rise of these generative AI tools, a lot more HubSpot usage is at the individual and the user level. And so that was something that we really focused quite a bit on upfront. Most of the work was actually on just making sure that our APIs were up to the task, respected individual user level permissions could be exposed as, you know, available APIs to tools invoked by ChatGPT, things like that.

No, that, that's great because that's.

It's going to be.

That's a pretty huge part of it, I would say too, because it's just, it's not, I wouldn't call it open access, but you definitely want to make sure that the data that, you know what I mean, the data that you're pulling is actually data that you're supposed to be seeing as opposed to not so. But yeah, and we do have another. Put this up there for everybody to the landing page. If you go to HubSpot for those listening on audio HubSpot.com AI tools/OpenAI connector so I just want to say thank you because this is, and I was kind of teeing up a little bit for the. This has to have been done before it. This is the first of its kind. And then I got to say it's got to be really exciting to be working on stuff like this where you're kind of. It's groundbreaking, I would say. Right. I mean, I think that's a fair assessment. But, yeah, thank you for doing that and thank you for working on that and then coming on to be able to explain it because, yeah, again, this is. This solves a lot of problems and this is just like one of those things where it's like, it just makes perfect sense.

So, yeah, I mean, my path to this industry has been, you know, originally gadget obsession, and so I'm just like, you know, developer platforms, developer infrastructure is basically always at the forefront and the bleeding edge of what's possible with technology and with devices. And now, you know, with this whole AI revolution, it's now the forefront of what's possible with just software in addition to devices. And so getting to work on things like this is really cool and really fun from a personal satisfaction point of view.

Excellent. Well, thank you so much for spending time and doing this little sit down with me, Daniel. I appreciate it. And yeah, for those listening, we're probably going to jump back and you'll probably be listening to John talk about MCP Server. So again, I wanted to thank you for sitting down and talking and hopefully we would love to have you on the show in a future episode as well.

Yeah, we'd love to.

It's a lot to talk about for sure. So.

Sounds great.

Excellent. Thanks for coming on. I appreciate it.

Yeah, you bet.

All right, we are back. That was me talking with Daniel Jacobson and for everyone, that was. He is the director of product here at HubSpot and he got to work directly with OpenAI. So it was like one of those things where we had to grab him, we had to talk to him, because, I mean, just for me, I thought that was like, probably one of the coolest things and one of the coolest collaborations, right? And it was pretty instrumental to be able to have him on and just talk about it and just tell us what it was like to be able to do that.

So he's a smart freaking cookie.

Oh, yeah.

And big thanks to him for taking the time to do that too, because keep in mind, this is all going on right now and he decided to take, you know, a little bit of time to talk with me. And I was just like, who am I? Like, I just reached out him out of the blue and it's like, who's this guy, this Mohawk guy calling me? Wanted to talk about teeth research. Right? It's like, come on. But no, we really, really appreciate it because, like, that was. That was. That was really, really huge. But not the only thing that's really, really, really exciting. Really big, really Huge. Our guest, of course, repeat guest. I'm not going to say we had to bribe him to get back on here because we love him to death, but it is fantastic that he's joining us again. So let me let him introduce himself. John, take it away.

Yeah, longtime listener, second time caller. I'm John McLaren, I'm a staff developer advocate here at HubSpot and yeah, I worked with Daniel on launching this connector and getting our landing page and some of our resources out about it. And it's, it's been a very exciting and fast paced set of projects that are all interrelated. Yeah.

Is it, is it really not? Okay, I'll use the word nerve wracking, but maybe that's not the most apt description but like having to work on something like this in kind of an atmosphere where things are constantly changing in this category, like how was the pressure when it came to that?

That is a really good question. I would say the pressure to get something out rather than get it perfect was, was much higher. So what we know is basically LLMs are capable of kind of handling things if we need to change, say how the MCP server under the hood works. So related is we have our local MCP server that you all can download and use and we recommend using it through npx so you're always using the latest version and we have guidance on how to actually do that on our landing page. Developers.HubSpot.com MCP repeat that one more time. Radio video under the hood. The OpenAI connector is using model context protocol. And so the cool part is LLMs kind of make this a little easier to get something out there and then iterate on it and not worry as much about say like the integration layer with the other companies work. Because in this case it's OpenAI's LLMs going, okay, I understand how this MCP server works. I'm going to connect to it, use specific tools with HubSpot and then I will plan out all the steps I need to take to do whatever research that that user wanted and get them back a useful result. So if we change what those tools are under the hood, it's not really a big deal as long as the functionality is still there. So the tools could be different later, they could look different, but the actual functionality, as long as we maintain that the LMS can just kind of reason about it, which makes getting it done, getting it out there more important than having everything perfect because we can still perfect things after it's launched.

How long been working on this kind of abstraction layer. I know Daniel says something took four to six weeks to get this integration done with them. How long have been actually working on this? Because that just does not seem, I mean, to do this right. It seems like a very, very short window.

Yeah. So this is the fastest product like launch that I think I've ever been a part of at HubSpot. So week one of that four weeks or so was literally us planning out, like, should we do this? Is this the right approach? How do we approach like actually launching this? How do we move fast on this that we can get it out there, get it in people's hands right away? And then week two was the actual week that we started actual development. The bulk of the work on development happened in week two. We were already at an executable MCP server at week two. And week three was polish and testing and some internal like kind of like proof of concept stuff. And week four was all about how do we like get this in people's hands the most effective, efficient, educational way. So even things that normally would have taken a lot more time, like our landing page, we streamlined every piece of that whole process to get. Get things into people's hands as fast as possible. The video, the video that is on there. Thank goodness for Dennis here because I recorded that basically same day as Dennis said it. Like the, the. That's how insane things had to go. So rumor is Dennis is pretty good with video.

I just, I, I heard that. I don't. I mean I just said, yeah, thing.

Never expected that.

Yeah. You guys don't hear it? I'm constantly swearing and he's just editing it all.

It's just him going, oh, I gotta cut this again. I imagine it was probably. You said there was that, that that week or so that you were, you know, there was like not, not a debate, but it was like, you know, is this the right move to make? I. I imagine that the fact that probably. What is it? Around over 75% of HubSpot users are pretty much already using ChatGPT. That had, I mean that had have been a pretty big, like when you're, when you're putting the things on the scales of. That's got to be one of those ones that's like, yeah, I feel like this is going to be one.

Yeah. So one of the like influencing factors was like, okay, so yes, a lot of people are using ChatGPT, a lot of people are using Gemini, they're using all of these different large language model services. But one of the things that kind of Motivated us is we, we've seen with other companies like community created MCP servers, spinning up. Some did spin up for HubSpot, but we realized we were like, okay, well there's a bit of a security issue. Like we, we want to make sure that when people are connecting to HubSpot that they are connecting directly to HubSpot so that their authentication information goes to HubSpot. So that was a motivating factor. Obviously also the utility factor of like, hey, what can we do by just allowing LLM to like take actions inside of HubSpot account or read the data? So the OpenAI connector is a read only solution right now. The MCP server, however, is read and write into a few more types of objects than the OpenAI connector. And so what's really cool is like we popped that thing in the cursor, got it all set up and we're like, what would a developer want to do if they could just ask about the data in a HubSpot account? And we were like, well, one of the things that I've done is I want to build a serverless function and I need to know what the structure of that data is. What better way to figure out the structure of that data than to just go, hey, give me the structure of the data for a contact record or for whatever custom object I'm trying to work with. I can just literally copy and paste that dummy data or placeholder data to work with as I'm developing. And so it just unlocked a lot of different cool doors and a lot of them we haven't even seen yet. Like, I know there are some creative things that our developers are going to do with this and there's a lot of creative things that people are already doing with the OpenAI connector. Things that I wasn't expecting, like people trying to create different reports for how they're handling prospects or how they're like looking at their active deals. Like these are things that, yeah, in retrospect it's totally obvious, but some, some of these, some of the stuff with the MCP server stuff, it's not like we knew what people were necessarily going to do. It's a little bit like how our APIs are. Yeah. So, yeah, right. Like with HubSpot's APIs, we don't necessarily have all the use cases figured out yet. We release endeavor, we, we might have some vague ideas of how you might use something. But the coolest part to me, and this is, this is why I love this, this type of work is, and I just love Working with developers is because y' all figure out weird ways to do things that we didn't expect or design into HubSpot. So like I remember doing this myself before I joined HubSpot, like pushing HubSpot beyond its limits. And that's the coolest thing to me. So. Right.

Do you think that for something like this connector and OpenAI there'd be a. I know it doesn't exist yet, at least I don't think it does, but like an anonymized way of seeing how people are using the HubSpot connector, like in conjunction with other connectors, for example, like visualizing, you know, data points like with some other, I have no idea what, you know, figma or something like that. There's some sort of connector that can, you know, put it in there for you. It'd be really cool if like as I'm researching what I should build as an app to see an anonymous. This is the trends, how it's being used. Do you think that's something that could be done or would be done?

I think that's an interesting question. I don't know right now if or what exactly we're tracking. What I can say is we do know what APIs people are using so we can tell what tools get used. But that doesn't mean that we are seeing the LLMs side of it.

No, I don't think we can do that.

Right.

That'd be something that OpenAI would have to provide. I know you don't have.

I'm cool.

Quite aware that you do not work for them and you don't have an answer to this, but do you? I just feel like at some point in time, marketplaces and all that good stuff, there's got to be some sort of model for this, right?

Yeah. So what is interesting is in the model context protocol there is a property that the, that the LLM can provide to an MCP server that basically allows that MCP server to get feedback, give it some context to what the user was trying to do. It could end up just passing the literal prompt. I don't know that OpenAI is doing that. I did not check.

I do not throw this at you blind. I know that.

All good, but. So there is some possibility for us to get that data. But yes, you're right, it is definitely on OpenAI right now. If they want to publish that data of how people are using it on the LLM side. Yeah, yeah. Related to that, they cannot train on companies data that is using the CP server.

Do you know Nick I apologize again for taking over and asking all these dang questions, but that's who I am. Do you know when or if we will make it writeable?

I don't know. On the OpenAI side, I know that it is something that we are thinking about. I don't have a timeline right now. The priority was making sure it worked great with their deep research functionality. The the MCP server, however, that already supports write as we have the local version and coming in the future, we do have a remote version. That remote version is how our OpenAI connector is working.

Let's back up and say what does that mean?

Yeah, great question. The way that the local MCP server works is basically every time you want to have your AI tool make requests to HubSpot through what are called MCP tools, it means that ahead of that interaction you need to be running on your machine a local web server. Basically that goes, hey, I understand what the this LLM is asking me to do. I have this specific MCP format that it is providing me some information in. Then it goes, all right, I'm going to use the relevant tools script code to basically hit HubSpot APIs and then I'm going to take that data from HubSpot's APIs, bring it back, give it to the LLM so that the LLM can do something that's all running locally on your machine. When you run npx, that server is hosted on your machine temporarily. Then if you just used NPM install the same deal, the remote version is a whole different game. Instead of it being on your machine, it is hosted in HubSpot structure. So the positive is there's a lot lower barrier to entry. So you don't have to be super technical, you don't need to be a developer, you don't need to have node installed, which is one of the dependencies of the local XP server. So huge benefit. Another benefit that I'm excited about is this concept of user level ol. So one of the things that we wanted for the remote server, which is more for the general user, the developer focused local version. Right now the way that you interact with it is you need to create a private app that feels like it's a little tricky for a non developer to figure out. You got to deal with secrets and you have to edit a JSON file. That's not something we want regular users to have to deal with. But with the remote server we can actually change that. So instead what we can do is we can make it feel like installing an app that you'd find on the HubSpot Marketplace in a way is the permissions screen where it'll go. Do you accept giving this AI tool these following permissions and then as soon as you hit yes, install then cool. It works out of the box and when I said user level. So it will be scoped to the user's permissions, it will not have access to. So say even if there is an MCP tool for viewing contact records, the user doesn't have contact records access. Then the AI agent does not magically have that in the local MCP server version because it's based on private apps. It's based on what the scopes for that are. So this is just a way better way, a little bit more security and so we're real excited about it.

What if you don't want one of. I'm an admin on an account and I don't want a user to have access to this. Is there a way to negate that?

That.

So as far as the OpenAI connector. The OpenAI connector requires an admin, a super admin to basically be the first person to install. After that, then anyone on the account can use it, but before a super admin has done it, you just can't use it. I see. So. And as far as the remote MCP server, we will probably follow a similar paradigm to that. I will admit that's still being figured out right now.

How's it work? Let's say, man. Nick, stop me, dude.

I like your questions.

Honestly, how does it work if I am a user and my permissions has changed, how quickly are those refreshed through the tokens to ensure that I am not like I. I've transitioned from sales to service or something like that. When. How quickly will that be applicable to the new connector?

That's a great question. So that would be based. Based on the refresh token for the OAuth connection.

So it's a. It's a standard OAuth flow. Then the. Okay, I got you. That's pretty quick.

Okay. Okay.

The Excellent. Yeah. Okay. No, because I was wondering about that actually. That was. I mean. No, those are good questions because like one of the one things you don't want to do is to be able to have to try like based on your user permissions to be able to ask a question and then have all of this data dumped on you that you're like, I'm not sure I should be looking at this. This is. Okay, interesting. You know what I mean? Like you've always people, I'm sure in, in the development world have probably come across a situation like that once or twice where they're like, oh, I shouldn't be like, this is interesting, but I shouldn't be seeing this.

So yeah, I actually want to chime off that. So an example of like data that you'd be like, oh, I don't think I should be looking at. That is. So if you have sensitive data. So if you have properties hotspot that are marked as sensitive data, then those properties are not visible through the MCP server or through the OpenAI connector.

How are they represented? Are they just like X's?

They just don't get it. The properties just are not.

It's like this redacted sheet where everything's just like.

Childhood.

Sorry, sorry, I didn't know this was a counseling session.

What about John? What about regionality? Like, I know we have hubblets in the EU elsewhere. How does OpenAI honor that? Or does it or is that important?

That was a great question. So it's a little bit complicated because there's a lot of regions involved, but the landing page, HubSpot.com AI Tools OpenAI connector has the latest information. So in case what I say ends up outdated, look there. But the European Economic Area, a word I just learned, abbreviation I just learned. Switzerland and United Kingdom customers will need a ChatGPT team, Enterprise or education plan. And all other regions can use any paid ChatGPT plan, Enterprise Team Pro plus or education to access the connector and as far as, let's see, automatically be available to. Okay, so the Deepre Search Connector is also available to all HubSpot customers across all tiers who have a paid chat.

As long as you have a paid one. Yeah, right, right. That's fantastic. You know, one of the things that really struck me as fantastic because again, I'm one of those people too, where it's like being able to see what everybody's creating with this, whether that's reports or how someone's using it, that this isn't one of those things that's kind of in one particular area. Right. There's, I guess you could say there's many, many like go to market teams that can use this, use the connector, use the deep research. So it's not just one of those things where it's like, hey, yeah, if you're in, if you're in support, this is going to be great for you. It's everything. Right? It's save me, correct me if I'm wrong, but like, you've got marketers Sales support, anything that can basically contact deals. Right. Company, like utilizing those. Like, there is just an infinite amount of possibilities here as far as solutions for not just the developer. Because again, like the. The everyday user as well. Because, I mean, a lot of this is. I don't want to say it's specifically geared towards users, but like, don't get me wrong, developers also have accounts and developers are also service providers and these are things that they're telling their customers as well. But it really does kind of provide this sort of blanket. Maybe this is the wrong way to put it, but I look at it as like this blanket solution that covers so many different teams within one organization that, yeah, being able to see how each individual one comes up with a way to use this and a way to solve for whatever X would be in this case. It's just something that I'm. It's really exciting to see.

So. No, Nick, when. When you were talking to Daniel and you said, how did you do this before? And he said, you didn't. You know, I think that might have been like the most powerful things. Like, you know, I guess you could write an app possibly that would be able to go through all this data. But good luck and good luck making it not focused onto the most specific niche, you know, possible. Like to be able to generalize this and actually still have the depths. I don't know, man. We live in a really cool, amazing, weird world that's moving so fast and.

Not just at the speed it was done too. But then keep in mind, like, it was the first of its kind. Because that was like I. I was being a little, you know, facetious with my question, but it was like, yeah, you know, I'm sure. I'm sure there's a million others out there just like this. It's like, no, there's. It's like this is it. This is the first one in that record time with that much consideration put into it at once. It's like, I don't know, you just think back five years ago, maybe six, and that this was. Even that process was just unheard of, right? It didn't have to be specifically AI, but just that process in general with that speed is like, unheard of.

John, I know that this open AI thing was quite the catalyst to this first. We'll call this the mvp, I guess, right? Like the. The first version. I can't imagine iterating as quickly again for V2. I mean, is that. Is. Do you think we're gonna keep this clip going? I mean, it seems Impossible.

I, I, I don't want to spoil anything, but I would say that spoil.

Please spoil it all, man.

We have not slowed down at all. If anything, we've been energized by a response that people have had to both the MCP server and the OpenAI connector. And we are, we are emboldened to keep moving quickly and change how we work. Even just in general, the amount of AI stuff that our teams have to use in order to keep up with everything is insane. Where we're all having to work way more cleverly. I, I joke that. Well, my co workers within the developer product group have joked recently. They're like, I'm pretty sure that John had stunt doubles or, or his or his part AI at this point because I've had to get extremely clever because I'm, this is only one, this is only like one set of the big projects that we have coming. So I'm really excited for people to see the rest of what we have coming. And some of it obviously is really, it will be coming around inbound time, but we have longer term plans that are really exciting. And the funniest part to me is I'd say longer term, but every time we decide something is like, okay, here's the short term version of what we're going to do. Here's the long term version. Long term version ends up being like six months shorter than we expected because everything just started moving way faster.

I imagine you discuss short term in matter of seconds and long term in matter of minutes. Right. As opposed to, and correct me if I'm wrong, what you're saying is we could potentially right now just be talking to the AI bot version of John. You could be doing probably three other shows simultaneously. Is that what I'm getting right? Yeah.

Well, you know, you weren't supposed to find that out. That is wasn't, it wasn't in Total.

Recall where it was like under the eyelid, right. There's like a mark to show. Isn't that what it was? I can't remember what movie that was, but yeah, we're gonna have to start asking every time we see you in person now. Can you just lower your left eye.

Just a little bit? Real John has been on sabbatical for a long time.

He's on a beach somewhere just, you know, sipping a Mai Tai. He's not even here.

No. Just understand why else has my background not changed.

Right?

That's fair.

Hey, Nick.

Nick. When you read the interview, you said data paralysis. Is that something that you came up with? Or is that like something you heard elsewhere? Because I think it's such like a.

Is one of the.

When I was watching it, to prepare for this episode, because I do that, I wrote that down. So I thought that was a really, really, really important way of putting it.

You know, I will claim that, sure, but I guarantee you probably not. Because I've always. I've always heard it in the context of choice paralysis. So it's like. And I've always. I've always experienced it. I don't think I've ever called it that, but I always experienced it, like, from an SEO standpoint, when it was collecting metrics, like in the real estate world, when you just. Because, you know, like if you think back like 10 years ago, maybe even at that time, you know, people were collecting data because you had to. Now you did have to, but nobody was saying, well, what do we do with all this data now that we have it, right? Whether that's, you know, even if it was something simple like Google Analytics metrics that you were collecting, you still had to do something. That's why I think, like, with deep research and everything like this and this connector, it becomes so much more important because you go back to those days and you just remember collecting mounds and mounds of data, and you're trying to sift through it as a human being or even a team, and you're sitting there trying to say, like, okay, well, how do I make sense out of all this? This all looks very pretty. When you put it on a line graph, you look really, really intelligent, right? And everybody around you goes, but at the end of the day, it's like, where's the actionable items, right? You'll have a CEO or a CTO looking at you like, great, now that I spent all this money on all these tools and gathering all this metrics, what do I do with this and what's the next move? And if you're just kind of standing there just still showing bar graphs, like, smiling, you know, you look really, really stupid.

So it was just.

But yeah, it kind of came from that. Because I just remember looking at all that data in the past and being like, it'd be great if I could make heads or tails out of all of this. And then multiple tools too, as well. I mean, that's, again, that's no trade secret. The big thing about HubSpot that I love the most is you don't have to go to 50 different tools to do 50 different things. It's all pretty much right there. But when you start Adding these other layers of communication on top of each other, like this connector and other things. Yeah, that was always the big fear was that kind of paralysis that you would just run into, so. But things like this solve that, like, beautifully, you know? Yeah, that's where I came up with that. So give me credit for it if you want. I mean.

Well, I just did.

That was.

That was for you, buddy. I was, I was like my, My. My Father's Day treat to you to give you a little like, you know, you came up with something. You'll be known for years to come.

Yeah, I could. I can charge one if people want to use it. That's fine. Put it on a T shirt or something. But yeah, it's good. You know, I gotta make a living somehow. You. This ain't paying the bills. I'm just saying. But no, you know, John, I was going to ask you, is there, like, going along the lines of, like, stuff like data paralysis and like, problems you run into when you were doing this and you were kind of helping create all of this and, you know, working with, you know, multiple teams, not just within HubSpot, but outside of it? Were there any problems you ran into that were kind of. I don't want to say instrumental, because problems aren't really instrumental, but there are problems that you run into that are more. They kind of make you stop and like, you don't even have to rethink, but they kind of really change the way, you know, you were kind of moving in order to make sure it's done right. Did you run into any of those kind? Because there's, like, really big ones that I can. I don't know, I can think of with. With other things that not just HubSpot, but other people have done in the past with big projects or big collaborations where it's like, hey, we had to stop on a dime here because we wanted to make sure, you know, this came up, that we wanted to make sure that we prepared for everything. Did you have any kind of moments? And again, it was a very short period of time. So, I mean, but was there anything like that that kind of happened where you did stop and say, okay, that's something definitely to consider. Let's make sure we don't. That doesn't get lost in the weeds?

Yeah, so there's been a few different things. So as I mentioned, I'm working on a ton of different projects, and many of them you'll. You'll be learning about soon, but a lot of them are interrelated to each Other. And one thing that we did realize with the MCP server in testing, and this is not specific to any of the LLMs that we like advertised as being great ones to try it with. This is just general something that we, we had to change our mindset a little bit with was the way that LLMs work best is when they have the granular control to do whatever you want. And it like. So for example, we, we might be thinking along the lines of like a specific use case. Normally that is how, that is how software usually works. If you go, okay, this is the job to be done, that needs to be solved for, and then you work back from that going, okay, I need to make sure that there's an API, there's the UI that fits this with the LLM world. It doesn't work that way. You don't think in that way. You think the other way around. You go, what are the tiny little pieces that are needed to solve for a job to be done, but is not the job to be done is not the solution to the job. So an example of like, all right, if we want the LLM to be able to grab a ton of data from Hudspot, then we obviously need it to be able to grab like object records. We needed to be able to grab associations. But we can't be prescriptive in how we create the MCP tools. We need to give it as much flexibility as we can. And that's a weird way of thinking. It fits more how we think, like giving developers APIs, but it doesn't fit the like. Even with the APIs we often think about like, okay, here's some use cases that we build around. Instead, it's basically the user can enter any weird request they want. Like, I want only contacts that are named McLaren that were born in this date range and are associated to this weird deal. They can ask for that. And we need to give the LLM the tools that it needs in order to be able to answer that question efficiently.

Do we apply the same restrictions, like API limits, for example? Is that something that we are actively changing specifically for LLMs?

So we're definitely thinking around how we handle our rate limits in this new world. But right now the MCP server is completely using the normal rape limits that apply to if you were to create a private app, because it is literally using a private app. The OpenAI connector is a little special. It's a direct partnership between HubSpot and OpenAI. But yes, under the hood, there are still rate limits. And so that also impacts how we think about things. What you might notice if you were to dive into the code, the tools for getting objects and creating and updating objects. We're only using the batch endpoints. Even if you only have a state one record, that just helps ensure that it doesn't need to make 50,000 requests.

Yeah, but like if, I mean, as you just said, I want the McLarens who are associated to X, Y and Z. I mean, now we're starting to talk about some calls, right? And if there are a million, 2 million records that you have to, you know, parse through, would I, would I actually see an error in chat GPT if, you know, I hit that?

Yeah. So if you were to hit narrow, you would end up seeing a. Basically you would see the, in all the different tools, they kind of displayed a little bit differently, but you'll see that it tried to use the tool and then usually the LLM will go. It will just directly respond to you and be like, I didn't get something, I'm going to try something else to get that data. Or it'll be like, I hit the rate limit, so I'm going to wait and then I will retry.

What if it's, what if it's not solvable in that sense? What if they have to actually reconstruct? I guess you just said the first time. So like, if hit the rate limit the first time, it might try a different method of.

Yeah, it might try it multiple times, to be honest.

Can it catch like a, you know, a set and hold it in queue while it's trying to get the rest of it? Or does it have to be all in one shot?

So this will depend on the different AI tool. But largely what will happen is if there are, for example, if you needed to do two batch requests, I'm throwing out a number. I forgot the exact number that it's actually here. But say we had a limit of 100 contact records. I think it's a lot higher than that. I'm just throwing that one out there.

I think it's not much higher, actually.

I don't think. But yeah, okay, so say it was that. Then you will needed 200 records. Say that was what you were requesting, was 200 records. So it, no matter what, is going to have to make two.

That is a good point. Yeah, yeah.

And so what they do is they will store that in the context for the LLM Gotcha. That gets passed to that next LLM iteration.

Oh, wow. Wait, wait, wait, wait, wait. Iteration within that. So if I, if I Kept a convert, man. I'm sorry, Nick, again, if I kept that conversation, I came back to it two weeks later, would it still use what's. In that context?

Yeah, actually it should. Unless you directly gave it some like, hey, only use fresh information and get caught with the date change. Then it would probably make any requests. But otherwise, yeah, it'll use what it had stored in its contacts.

I think we should add to it, by the way, just post in the developers forum, you know, if there are issues. Is that the standard. Just click this button, post in the forums. We'll get to you someday.

Well, so actually that's a great, great idea. And we sort of already implemented it in a different. Really? Yeah. So you're using the local MCP server and I think our plan is to do this as well for the remote MCP server. I don't believe that it's implemented in the OpenAI one, but I could be wrong. So if you say that if you tell the LLM, Hey, I want to get some feedback on. On HubSpot's MCP server, then what it will do is it will actually give you a feedback form to provide direct feedback to HubSpot. And it's not. The feedback is not passed through the LLM. It is. It literally gives you a link. That way you can trust that the LLM is not like summarizing what you said in an incorrect way or anything. It is verbatim what you. Your opinion is so we can improve upon it. Cool. Yeah.

Nick, Shutting up. I know I said if we don't watch out, I'll. We'll make this go an hour and a half long. So I'm going to shut up and let you talk now and I will be quiet.

No, no, we did, we did, but, but no, that was, that was a really good question though, because I was actually curious about that myself because I would imagine it would treat it like a session and keep. And unless you opened up a new one or unless you told it to specifically disregard X, Y and Z or start fresh every time.

That.

Yeah, no, that's a very. I, I would wonder how long it would kind of keep it in. Like not in the buffer but like, you know, keep it.

Yeah.

In the queue, if you will, just to kind of make sure. But no, no, that, that makes perfect sense. Dennis always feels like he's talking order, but he's not. I'm like, literally like some of this stuff like I'm still learning on the fly as well, like is, you know, I'm, you know, at Full transparency. Like, I'm ecosystem quality. So I'm not in the developer advocate, so I'm not as close to some of this stuff, like on a daily basis. But, you know, sometimes selfish, selfishly, sometimes I get to literally learn on the job while we're doing this. And it's like, so some of these things, it's like, they're really good questions. And, you know, it helps at least me understand stuff better as well. Because, yeah, like I said, with things constantly changing like this, having somebody and, you know, again, Dennis will say I'm kissing up, but having someone like John on the team in the department, like, that's again, I wasn't even kidding. Living vicariously through you via AI is like, that's true. It's absolutely true. Because there's some stuff where I'm like, I'll hear about it or I'll listen to it and I'll be like, yeah, I'm sure John's already caught it. I'll ask him. I'll ask him later what this is because, you know, I'll read an article or something or something, and I'll just be like, yeah, I'm pretty sure I could read this whole article, but I'm pretty sure probably John knew about it like 18 minutes ago when it first, you know. But no, it does. The questions make perfect sense, Dennis. So don't ever think you're talking over me. Besides, it gives me a chance to not talk, and that's probably pretty good for everybody.

So shut it down.

Don't worry about that there. But I know we're coming up on time a little bit, and I did want to ask to you, John, specifically, is there with everything we've talked about thus far and then like, knowing that things can change, you know, in an instant, you know, five minutes after we hit end, are there things right now that you could speak to as far as, like. Because I know we had asked. I think we had asked Vikram from Prompt Armor with regards to OpenAI as well. Like, what are people. What are. Like, what is the one thing or one or two things people right now are missing as far as. Because again, there's a lot of information out there, right? There's a lot of iterations on things. And with things constantly changing, are there things out there right now that just people may have an idea of, but they're Especially with regards to, like using an MCP server or, you know, like, how it interacts with everything, Are there things that people should know right now that they might not, you know, that are just kind of really important.

Yeah. Oh, that is a great question. So I touched on this a little bit earlier, but what I would say is so the entire idea of model context protocol is kind of like a nebulous too sciency term for most people and doesn't mean much. And the actual MCP tools bit and versus like the MCP client versus MCP server versus host, like all these words. The. The thing that I hope that most people understand is that the MCP is the connection between your AI tool and the SaaS product that you want to work with. Or it could even be tools that run on your computer could use mcp. But what I do want people to definitely know is like, hey, yes, the open source community is freaking amazing and has like provided these open source MCP servers for a lot of different cool tools. I just want to encourage a little bit of caution. Like remember, if you are connecting your HubSpot account, you are connecting your HubSpot data to whatever tool you're you're connecting and that middleman, that is that server. So you need to make sure you trust that server and you trust that AI tool. Otherwise yeah, that's that. It's you're just adding more, more companies, organizations and individuals into that web of trust that you have for your security for your company.

So you're saying, John, it's totally fine to connect to. Dennis's not a spam bot MCP server.

I swear it's me. Dennis is the name of it.

Only trustworthy Dennis over here totally not going to change your password to password exclamation point.

Well, and final question, which for both of you, which I think is the most important, when you guys talk way too fast, do you also say chat gtp?

I have done it Acronyms. Like I thought us web developers had too many acronyms before. And then I also learned like, so my wife is a teacher. They have a million acronyms and they even said an acronym to make fun of the fact that they have too many acronyms. They call them PLAS, three letter acronyms. And so yeah, I think in this new AI world where we are inventing A to A MCP GPTs, like there's just too many, too many acronyms and we're all gonna mispronounce them like Jeff or gif and we're just gonna have to deal with it.

I try to sound so smart, but I butcher it every time. Every time. I do. I know I do. I caught myself doing it.

Yeah, we're just gonna have to get AI to retroactively fix our pronunciation and videos.

I think that we call that Dennis right now. But like, yeah, it's. Well, I wanted to say thank you, John, for coming back and talking to us again. We didn't scare you away, which is fantastic. I mean, you know, always. But yeah, I mean, I think it's really, really important for people to be able to. Because again, not, not everybody is as in depth or in the weeds like our teams are and at least people at HubSpot are. So they're, they're. Not that they're getting this information secondhand, but it's not coming as free flowing as it does if you were sick in the walls HubSpot. So that, that makes things a little bit more difficult. And I do really appreciate again, like Daniel and yourself and others who worked on this because I mean there were, there were multiple teams, you know, across the whole organization that made this possible. So without them it's, you know, we're kind of parroting a lot of what you're saying and what a lot of Daniel and everyone else is saying too. But without that work, it's like, yeah, it really is fantastic because it's a nice way for people to be able to stay on top of things because you can get, I'll be honest, you can get newsletters, you can get other stuff and you get to. But again, it's like, it's like reading an instruction manual on a thing you've never used before. Yeah, it's going to take some time.

So reading, reading the documentation on MCP alone, I was like. So I get, is you have to start getting like the, the key is you have to get clever about how you digest the information.

The key is have an abstraction layer like John McLaren on your side.

Clone ID is great then because this is really going to help. This is, this really solves the problem. Everybody can learn MCP and everybody can learn everything they need to know as long as they have a. Basically an AI John next to them. It's, it's true. Yep, absolutely.

Yeah. Did I mention I'm going on sabbatical? Good luck. You're gonna have to figure out the.

Clones came just in time.

Yeah, right.

Well, again, we want to thank you for coming on. We really do appreciate it. We love these conversations. And I will leave you with this little. Dennis loves these because, you know, they're very insightful. I kind of look at them as like the Jack handys of the SNLs, like back in the, what was it, like 10 years ago or so.

Yes, I Love it.

But. But no, you know, Chat GPT does support about 60 languages so far, but the one language it'll never master is the language of love.

Oh, oh, bye.

Did you like that work?

That's is when they have the granular control to do whatever you want and it like. So, for example, normally that is how software usually works. If you go, okay, this is the job to be done that needs to be solved for, and then you work back from that going, okay, I need to make sure that there's like an API, there's the UI that fits this with the LLM world. It doesn't work that way. You don't think in that way. You think the other way around. You go, what are the tiny little peacocks that are needed to solve for a job to be done, but is not the job to be done? Like is not the solution to the job to be done. So an example of like, all right, if we want the LLM to be able to grab a ton of data from Hudspot, then we obviously needed to be able to grab like object records. We needed to be able to grab association. But we can't be prescriptive in how we create the MCP tools. We need to give it as much flexibility as we can. And that's a weird way of thinking. It fits more how we think of like giving developers APIs, but it doesn't fit the like. Even with the APIs we often think about like, okay, here's some use cases that we build around instead. It's basically the user can enter any weird request they want. Like, I want only contacts that are named McLaren that were born in this date range and are associated to this weird deal, they can ask for that. And we need to give the LLM the tools that it needs in order to be able to answer that question efficiently.

Do we apply the same restrictions like API limits, for example? Is that something that we are actively changing specifically for LLMs?

So we're definitely thinking around how we handle our rate limits in this new world. But right now the MCP server is completely using the. The normal rape limits that apply to if you were to create a private app, because it is literally using a private app. The OpenAI connector is a little special, the direct partnership between HubSpot and OpenAI. But yes, under the hood there are still rate limits. And so that also impacts how we think about things. So what you might notice if you were to dive into the code, the tools for getting objects and creating and updating objects, we're only using the Batch endpoints, even if you only have it record. And that just helps ensure that it doesn't need to make 50,000 requests.

Yeah, but like, if. I mean, as you said, I want the McLarens who are associated to X, Y and Z. I mean, now we're starting to talk about some calls, right? And if there are a million, 2 million records that you have to, you know, parse through, would I actually see an error in ChatGPT if, you know, I hit that?

Yeah. So if you were to hit narrow, you would end up seeing a. Basically you would see the. In all the different tools, they kind of displayed a little bit differently, but you'll see that it tried to use the tool and then usually the LLM will go. It will just directly respond to you and be like, I didn't get something, I'm going to try something else to get that data. Or it'll be like, I hit the rate limit, so I'm going to wait. Gotcha. And then I will retry.

What if it's not solvable in that sense? What if they have to actually reconstruct? I guess you just said the first time. So like, if you hit the rate limit the first time, it might try a different method of.

Yeah, it might try it multiple times, to be honest.

Can it catch like, you know, a set and hold it in queue while it's trying to get the rest of it? Or does it have to be all in one shot?

So this will depend on the different AI tool, but largely what will happen is if there are. For example, if you needed to do two batch requests, I'm throwing out a number. I forgot the exact number that it's actually here. But say we had a limit of 100 contact records. I think it's a lot higher than that. I'm just throwing that one up.

I think that's not much higher, actually. I don't think.

But yeah, Cool. All right, so say it was that then you will needed 200 records. Say that was what you were requesting was 200 records. So it, no matter what is going to have to make to.

That is a good point. Yeah, yeah.

And so what they do is they will store that in the context for the LLM.

Gotcha.

That gets passed through that next LLM iteration.

Oh, wow. Wait, wait, wait, wait, wait. Iteration within that. So if I, if I kept a convert, man, I'm sorry, Nick, again, I kept that conversation. I came back to it two weeks later. Would it still use what's in that context?

Yeah, actually it should. Unless you directly gave it some like, hey, only use fresh information, then it would probably make any requests. But otherwise, yeah, it'll use what it had stored in its contacts.

I think we should add to it. By the way, just post in the developers forum if there are issues. Is that the standard? Just click this button, post in the forums. We'll get to you someday.

Well, so actually that's a great idea, and we sort of already implemented it in a different. Really? Yeah. So you're using the local MCP server, and I think our plan is to do this as well for the remote MCP server. I don't believe that it's implemented in the OpenAI one, but I could be wrong. So if you say that, if you tell the LLM hey, I want to get some feedback on HubSpot's MCP server, then what it will do is.