Ep 42 - App Certification with the Ecosystem Quality Team | Developers:After Dark

Hello and welcome to Developers After Dark. It is the show that makes sure your happiness is horizontally scalable. So if you have to add more servers, don't worry about it, it'll be fine. But today we have both what I would call a dual purpose and a triple threat. Now, what does that mean? Number one, that means we're gonna be talking about AppCertification and the AppCertification program. Number two, at the time of this recording, it's Women's History Month. So we figured who better to have on the show and host the show than the very people that are absolutely vital and a huge part of both AppCertification and the program itself. So as a little added bonus too to our listeners is you don't have to listen to me talk as much. And an added bonus to Dennis and AJ who are listening, who probably don't wanna hear me talk at all, but Before we get started, let me ask you if you could, from the kindness of your heart, like and subscribe to all of our videos. Subscribe to the Developers YouTube channel, which on HubSpot, you're not just going to find us, but you're going to find many, many other videos from both HubSpotters or third-party platform providers, not to mention app providers, asset providers. There is a wealth of knowledge on the Developers YouTube channel that you could get lost for weeks.

Weeks.

So, but without further ado, you don't want to— you're not here to listen to me talk. What we're going to have is we're going to have our very own Jonna, who's been on the show before. She is going to take over the show and she is going to host, ask the questions. So without further ado, don't listen to me, listen to Jonna and let her and everyone else introduce themselves and let's get started.

Thank you, Nick. And hi everybody, I'm Jonna. I am on HubSpot's ecosystem quality team with these lovely people around me, and we'll see if it's a bonus by the end of the episode that you have to listen to me and instead of Nick, but I think we're going to have some fun. I've been at HubSpot for, I think, 7 years by now.

And yeah, I'm so excited.

I'm so honored to host. I'm with these amazing women here. So I'll let Britt and Marina go ahead and introduce yourselves.

All right. I'll go first. Hi everyone. I'm Marina Lupin. Thank you for having me on the Developers After Dark podcast. I'm also honored to be here with my fellow teammates. So a little bit about me. I am a senior technical program manager on the ecosystem quality team here at HubSpot. I've been at HubSpot for a little over 5 years and actually all of them on this team. So I feel really lucky to have spent my entire HubSpot career on this team and work alongside Jonna, Britt, Nick here, everyone on the ecosystem quality team and all of DevRel. So a little fun fact, I actually joined as an external hire. Into what was then called certification team with certification in my job title. You can verify that by seeing my, looking at my LinkedIn. I'm excited to be here today to talk about what app certification really means in practice and how we think about it at HubSpot.

Hey everybody, my name is Brittany. Happy Women's Month and official first day of spring, I'm pretty sure, I think. Fun!

Yes!

But a little about me, I've been at HubSpot for actually a little bit over 8 years. 4 and a half of those years spent on the best team, which is the STEAM ecosystem quality. I am a senior technical program manager, and when I started, the team was also still called certification team when I joined. However, I actually worked more closely on quality programs for themes and modules with Nick during the time of the template marketplace relaunch, but then over time transitioned into focusing to, you know, apps and certification, which brings us to where we are now.

Yay.

Cool.

So excited to be here with you both and talk certification. Marina, literally the certification original OG team. So I think this is going to be fun. Let's dive right in. So for someone new to the HubSpot ecosystem, how would you explain what app certification is and why it matters to customers and to partners?

At the end of the day, that quality leads to trust. So at a high level, like HubSpot app certification is the, or our official signal of quality for integrations in the marketplace. It means like the partner has gone above and beyond to meet HubSpot standards across all of those areas. But more than that, it's also a signal of commitment. So it's not just that you're building an integration once, but that you're actively committed to maintaining and improving it as our platform evolves. And for, for customers, that's incredibly important because it helps them, you know, quickly identify apps that they can trust and adopt with confidence. And for partners, I think it's a key milestone in their entire, like, growth journey. Increases visibility in the marketplace. It builds credibility with HubSpot teams. It, it's also now closely tied to the Technology Partner Program, um, playing a role in tier eligibility, access to benefits, and a deeper co-investment and opportunities with HubSpot. So for example, like to become and maintain a Premier, a Premier Partnership, um, you're required to have a certified app. So yeah, at the end of the day, I guess it comes back to trust on both sides and helping customers, you know, make better decisions and helping partners grow.

I think one of the biggest things is you don't want to have something where you don't know if it's going to be upkept. I think in any marketplace anywhere around the world, that's probably one of the biggest fears for a customer, right, is to, to sit there and look at something and say, I'm going to invest in this, whatever that may be, whether it's a theme, a module, or an app, and not be sure if come tomorrow, you're going to reach out for some sort of help or some sort of thing and no one's ever there. So, I think that's pretty, pretty important.

Yeah. Fully agree. No, I actually don't. I think, I think Britt and Nick both, both nailed the answer. It's, it's really, it is very much the, the signal. It's, it's the top quality signal of, of those apps coming into the marketplace and Again, as Britt mentioned, it's, it's the commitment to continue to meet those higher quality standards.

I've actually never thought of it like that before as the commitment. So I'm really glad that you guys kind of phrased it in that way because that is, oh, what a great, what a great positioning.

Yeah.

So how do you both balance being a gatekeeper for quality while also being a champion for partners who are still maturing their products? So sort of like Oof, we need you to meet these bars, but also we want you to have success, we want you to grow. How do you balance that? Marina, why don't we start with you?

Yeah, and I actually have an example as well. So, I would say the way I see it, it's I consider being a gatekeeper for quality and the champion for partnerships almost like two sides of the same coin, but they're also competing. And honestly, over the past few years, We've spent probably many hours during meetings actually discussing what that means, especially when it comes to certification. So I would say as a, as a gatekeeper for quality, the main goal is really to set the app up, to set up the app for success. And this means, you know, whether it's listing or going through certification or recertification or any other programs that we run. And that means having proper documentation, high API success rates. Following just general best practices by minimizing the scopes that are being requested and really helping at times find— helping developers find alternative solutions, especially if maybe they don't meet those requirements, maybe they're not yet ready to be listed in the marketplace or have that feature. So finding alternative ways that they can be successful. I also want to highlight specifically what I mean success here, and that is not adoption. We have the partnerships team that can help with that. But I think when we talk about certification and being a gatekeeper, it's really evaluating the functionality and how the app is built and really allowing the developers build the solutions that they want to build and deliver to the customers. The one actually quick example I do want to share as far as also being a champion, So a few, a few years back, we had a certification requirement where we asked developers to not use any APIs that were in beta. So whether these were public beta or private beta APIs, they could not be used in an app that was going through certification. I was reviewing an app. I think it was maybe one, one year or so into the job and I received a lot of pushback from this one particular partner during the review process, which led to many phone calls, many meetings with the developer and internally as well, which actually led to discovering that this particular endpoint has been in public beta for maybe over 3 years, but it wasn't being actively developed. So there weren't necessarily being major changes to this endpoint. And our biggest concern, the reason that requirement was there initially, was if an endpoint is in beta, then there are probably going to be major development changes. And we're saying certification is that high quality bar and something breaks, it is going to negatively impact the customer. And actually, after having those meetings with the developer, we understood that we probably need to kind of relaxed that requirement a little bit, and we've since dropped it.

Got it. And this like sounds like, oh, let's not put the onus of our product development cycle on our, our developers that want to reach that high bar. No, that's interesting. And we want them more to use our new tools too. That makes perfect sense.

Exactly. Exactly. And I will say, if you're the developer listening to this, I think you know exactly who you are. So thank you. Thank you for, honestly, it's I'm glad that many developers don't really necessarily take our requirements as a binary, like yes or no. We want to have these conversations and discussions to understand how we can make them successful and whether it's time to evolve our program as well.

Yeah, and kind of going into that a little bit more, a lot of what you talked about, the conversations and you're working with them, you're finding ways to optimize, finding ways to build better. Like, that's pretty involved. How do you, how, how do you manage that type of, you know, hand, almost handholding, almost white glove service as you're working with these sort of, you know, people going through the certification process? Is it challenging to, to manage that, or do you feel like, you know, once you've got the motions down, you're feeling good?

I would say it can be mixed. It can be sometimes challenging because, again, the marketplace and the tools that the developers are using and building is continually evolving. So sometimes it's, it could be just my personal knowledge gap of not fully maybe understanding the solution and how to help the developer, but I would say most of the time it's a really great collaborative process.

Britt, any insights to add on kind of that balance of gatekeeper versus championing partners through the gate?

Yeah, I don't know. I really like Marina how you framed that, and I definitely agree. I think for me, like, The way I balance it is just remembering that, you know, both roles are actually driving toward the same goal. And, you know, I think that's where like the balance like really shows up. Like we hold the line on the outcomes like we really care about, but we stay flexible on how the partner gets there as the ecosystem evolves. You know, so like, like the example you mentioned with the beta APIs, like we got pushback, but instead of just continuing to enforce this rule, like we use that feedback to reevaluate whether the requirement even made sense, and, you know, how we can alter that over time. So, yeah, just a lot of conversations that help, you know, the conversations and the feedback loop.

I love the way the partners are included in this, and they, they, they're actually helping to shape some of the requirements and some of the ways that, like, app certification runs. I think there's a lot of places, at least from my experience, where you kind of run into this wall where it's like, well, you have to do as we're, you know, as it's written, or, you know, thanks for participating. Where at least on our end and with all of you, it's more of a working with them or talking with them instead of talking to them type of like scenario, right? If you to use a, you know, whether that's the best analogy or not. But yeah, I find that to be one of the things that I think other places lack that at least our team and all of you do. So much better than most people do. So, I think that's fantastic.

Yeah, it's, it's like really like less about like gatekeeping versus like championship, but more about like partnering together, you know, to like get to that common goal.

That makes sense.

That makes—

and also the idea like, you know, we can iterate based off of not only just what our partner feedback, but what matters most to customers, right? Because you mentioned trust, customer trust is really what that value that the certification badge, it makes you more installable. So, as we develop and we learn what's important for partners to be able to build and what's important for customers at the end of the day, what moves the needle for their decision-making is, is cool that we are, you know, got this, we've got that feedback loop, you know, we are iterating and improving. And speaking of improving and the evolution of the program and how it's changed, we've seen a lot of changes over the past few years. I mean, Marina, you're, you're one of the OGs, you've seen all the changes, you've watched it take shape. For example, raising the minimum active install requirement from 6 to 60, huge change. What drove that decision and what did you learn from that change? And Britt, I'm going to tap you in for this one.

I'll be happy to take that. Yeah, that was definitely one of the bigger, more controversial changes that we've made to the program, at least since I've been here. And I hope I don't go on too much of a rant about this, but for a bit of context for anyone who's listening, about 2 years ago, I'd say we We raised the minimum active install requirement for certification from 6 active installs to 60. And because like that is one of the 2 baseline requirements just to apply, it was a pretty significant shift. But the change, like it didn't come out of nowhere. The program like has evolved quite a bit over time. Early on, it was more of actually a marketing initiative and it had multiple tiers, you know, different install thresholds. 5, 50, and I think the top one was like 500 active installs. And even things like NPS were baked into it. So then in 2020, we relaunched the program and simplified it into a single certification tier with a much lower bar with 6 active installs, just to make it easier for partners to get started and drive early adoption. But the challenge we started to see there was that the bar for certification wasn't that far off from the bar for being listed in the marketplace, which is still 3 active installs. So at one point, I think the majority of listed apps were actually eligible to apply for certification, you know, which it's quite a bit. So it told us that creating like a lot of work, we weren't creating a strong enough distinction with that. So from the beginning, we always believed that adoption was a part of quality, you know, that a certified app isn't just is technically sound across things like security and performance, but that it's actually used in delivering real value to customers. You know, raising the threshold to 260 was really about strengthening just like that signal. Installs are still one of the clearest indicators that an app is not only functional but truly adopted at scale. It also better aligns with— or better aligns certification with the direction of the ecosystem and the tech partner program, where adoption is closely tied to like partner success and like deeper investment to and from HubSpot. Uh, so yeah, that said, one of the biggest things we learned from this though is that changes like this don't impact every partner the same way. So for more mature partners, it made a lot of sense. Uh, but for newer or more niche apps, it felt like a pretty big leap. Um, so it really enforced for us that if you're going to raise a bar like this, you also have to create a clear path to reach it. You know, making sure partners understand the how and like feel supported along the way.

So.

I think when we launched this, like we had a pretty wide window for existing apps to meet this goal. Like it was about a year. And I took a lot of the feedback we received to heart and stayed like really focused and committed, like really focused and committed to helping partners, especially those who had previously been certified, find a path forward. So yeah, I guess at the end of the day, like raising the standard and evolving your programs is important, but also like how you bring partners along with you. Is just as important.

Marina, having been, you know, having started before, seeing the whole progression of the certification program, when this started, when this was launched, when this was even brought up, how did you, you know, feel free to be controversial. Did this scare you?

Were you like, what are we doing? This is a terrible idea.

Or did you feel like this was a great move? And how do you think it, you know, it landed with everyone?

All right. Yes. So as the kind of the OG, I would say, for certification.

Okay.

I think basically—

Marina, the OG.

Basically, I think it's typically the historian. I think the historian's the correct name. Yeah. I would say, yeah. So I would say, okay, the beginning, it was significantly easier to review the, you know, if it's 6 or so active installs, but I think The, those apps that were, many of them were certified, right? When they were like 6 or above that, they have also grown and been successful in the marketplace. So I, I thought it was a good change. Of course, that did create some maybe tension or friction for, for the customers and for the developers to kind of continue to maintain that or to try to meet that new goal. But I would say the One of the requirements, one of the main requirements we look for in certification is the API success rate and wanting that to be 95% or above. And when it's, when the app has 6 active installs, it is harder to find like those error spikes or those issues with the performance and reliability and actually provide better solutions for them of how to resolve those errors because the volume, the API call volume over the 30 days, which is what we look for, for 6 active installs, it's going to be small versus when it's 60, it's significantly higher, right? So it's no longer a few hundred calls, maybe it's into millions. And I find that that's where we can truly provide much better recommendations and we can look at data and say, you know, this is how you need to resolve these errors, or there's an issue with this specific endpoint or how to resolve these error messages. So there's not much room to, I don't want to say hide, but to hide when there are more customers adopting and using the app because there's just much more volume. And it also means that because there are more customers using the app, they're probably using more of the features. So if an app happens to have maybe, maybe 10 features, right? Maybe they're integrating with contacts, deals, companies, and a few other objects. If it's 6 customers or, you know, let's say 10 customers, maybe they're using only half of them. So, they're not probably fully integrating with everything that that app has to offer. But when there are 60 or more customers, likely all of those features are being used. All of those endpoints are being used. And it's just that much easier for us to review and confirm the functionality of the app. And provide the solutions on how they can improve. It does mean it's harder for maybe developers to meet that mark then, but I think it just becomes, um, much more productive conversation as well. Got it.

Would it be true to say too then that it also benefits the actual, uh, provider as well? Because that the more usage that they get, cuz let's face it, like even when you're, um, testing or when you are trying to like dogfood, if you will, there are things that as a developer that you might miss when you're building your app, you're building your theme, you're building whatever. So that requirement to have more of the active users, like you said, using the features because in a small data set, like even when you're doing SEO, I always find like when you're trying to like split test or you're trying to do whatever, like the sample size, you're not going to be able to reach that significance. With a small sample size. So for a developer, it, I think like when, when people do give that pushback, I think they sometimes forget the fact that like this will also force them to have people use their product more. And that feedback for them will be beneficial because really at the end of the day, you may find more things wrong, more things to improve, more things to, because really when they're done with all of that and they do finally get to the certification level and it gets into the marketplace as a certified thing, it's only benefiting the customer, you know what I mean? Being, you know, customer-centric, customer-first, solving for the customer, all of these things kind of move their way up the chain and eventually are beneficial for the customer. So I think in any marketplace, you're probably gonna get pushback when you have to change some sort of requirement or something people are used to. But I, I, I think your example of that with being able to kind of have people use more of the product to get better feedback, I think sometimes people miss that and it's a shame, but I think eventually over time people start to realize that, yeah, okay, I get it. This was a really good move.

No, exactly. I think it is also a great way to, um, almost like pressure test the app as well when there are just more customers. Because when, uh, you know, let's say it's only a few customers and they don't have as many, let's say, contact records, then again, there's not as much volume. But, I've seen many times during a certification review where, let's say if we, we take the 60 active installs, like 59, like 59 customers, like there are no issues. And then there's just gonna be like this one giant spike of rate limit errors from this one customer that just has millions and millions of records. And which shows that, okay, this something about this app in particular, there's a difficult time handling a customer who just has so many records and probably at that enterprise level. So that's where we can collaborate and help them improve those rate limit errors and simultaneously sometimes bring in other teams as well, like the solution architects, to resolve them.

That's a problem we have in listing with just the 3, right, where we don't look into API activity, but Yeah, like usage, it's hard to see when, you know, we have ways for them to do it, but I, that's a great benefit as well to be like, get more people using it so that we can give you better advice. And another change I wanted to touch on as well, and it's pretty recent, but we, we've moved to 2-year certification recertification cycle with clear expiration and renewal dates. Why did we land on that cadence and how do you feel that it changes the relationships that you have with the partners that you're working with. Britt, you want to start this one off?

Yeah, this is probably the other biggest change that we've made in the last few years. Yeah, I know, like we've mentioned this a few times, but like at its core, certification is meant to be that signal of trust in telling customers that an app has been reviewed and approved by HubSpot and meets our highest standards of quality. But the big question that we had to answer was like, what happens after an app gets certified? And like, maybe I'm airing a little bit of dirty laundry here, but like like historically, like certification, like it didn't have a clear expiration. It was more like a one-and-done milestone. And in reality, that meant apps, like, weren't being actively monitored in a consistent way, unless something was flagged after the fact. And that became— and that, I think, at first, like, wasn't, you know, the biggest issue, you know, going off what Marina mentioned earlier. But over time, that became a real gap and problem, because if we're telling customers that, certification represents quality and trust, we have to be confident that the apps are continuing to meet the bar, not just that they were approved, but that they're like over time continually meeting it. So yeah, as the ecosystem grew and like our standards continued to evolve, that, that gap just continued to become more apparent. We had apps, you know, certified across a wide range of time under very different requirements. and the process that we had just wasn't keeping up. You know, like that said, we had experimented with recertification in the past, but it was just, it was very manual. Like Marina could, yeah, could talk more about this, but it was very manual, hard to scale, and lacked consistent, like, partner engagement. Um, and it just, like, ultimately wasn't sustainable. Um, so yeah, moving to an automated, keyword automated, uh, 2-year recertification cycle was really about solving, like, solving all of that. It gave us a consistent, scalable way to ensure that certified apps stay compliant and up to date. We landed on 2 years, like, pretty intentionally. It just creates a consistent, predictable cadence for partners while still giving them enough time to build and iterate and grow before going through the process again. 1 year just felt like it was too soon. And 3 years, Seems like too much, but I think, and yeah, 2 years is the sweet spot. But I think the biggest shift was how it changed or will hopefully change the relationship we have with partners. I think, you know, it moves us away from that one-time review to more of an ongoing, like, lifecycle and relationship and, like, conversation. It creates more structure, more accountability, and also more opportunity for partnership. Whether that's helping partners stay current with, like, new best practices, new developments, or supporting ongoing maintenance and improvement, or just, like, prepping them for renewal in general. And yeah, and ultimately, that's better for everybody. It promotes a deeper investment in the long-term health of the app, which improves the overall quality and integrity of the marketplace in better supporting, you know, both developers and customers. And, you know, like at the end of the day, like certification, like isn't just something you achieve once, but it's something that is now, can be now maintained.

Plus it helps.

Yeah.

And HAP can change, right? Over time. Yeah. Sorry to cut off.

There's a, there's kind of like a proactive versus reactive type of mentality where it's easier to have these things be recurring so that way, like you said, you can keep up with the trends, you can reevaluate things instead of being reactive to say, well, this hasn't been checked in X amount of years, maybe it's time. Nothing's gone wrong yet. If something does go wrong. But yeah, I— it's to me when, when that came around, when that rolled around with the other changes, it just was one of those things where you heard it and it just made sense. Right. And then 2 years being like the Goldilocks principle of years, you know, as far as like your choices, it just everything, you know, I wasn't directly involved in it, but like as it was being explained by both yourself the Marine and others, it just, it was one of those things that to me made perfect sense. It was like, there wasn't anything where I would say like, yeah, but there was no yeah, but moment. It was like, that makes perfect sense.

Yeah.

I wanna, I wanna add, and one, I wanna say Britt did an incredible and phenomenal job at rolling out recertification. There was so much, so many metrics pulled. There were multiple phased rollouts. And actually introduced a lot of product changes as well. So now developers can see that they can, and, you know, click on renew certification within their developer account. And yeah, as Britt mentioned, we tried doing recertification before it was a fully fledged program as a pilot. And it was incredibly, I would say, painful for, for us reviewing it and even more painful for the developers. Partially because we didn't have the correct points of contact because people, people come and people go, right, in different teams. So someone who was at the company, we didn't have the latest contact information. We would try to reach out, try to get them engaged. And, you know, sometimes it would be like we would remove the certification and then all of a sudden someone would be like, wait, why is there this change? Like, I was never notified of this because they weren't and we didn't know who the, the correct point of contact was. It was challenging, and thank you for, for taking on this project and, and rolling this out because this is such a, such a big and important initiative of really making sure that recertification is a commitment from developers to continue to uphold these standards without us necessarily constantly monitoring and creating all of these additional flags of, oh, are they still meeting this bar or not? And then introducing the conversation. So it's, I'm glad that everything is outlined and we have a great playbook for it. So thank you.

I'm like, okay, now I'm tapping, I'm going to tag Britt as the go-to person for when we want to do the renew listing program. So Britt, get that ready on your roadmap.

We'll look to you for the work there.

That's going to be a fun one. You can stand by screens.

Oh my gosh.

Like diving into the actual review process, I'm a partner, you know, our favorite example, I'm acme.com, I've been in the marketplace for a while and I, I want that badge.

I wanna apply. What can I expect?

Can you walk us through what actually happens from the moment John at acme.com says submit for certification to the moment that I hear back from the team? What does it look like?

I can start off and then feel free, feel free to add. So I would say the moment, Johnna, you hit submit, the app will enter into a queue. We like queues. We have a lot of queues for everything we do. Love queues.

Love queues.

Yeah. From review listings to updates to certification to I don't know how many queues. We actually don't have enough queues, but yes, the app will enter into a queue. You know, it's, it's a collection of all of the apps that have applied for certification. And we will know that, Janna, you applied, so you would be the submitter and you are going to be the main point of contact. At least that's how we see it. So if you don't want to be the main point of contact, have someone else submit. Um, and it will actually be unassigned. So we don't necessarily have, um, where, you know, we take turns assigning the, the reviews. I would say a big portion of it is depending on our workload or maybe our familiarity or not even familiarity with the app, we will take it on. So let's say I'm the TPN, I see it, I will assign it to myself and I will start the review process. Britt, I'm going to plug you in for any automation. Do you want to talk about that?

Yeah. So once the, once your app, Janna, is submitted and we receive it before it's even assigned, you will receive an automated email with an initial checklist that you're able to go through, and it just gives you a look at all the resources we have, you know, our certification requirements, the certification course, and some of the top items that are often missed, uh, when, you know, prior to submission. Uh, so you'll receive that email, and it's something you can work through before we even get the chance to assign the app to ourselves or get the chance to send you the initial review.

Cool.

Okay. So I've got, I'll, I'll know some of the things I can, some boxes I can check before I hear back from you. That's really helpful. Okay.

Yes. It'll include like a self-guided checklist of, I will say anecdotally, the most common errors that we find that I would say are the easiest also to resolve. Yes. I will say the, yes. And I, so before we actually review everything, We review two requirements. There are minimal requirements to be eligible for certification. So just because you apply does not necessarily mean you are eligible because there are certain checks and balances that we need to review. So we require that the app has been listed for more than 6 months on the Marketplace and has not been rejected in the past 6 months. Now, it might say in our internal system that it was rejected for various reasons, but we have our own internal notes that we can add if there is an exception made. And that's very much just us wanting to have a log of all the activity there. The other requirement, as we already talked about, is the 60 active installs. So we will review that. And if the app does not meet, if your app genre does not meet these two requirements, it will immediately be rejected.

Heartbreak.

Yes.

Okay.

Yeah. So we will send you, we will send you an email. We will notify you and explaining why it was rejected. This does happen specifically around the 60 active installs. Again, airing out the dirty laundry. Unfortunately, the developer account currently in that gating mechanism looks at 60 total installs instead of our actual definition of 60 active installs. Installs, where an active install is an account that had at least one successful API call in the last 30 days. So that is unfortunately not part of that gating mechanism. So you might have, you know, 60 total or even 200 total. But if you don't have the 60 active installs, your app will be rejected. We'll send you an email. We'll explain exactly why. And I personally create an alert in our own internal system to know that if your app reaches over 60 and you're eligible, then I can reach back out to you and say that you are, you are now ready. Marina, what a kindness. It does. I try to make sure I do the same.

Yeah.

It happens a lot and we feel it's something we hope to improve though. Like we're working on it this year.

You know, this is, this is why I say like it can be rejected in the past 6 months. And then we will actually say like, yes, it's rejected, but there isn't, you're an exception. You can reapply again. It's okay. Like, please do reapply when you're ready.

I was going to say, I, Janna, the owner of businessacme.com, would have hoped that if I'm not eligible, I can't click that submit button, but it sounds like there's some things in the works, some improvements for, for some of that to maybe get into place. So that's very cool. That's very cool. And outside of obviously those big minimum eligibility requirements, what are some of the most common reasons that apps may fail, like the trends.

I can, yeah, I can answer that. I would say, yeah, the most common trends, and this is honestly not even a requirement, but I find that the most common reason apps fail is because they're just not ready. And it simply means, and this is a very abstract umbrella answer, I would say it's like it was not a good time for them to apply. And this could be mean that Maybe the app is going under, going through a rebrand, right? So then like all of their documentation, their logos, their names, like everything is going to change. So which means it's probably harder for honestly for us to review what the app is currently looking like if we know that there's a name change and all this documentation and copy, marketing copy change. Another reason it could be not a good time is if there's a big change in priorities. So maybe they thought it was a good time and then maybe someone went on long vacation or, or leave. Maybe there's, you know, a restructure within the team and who's owning certification review and that main point of contact is no longer there. It could be that they're developers working on a big feature release and there's just constant changes to the app. Which also makes it difficult for us to review it because we might be saying, well, you're not using the scope and, or asking how they're using a particular scope. And the developer might say like, oh, well, this is going to be released in 2 weeks. And then that's where I would say it's probably best to just hold off, release it, wait until your app is in a more of a fixed state and you're not actively developing it and then apply. So I'd say that's honestly the most common reason. It's, it's just the commitment.

I imagine too, with the branding example, there's gotta be providers and developers that are greatly appreciative of that, even though you said like it may not be like an actual, like physical requirement requirement in the traditional sense, but the last thing people want is to download an app and then a day later is the thing I just downloaded isn't called what I just downloaded. Like, what's going on here? Like, I'd have to imagine that there is some sort of like really great appreciation from the developer end of things that we're even looking out for those things where most places, if it's not like black and white and it's not ones and zeros and it's not, it doesn't check that like if they're, if not their box, it's, I mean, it sounds like, and I kind of already know the answer, but we go an extra step further than most people would to kind of look out for these type of things that could have a negative impact from the customer standpoint, which then trickles down to the provider, possibly bad review, like, you know what I mean? And the confusion. So I think it's— is it safe to say that we kind of go above and beyond what you would typically expect in a marketplace?

Yeah, I, I mean, yes, I would say that at least we try to. And I, I mean, I would like to say that, you know, I hope that we do. And I would say if we do give, we're generous with our time and whether that's providing the solutions or even asking like, do you want to pause? Do you want to restart? Like what, how, what's like, what is the best approach right now? Because at the end of the day, we also do value our time and we value the developer's time. We ask that the certification review is completed within 60 days. Right? So there's a lot of this back and forth or trying to reengage the developer, then it does extend that timeline and it's significantly easier if, you know, there's that commitment upfront, not just from the person submitting it, but from, you know, maybe the person who's responsible for the marketing, the supporting documentation, the development team as well. But Britt, do you want to mention any of the other Reasons that are actual requirements that YApps might fail. For sure.

And just to expand on the first thing you said too, like, I do think it's a big part of it is honestly just underestimating the time and effort it takes, especially now, like, quick plug, that we have certification statuses, like, directly in your developer portal. And, you know, developers or whoever's in the portal is able to see that, like, hey, my app is eligible for certification. Click here to apply. Without really understanding, like, how big of an investment this could be. You know, it often spans multiple teams, like, from engineering to product to marketing. It can be a pretty big lift. So maybe, yeah, just, like, knowing that before going ahead and applying for certification is, like, pretty huge. But also another reason I think, like, personally, I see a lot of apps fail, and this has come up a few times in this conversation, is just, like, due to, like, feature success rate. One of our requirements is that we have apps, or that we have is that apps need to maintain at least a 95% success rate across all of their activities. And that's one that can be especially challenging for partners, but not because they're, like, necessarily doing anything wrong, but because they don't always have the same aggregated, holistic view of performance that we do. So from our end, you know, an integration might seem like it's working well on the surface, but when we look at the data, we might see things like a high volume of failed requests or patterns of rate-limiting errors or like maybe just like that one specific portal, like the example we mentioned above. And like once we surface that, we can say like, hey, a significant portion, like, you know, 75% of your API requests are failing due to something like 429 errors. Partners can absolutely go and fix it, but before that, they don't often have full visibility into like how widespread those issues are, or like what the best practices can be, or like, you know, and like even when they do go to fix it, it takes time, you know, and our window is typically 60 days, which isn't always enough, especially if a, you know, a partner's looking at their app and they're like, hey, maybe I should just rebuild this from the ground up. You know, that doesn't always fit into that 60-day window. And we've been known to make exceptions to this timeline. But I definitely see that as one of the more like time-consuming or like challenging requirements to resolve within that window. For sure.

Got it. Got it. Got it. And that makes sense too, especially if you're looking at it over a period of time, you want to see consistent success rates. You know, once you implement change, you're not going to see it happen immediately, right? You're going to have to wait for that, that change to take effect amongst, across all the customers that have installed. I want to walk it back, especially knowing that one of the biggest things you run into is that, that's john@acme.com. I did not know how much of a time investment certification would be when I clicked submit. In more broad strokes without needing to get into the nitty-gritty, what is it that you, that you're looking for when you're conducting your reviews? Or like, what are the main things that you're reviewing my app against? Marina, do you want to dive into this one?

Yeah, I would say it's, yeah, the success rates and scopes. I don't actually, I have no idea if I'm at— can you repeat the question? What are you asking?

Sure.

Yeah.

So, okay.

From a time investment perspective, I had no idea how much it would take when I clicked submit. So, you know, around that, knowing it's, oh, there's, there's a lot that goes into what I need to do, or, you know, what threshold I have to cross to get certified. What are some of the things that you are reviewing that result in sort of this big investment I'll need to make? So what are the broad strokes categories or the broad strokes broad strokes areas that you're looking into when you receive my submission and you're giving me feedback?

The requirements that take the most time commitments from you to resolve?

Or just even just generally the requirements, just what is it that you look for when you receive a new certification review? What kind of what's, what are the broad strokes things that you're looking for that you're checking against?

Okay, so the main, let me pull up the certification requirements to make sure I don't miss every single category. No, no, no. So yes, to kind of bring it back. So basically the question is, you have more than 60 active installs. So what is everything that we actually do review if we review your app and the feedback that we share? So yes, one, we will have a template that we follow. So every single app is evaluated, I would say fairly, and against the same requirements that are on our public documentation. There are, I would say, on the doc that we will share where we do share the feedback. There might be new requirements that are maybe not must-haves, but it might give you actually insight into maybe something that might be rolled out in a, in a few weeks or in a few months that we might be either trialing it or we just have not yet fully committed to and released. So there also, we will include additional recommendations. So there might be some additional things outside of the certification requirements, but the broad strokes of the requirements we do review, we really break it down into a few different categories. So we have the main category, and I would say this is the most important one, and we will receive pushback, and this is where we push back again. This is where we really want to make sure that we land is the security, privacy, and compliance. Compliance, this is a non-negotiable. We want to make sure that your app is only associated with a single app. You're not asking developers to go and create a private app or using multiple apps. It should be one app, one listing, one and done. The other one is you're using OA. You're asking and only requesting the scopes that are actually needed. So in that one example, if, if you're a developer and you're saying, I have not yet, um, like maybe you have not yet introduced, you know, company sync, let's say functionality, and you're thinking of launching this in a month, don't add company scopes, add it to your test app, add it to your second environment, test it, make sure that works. And then when you know that launch day comes around, then add those scopes to your production app. So we will review every single scope. And this is, I would say, a common failure rate. And almost every single app has some feedback around this because honestly, we just don't really have good developer-facing tools that help developers verify how a scope is mapped, whether that scope is used. And there are hundreds of scopes. At this point. So it is very difficult, I would say, to understand how each scope is used, when it's used, whether it should be conditional, required, conditionally required, or optional. So we would also guide developers to maybe move that around and shift it, whether it's actually move required scopes into optional or maybe the other way around as well. We want to make sure that their company domain is owned and verified. For the next category is, oh, and also for compliance that you're following all the developer terms, all the, if you're using sensitive data, the sensitive data terms, the branding compliance. So all, all of the terms that you're agreeing for being a partner in the marketplace and using HubSpot products, you're agreeing and following it. So we will, we will review that and we will, provide feedback, and that is a non-negotiable. The next big section, as we talked about, is the reliability and performance. And this is that you have the 95% success rate across all of your activities. Got it. And also, we will check if you're using maybe some deprecated endpoints that are causing errors, or if we see that you're using an endpoint that might be sunset in a month or so. And if we notice that, we will notify you. It will not be a failure, but we just want to make sure that you are using the latest versions of our APIs. Got it. Then we have the usability and accessibility, which realistically just says what your app says it does, that's what it does. And all of the images are showing your latest app so customers know and trust what they see in the listing is what they're going to experience when they install and use that app. The, and that means like the pricing is correct. Let's leave it at that. Oh, and the supporting documentation is there. A customer should not be confused how to install your app or how to configure your app. So we highly recommend using visuals, videos, demo videos, images, you know, GIFs or GIFs, if you want to call them that. I'm sure I'll get heat for this. We don't need to start that debate. Maybe we do.

I don't know.

Maybe we do. But no, the support and documentation is important. I would say it is. It really makes a much easier experience, one for us to test your app and to know what it does, but also for customers to have a good experience and not say, okay, I can't set this up. This is not going to work. I'm just going to uninstall this. And the last but not least is value. This is kind of going back to the gatekeeping championship and the success is we really want to confirm that realistically your app, what this, the, the solution that you build for customers is, that's what it is. And that's what it does. And it does it, you're doing it successfully. And we're not necessarily here to judge whether customers need this or whether they will find it useful. So we trust that the developers have completed their own customer research and they know the customers want this and it will be successful. So yeah, as far as value, we just want to see those, the 60 active installs. Britt, anything, anything I missed? That was a pretty long answer, but hopefully I covered everything.

I know, not that I can think of. I think, I think you did a great job in covering everything. It's a pretty extensive list.

Yes. Okay. I get why it's— John at Acme.com now has an understanding of why it is a big investment. There's a lot of really good feedback that I'm going to get out of this process and optimization. That's incredible. Do you, knowing that, knowing how much goes into it, how much you're reviewing, how many improvements I'm going to make, from start to finish of this process. Do you have any before versus after stories where an app leveled up and clearly, because I can't imagine apps coming out of this are not just seeing massive improvements, both in performance as well as in, you know, a customer adoption. Britt, do you have any examples?

I'll start if that's okay. I don't know. We can't, well, one, we're not going to, you know, drop names, right, as far as the success story. Yes. But I have seen where maybe the developer actually had two public apps, you know, which realistically makes them not eligible for listing or certification, but they were just trying to work around a particular solution that maybe was not yet available. And then we were actually able to get on a call with some solution architects and change their app so they only have one. So that's a very broad, not specific answer for a success story, but it doesn't mean that the customer only has to install one simultaneously. Sometimes we have split add-ups as well. That is more on the rare side. But Britt, do you have any success stories?

I definitely— I don't have any specific examples off the top of my head, although I know there are a bunch. I think certification, or I hope certification helps every app level up in a improve, you know, to some extent you do get, you know, one-on-one time with someone from this team. And like looking back, like rarely have I seen an app apply for certification and pass immediately. Like I think I've maybe seen that happen once. So it's like, it's typically like a lot of apps like are failing to meet a lot of the, at least half of the requirements, you know, that we enforce like before they even apply. So I think that alone is hopefully helping the apps level up and improve. In some regard. But yeah, if you have, if you have, if you're an app that has applied to certification or has been certified or recertified, like we'd love to hear your success stories. Like, please share with us. Love to hear them.

CTA. Perfect.

I love that.

And yeah, it sounds like again, so many things.

Yeah.

Yeah.

They can put it in the, in the comments for the, you know, on YouTube here where we're going to post this. And then once we put this out on LinkedIn as well, like we would love to hear that feedback and then we can kind of bring that in-house and everything like that. So yeah, if you're listening now and you're on YouTube or, and you see us posted on LinkedIn, feel free to drop us a line and let us know.

Put a comment. We want to hear your success story, your specific one. He's very lock and key on there on calling out any specific partners. I get it. I tried to sneak by you, but that's okay. That's a great CTA though. I hope we get some fun stories from our partners. In that vein, you know, Brett, you mentioned 50, you know, they're missing half the requirements when they come to your door. If you could give Aptean sort of a pre-flight checklist before, before Jonna clicks submit, what 3 to 5 items would you put on there?

This is, that's a toughie.

I mean, you do get a checklist after submitting, but no, prior to submitting, I definitely think like, you know, just knowing the time commitment, it's really important to understand, that and like the requirements that will be asked. So, you know, hopefully a partner, Jonna, you'll take the time to, you know, go through the academy course or go through the, the requirements. You know, our review shouldn't be the first pass, um, and oftentimes it feels like it is. So definitely, yeah, just taking the time to review the requirements. I think like some of the easiest things, like right off the bat though, is that's like very often missed about like 90% of the time is definitely like verifying that company domain. You know, you only have to do it once. It verifies it for all your apps. One and done. You're good to go.

Checks a box.

So that would definitely be on my list. I think like reviewing the scopes to as much as your ability, just making sure that you're only requesting those that you're going to use, those that you need. Reviewing your documentation, just making sure, you know, HubSpot spelled correctly, even though I think our validation will do that for you. And like your install guides in there, you know, from like the beginning to the end, even to like uninstalling. Those are the first things that come to mind for sure. Marina, do you have any big ones?

Yeah, I would say the, I mean, those are great. I would say that's definitely an excellent start. It's, and the key there, we should not be the first ones to do this. Like this should not be the first pass. The, I would say the additional things is to have an up-to-date listing, have up-to-date pricing, This should be— this is not— this is a certification requirement. It is a listing requirement, but it also, it is the best way to really showcase like the latest version of your app and the pricing model. That way customers again know what they're installing and what the experience is going to be like. The additional thing, and this is part of the form when you do submit for certification, is having a demo video. And, and this is, this is not a marketing video. This is not this beautiful, fun, catchy song marketing video. This is very long, probably boring video where hopefully you're specifically describing the end-to-end process. And we can also drop a link probably in the comments or somewhere for what it is that we expect in that demo video. Where you are acting as the customer and go, go to the marketplace, install the app, go through, like almost have, you know, support documentation on one side, you're up on the other side, go through all those configuration settings, test your app, test all of those features, and then disconnect, uninstall, or show how all of those changes are going to be in the HubSpot side on on your platform, what that experience is. And one, this provides us a great resource of how you expect the customers to use your app. But I think it also, in some ways, forces really the developer and like you as a partner to experience what it's like for customers to use and go through that installation process, that end-to-end process, and also use not use their latest production app and the state that it's currently in. And if you're doing this before you're applying, then you'll probably see if, you know, if you added a scope that maybe wasn't supposed to be there and it causes an installation issue, you are the developer, know this firsthand, not us necessarily. So having that demo video and testing your app thoroughly, um, should definitely be on that pre-flight checklist. The, the scopes and errors, I would say for me, it's almost secondary. Yes, this is something that is extremely important, but we also understand just how difficult it is to properly, um, remediate those errors as well. Uh, so we are, we are here to help do that.

Force dogfooding, find, find some of those usability issues upfront, solve them before we flag them to you. And then for the things that are very HubSpot specific that we have a better view of, we'll help you out.

We'll hold— I love that.

Okay, so I think my last question for you lovely ladies today, for apps that are already certified, what would you say are the best ways you've seen partners leverage that badge in marketing, sales, product and how are they actually moving the needle on showcasing and, you know, really just being success in the marketplace as a certified app?

Right.

I'm going to tag you in first.

Okay.

Well, I mean, we love to see it leveraged. We love especially, you know, let's say the most common places I've seen it shared was like LinkedIn posts, blog posts, you know, company websites, websites, you know, set up guides for the app, the app listing, of course. But we also love to be tagged in these success stories so that we can also help share and promote your app as well.

So, so Marina, do we have— what resources do we have for partners when they have gone through the successful certification process?

Yes. So once your app is certified, within the actual approval email, you will receive a few resources and actually a few soundbites immediately. So we actually encourage partners to celebrate the fact that, you know, they earned the certification, they've gone through this process. So we have a soundbite for them to share, let's say, on social media, whether it's LinkedIn or other platforms. There is a brand folder as well that has access to the certification badge that can be included on your website, videos, anywhere you want to include that. And there is a PR kind of communication email as well. Or template to complete to get that approved.

Got it.

Okay. So we have resources. Perfect.

I was going to say, once Janna@acme.com gets her app certified, take to LinkedIn, take to the socials, tag Ecosystem Quality. If I'm working with Britt, Britt, thanks so much. I'm certified. I'm going to have an email, it sounds like, with all these resources, badges, comms, imagery that I can use to share my success. And I hope Anyone watching this, if you have gone through this certification experience, if you see this posted on LinkedIn, put a comment, share your experience, let us know how it went, let us know what you liked. You know, we want to hear from you.

So that's, I feel like those are, those are my CTAs that I'm taking from this.

So perfect.

And I would say if you're a partner who's completed it, the review, or honestly, even if you, if you did not complete and, you know, maybe there was a time commitment or something else that happened. We absolutely love to hear your feedback on how we can improve, whether it's the documentation or how we communicate with you or just the general process. You know, maybe there's a specific requirement that was maybe painful or just wasn't clear. Please, please share that. That is not going to impact your standing with us or your certification badge. You know, you can be— you can provide constructive feedback and criticism. Criticism. Again, we're very much here to serve the customers and the developers. And end of the day, this is one benefit that I think Britt or it was already mentioned, like, this is free. This is at no cost to you as a developer completing certification. This is very much like one-on-one time with one of us to review your app and provide improvement. So again, if you have, if you have feedback, whether it's program or how we work and communicate, we want to hear it.

Awesome to hear that you guys are open to that feedback.

I love it.

And that is all the questions that I had. I want to thank you both for answering our questions so thoroughly. I want to thank Developers After Dark for giving me this true honor of being able to guest host. You know, it's It's been such a joy to talk with you lovely ladies, and I will pass it back over to Nick to take us away.

Thank you guys.

Well, there you have it, folks. Thank you so much to everybody that listened and watched. Uh, Jana, I want to say thank you for hosting. It is awesome. Now I know if I ever disappear, get hit by a bus or anything of that nature, abducted by aliens, that the show will be just fine. It does. I mean, I didn't want to leave it in the hands of Dennis, but I mean, it's, you know, now that we have Jana, everything will be just fine. And then of course, Maria and Brittney, thank you so much. I mean, I'm a little bit biased because I'm on your team, but like the work both of you are doing, and Janna as well, is just tremendous. I think for anybody that is an app developer that wants to really participate in the marketplace and be successful, right? And then solve for the customer, watching something like this and being able to find out what App Certification is and how the program works is tremendous. So I just want to say a big thank you to all of you. And then a big thank you to everyone who has watched and listened. Please make sure you like the video and then make sure you subscribe to the Developers YouTube channel. And from all of us here and from our new co-host for the day, Janna, thank you so much, and we'll see you next time. Thanks.

Yes, thank you. Thank you for having us, Nick.