Beyond Health Scores: How to Build a Renewal Probability Model Built for Modern CS

[Speaker 1]
Thank you very much. If anyone is wondering, the movie is Don't Go In The Park Alone. It's on eBay for about 200 pounds at the moment.

I will settle for 20 if anyone does have a copy. It's a terrible movie, don't buy it. Like most of them, they're all awful.

No, they're not all awful, they're good fun. Anyway, but I'm not here to talk to you about that. That could take up about a couple of hours worth of me chatting.

I want to chat to you all about a topic that's near and dear to my heart, which is renewable probability and moving beyond traditional health scoring in 2026 and beyond. So as the intro said, my name is Sean Reid, Head of Customer Success in Supermetrics. We are a marketing data software.

We connect all of your marketing tools into one spot so we get all the data into one place and create wonderful dashboards for you to be able to show fantastic marketing reporting. Outside of that, I'm also found on LinkedIn a lot waffling and just talking to myself at times, but having a bit of fun with that. And most importantly, I am dad to two wonderful little girls, Vivian and Evelyn, who I have a little ask of you all later on before we get to the Q&A.

So you know the Q&A starts when I bring up my kids. But let me bring you on a little story first. When I've been thinking about health scoring and I've been thinking about the journey that customer success has gone on over the last decade, I go back to when I started as a CSM.

I was in HubSpot. We were very early days HubSpot. We were at the stage when most of our phone calls would start with, hi, this is Sean from HubSpot.

H-U-B-S-P. And when we spoke to customers, we had health scoring. And health scoring was fantastic.

It was a great tool to get us started with customer success. We could see how our customers were doing. We would rate them on green, yellow, orange, or red, and be able to say, this is us being proactive CS.

And I dare say for most of us in the room, this probably feels familiar. That is the health scoring model we had. But that was 10 years ago.

And over time, our customers evolved a little bit. Their graphics updated, their use cases got more complicated, their systems got more complicated, and their questions got more complicated. But we were looking at them going, yes, you are a green customer.

Everything is fine. They were going, I don't feel fine. I still have a lot of questions.

You're green, though. Don't worry about it. Then they got more advanced.

And we were still sitting there going, well, you're green. Everything is fine. Don't worry about it.

Your usage is really good. But as they got more advanced, their questions got even more advanced. And they were stuck with more pressure being put on them as their tech stack got bigger and bigger and bigger.

But we in customer success were still in the corner going, but I don't understand. You're green. Why are you unhappy?

Why are you now ignoring me? Why don't you return my calls? Why don't you answer my emails?

I'm right over here. Hi, please talk to me, customer. You're green.

What's going on? And we had so much pressure being put on us to match the modern customer. But we were still stuck in the past.

The frustrating thing about this was when we would leave our CS land and go to all of our other teams in these situations, it felt like they had also moved on too. Their processes were more advanced. And they were getting investments that matched how complicated things were for them.

From our sales teams to our marketing team to our finance team, our people team. They're all doing these amazing things. And they're able to show, look at all this incredible stuff that our systems can do.

Look at all these amazing new methodologies that have been launched. And we were just walking around with a churn problem and going, but my customer's green. And we're not meeting my numbers.

Someone help me. And it felt like no one was helping. It felt like we were still stuck with this way of working.

And the problem was, there was a cost with all this waiting. As we were waiting to get investments, as we're waiting to find out more and more for ourselves, we were getting all this bombardment of, if you don't respond to customers now, there is going to be massive churn problems. You need to be more strategic.

You need to be more proactive. But our health scoring wasn't allowing us to do that. And this is not to turn around and say health scoring is awful.

It's to say that health scoring was stuck in the past. So what is the next level for us? And what is the thing that gets us to get into modern CS with everything that you've seen in today, what you're going to see tomorrow, and what you see with so many other talks about modern customer success?

It's renewal probability. Renewal probability is the thing that brings us to that next level. And it evolves our health scoring model.

Now, what exactly is renewal probability? I've just been talking about it. It's like real probability, renewal probability.

Well, let me go to those first bits of the cost of waiting. And what I mentioned there in my little scroll of saying, like, there are three big reports you have to look at. This is the three things I've noticed that have definitely highlighted the problem we have in modern CS, and how fighting churn is a time-sensitive game.

Number one is kind of something we should be really looking at as like a positive for us, right? It's five to seven more times more expensive to acquire revenue than it is to retain revenue. And with that in mind, it feels like for us, well, now is the time for CS to be more of that revenue ownership.

We can show that when all these businesses are watching their purse strings, when there's so much going around of how do we reduce costs so we can spend more on cloud tokens or whatever it may be, this is where suddenly for a business, it makes sense for us to invest in CS. But according to Churn Zero, they found that 70% of churn is preventable with early action. What this means is if we went back and said the signals existed for this customer, doesn't matter what they are, if we just hadn't had a human get in front of them at that moment, no churn.

And I see this myself. When I started in Supermetrics, I made my business to interview a whole bunch of customers who had actually left us. They were the first ones I talked to.

And with all of them, I asked the same question. As we've gone through this call, if the things I was talking to you about now were brought up when you made the decision, would you stay? 100% said, yes, we would still be a customer if we had this conversation.

So it becomes really clear to us that, well, early action is the key for us. But early action itself isn't enough. You don't just turn around and say, okay, talk to someone early, it's fine.

Because we have another problem. According to CEO Hangout, they had found a study across, I think it was like 200 use cases here, that the average lag between a customer showing the early signs of we are going to churn versus a CSM picking up the phone and having a call with them was 90 days. A full quarter had passed where a customer was sitting there going, I'm not feeling valued anymore.

I'm not feeling like this is what I need. And if we go back to those slides where we saw how much we can be that revenue generator for the team, how we can see that early intervention is a huge thing to get us around churn, this becomes glaring. Another way to say 90 days is a full quarter.

And this is why fighting churn is the time sensitive game for us. And the problem is most CS teams today still operate in a rear view mirror. Cancellation requests are locked, NPS is dipping, customer escalations are happening.

And we view them as this is the reason why someone is leaving. They are not. These are symptoms of a bigger problem, rather than the signal themselves.

And by the time this has happened, by the time NPS has started going down, by the time usage has started going down, by the time the cancellation requests has come in, a full quarter has gone by. A full quarter where CSM could have been involved. A full quarter where we could have been on the phone or speaking with a customer to say, let's turn things around.

And that is the problem with our modern health scoring. Our health scoring is built to be reactive. A customer has taken a full action and now we see it in the tool, you need to get involved.

That's reliant on a system. And now as we move to a world where we have consumption based pricing, it becomes more apparent that we need CSMs to be involved and engaged with our customers at a high level to avoid all this. And that is highlighting our problem with modern health scoring.

And the modern health scoring, it does have that flaw of it just being something that holds back our teams. If I think about the way most health scores are set up, they are very reactive. A score has dropped after the risk is already felt.

The health score drop doesn't happen when the early signs are developing. Health scores can also be incredibly vague. This health is yellow.

Great. What the hell does yellow mean? And I find if you go to other teams and say, hey, you know what, go to this isolated CS part.

This is yellow. I don't know what yellow is. I'm in finance.

Is that good? Is that bad? And in fact, I've seen it from so many folks who speak to other departments about health scoring, and no one seems to be aligned on what a color-coded system actually is.

And if they don't see that, other teams have a really hard time feeling like I can influence this score. I can play a part in what this turns around with. And as we all know, because we're all CS professionals, I'm preaching to the choir here with this part, we know that retention is a team-based sport.

We know all of us are involved with it. Every department that exists in a company plays a role towards retention. But we use one of our core metrics as a CS piece only, and we make it hard for them to get involved.

And then finally, they're product-centric. Usage has gone down. Logins are now at a lower rate.

The amount of connectors that have been here have gone down away from our baseline. That's the product. We all have cases, and I've seen it, where the product-centric part might show that it's a negative, but they're actually positive, and vice versa.

We are now in a world where tools tend to be used all the way up until the renewal date itself. And if we look at just how much you're using the tool, we're missing the sentiment. We're missing everything else that goes behind how they're using the tool and how they feel about it.

And my biggest flaw with this is that they tell you everything today, they don't tell you tomorrow. The health score is an amazing way of being able to go, right now, this is our usage level for our customer. They can't tell you next week.

They can't tell you next quarter. And if you also ask your customer who, say, is green if they would renew today, the health score gives you that accurate answer. The usage is high, therefore, they're likely to renew.

But renewals are not won today. Renewals are won from day minus 180, the day that a prospect went onto Google and said, I have a problem I'm looking to fix. That is when we set expectations.

That is when we set them up to go, here's how we solve for you. And the ultimate reflection of how we solve for you is a successful renewal. So we then move into renewal probability scoring.

So what exactly is this? An RPS, or renewal probability score, if you haven't heard of it, it's how likely a customer is to renew, expressed with a single number instead of a health scoring label. So instead of asking the question of how healthy is a customer, you're going to turn around and say, is this customer going to renew at the end of their contract?

And that is what we're measuring. And that is a very important shift, because now you're moving from a product centric score and a product centric mindset into a revenue one. You're going to go to one that says this is how we're investing in A, B and C, and how likely it is to develop to a successful renewal.

We're looking at how maybe a customer's looks healthy on paper is actually going to leave or how someone who might be average is actually virtually guaranteed. And that last one I think is important, because we could waste our time focusing on customers we don't need to focus on, just based on what their health scoring was, while renewal probability makes it more advanced. So by moving to this kind of mentality and this different kind of way of approaching our health scoring, things change magically.

And this is something that we've implemented in Supermetrics. It's something I've seen some other companies starting to work with as we've been talking about this, and they're finding the success. Number one is we now have clear language by moving away from green, yellow, orange, red and saying 34% chance this is going to renew.

Can't argue with that. 34%, that sounds awful. This is clearly something we need to get in front of and we need to be able to solve for.

It causes it then to be cross-functional, because sales, finance, leadership, they can all act upon it if we have a shared business wide metric. That just happens to be that CS is the main driver for it and holding the reflection for the rest of the business. It is also action orientated because every input maps what makes renewal more or less likely, as opposed to just pure usage.

And it's forward looking, it predicts outcome before it happens. Now, you're probably going, okay, great. Yeah, I understand this.

But what the hell is it? What does it look like? I actually have a video in a little bit I'm going to show you.

But let's talk about what feeds your score because you cannot have it be something that's purely product, it needs to be a mixture of the product and the human. So for the product side, the things that are automated, we're looking at things like sure enough, their product usage. So we're saying what's the feature debt?

What is the login frequency? What is their core workflow completion? And monitoring that baseline reporting over time?

Are they consistent with where they're meant to be? Do we see dips? Or do we see it going up?

MPS and CSAT, we do bring this into it. But it's not like MPS is seven, oh my god, they're going to churn. It's that's your first response.

Let's look at the trend over time. How is MPS developing? We want to get away from this in the moment metric and instead go to over time, their MPS is improving over time, their MPS is going down.

Maybe it's static. Hell, we all work with different markets. We know some markets will give you a seven.

And that's practically a 10. We know some markets will give you a 10. And they're going to churn anyway, because they're just being overly polite, which takes that trend and trying to figure it out.

Then looking at things like our website, our community, how often did they visit our help documentation? How often did it go to our message boards, our forums? How much did it go to the help center?

How often do they attend events like this that we're hosting as a sign of good insights to them? And then billing, you know, are they late paying? Do they have a plan downgrade or seat reduction?

For us, that makes up about 40% of the overall score. 60% comes from your CSMs. And this is the key differentiator. We look at the human sentiment.

So for example, we will use Gong to log call sentiment, we're looking at tone analysis, keyword flags, stakeholder engagement and calls, how strong are they are doing multi threading with their accounts, how engaged are they with everyone that needs to be engaged with the account. We look at email and chat signals. So your response time, the language sentiment, again, what the escalation patterns are like, CSMs notes and their scoring.

So they're going into gainsight, they're saying I had a call with the customer, here's my sentiment score, here's how I feel about it. Here's how I feel about the strength, here's the champion stability, here's the executive alignment. And then the qualitative context that AI can never catch.

Is this customer going through budget review season? Has this customer gone through an org change that changes their strategy, and therefore we could be at risk as a vendor? Do we still have a strategic fit with these customers?

Those are things we encourage our CSMs to log, we encourage our CSMs to take note on, and it feeds the overall score. So that way we marry the two to be able to say the chances of you renewing are X. So if you're looking to start this and go right, I want to build out something like this, what do I need to do?

Your first step is look at your signals. Go across everything like your CRM, your product, go to GONG, go to your billing, go to your community, and find all the things that could be a signal that we can start to automate. Then we look at the sentiment scoring.

So saying right, based upon how our customers react with our CSMs, what are the trends we need to look for? Maybe we have a very complex solution. So we need to make sure that the multi-threading is really strong, we're engaging with all the different departments.

Maybe what we have is pretty straightforward, so it's just one user we need to keep engaged. And looking at what their sentiment is, put it together, put your weighting in place and say your output is your renewal probability score. And then start getting cohorts of this is high, medium and low probability buckets.

And then start getting your CSMs to action it. Now, I talked about how this could be something that goes beyond customer success. And that is the key with this.

Another one of our failings as a department is, sometimes we will go to meetings and go, this is a CS thing. When in reality, we should be the reflection of everything that's happened with a business. So how exactly does this help you go beyond customer success?

If I'm continuing to punch or kick poor health scoring while it's down, health scorers can't explain to finance which accounts are going to renew in the next quarter. It's impossible for you to be able to generate that with a traditional health scoring. Health scoring don't give sales that clear signal on expansion readiness versus risk.

For example, if they saw one where this is green, great, I'm going to do an upselling. No, this customer hates us, this is the wrong time, or they're going through a new finance cycle. We need to think about this differently.

That is a challenge that health scoring does not allow sales to expand revenue. It also doesn't give the board a portfolio level renewal outlook. And the motivation for non-CS teams is quite low.

If we're saying by doing this to, oh, I don't know, our product teams, who we hear customer feedback from saying this part of the tool sucks, and our customers hate it, and our health scoring shows it, it's easy for them to ignore it. But if we move to saying, here's the probability of renewal, for our product friends, for example, we can say, we can see that when usage drops in this product, we see an increase in churn. This is why we need you to come in to help us out.

If we see, for example, that sales now look at it and go, hey, you know what, chances of renewing are really high, they're in a good stage, they're using the tool well, that's a perfect time for a sales rep to go in and do the easiest sale in the world. We see you're finding success in what you're using, we see the sentiment is really good, do you want more? Do you want to expand more?

Fantastic. Finance then, most importantly, is able to have a model where they go, we can budget correctly. Anyone who saw me today, for example, on the floor, you might have seen I was just jumping back and forth to Slack because this is where I am at the moment.

We submitted our renewal probability to our finance team for the next year, and able to show them, this is what our targets will be based on the model, and based upon what we see happening with sales combined. It's an amazing tool, because then they were able to look at it in advance and say, great, yeah, we can see what our new probability is. Shows them how we can budget accordingly.

So let's take a look at it in action. And this is where I'm going to show you something pretty fun. I'm actually going to show you a dummy version of the renewal probability scoring that we have actually built ourselves.

So this gives you an idea of how it all will work. I believe this is a video, so it should play for me. By the way, if there is any Supermetrics customers here, this is all dummy data, so you don't need to worry about your names appearing.

But right now, what you see here is what I call our forecast. And I'm able to tailor it by all the teams. But up here, I have the model forecasted GRR, this one.

This is where it's taking all of the signals that we see from all the product inputs and all the CSM input and saying, right now, there's an 88% retention rate here. And there's a lot that I've set up. But as a CSM, I can go in here, I can see my renewal probability based on all the inputs, and what is driving it.

We can see low adoption, we can see usage rate has gone down, there's a risk rising, there's no, the auto renewal is off. And then we have some of our commercials in Salesforce. So we can also flag, hey, this isn't in Salesforce, this is.

So this is for a CSM, a quick piece to go, I need to act on it. And the model will also tell them where they're being conservative, or where they're being not conservative enough with this, or they need to adjust their expectations. And the beautiful thing about this thing is the model learns, and it's able to refine what it sees based upon CSM saying this is different.

But I'm a CSM, I can go in, and I can see everything that actually caused the model to be set up as it was, I can see the changes that I need to act upon. And it means I can now get in front of the customers much sooner, because all these actions were not signal wants to go, it's right now for these ones, it's all data queries. That's fantastic.

If I'm a high performing customer. Let's go to a low performing one, like, oh, I don't know, as anyone who knows me knows I'm a big Pokemon fan. Let's go to Team Rocket.

And Team Rocket, oh, dear God, worst case is 0%. Forecasting model is 64.4. It's giving all of these signals to go, you need to get in front of this. The model is telling you, these are all of our signals.

And if we don't act upon them, they haven't logged anything in Salesforce. They're dreadful at their job, typical Team Rocket. But again, negative signals are popping up.

And that's what we want from a Renewal Probability Score. Now, you may have noticed the only color coding I was using was like red, green, just as a catch your eye. This is something you as a CSM need to get in front of.

But when we go to Ops, when we go to Finance, don't see color coding, they see numbers, they see data. And that is where we're able to comfortably say, you are likely to renew because of A, B, or C. And as a CSM, these are the ones you need to get involved with.

So, if you Oh, and by the way, as my last piece as well is, this then all gives us a number and we're able to feed it into our systems then as well to say, hey, Renewal Probability is X. And our implementation team for Gains out of here, this is something that's gonna be brought up in a future call. We want to get that score fully set up.

And one of the reasons we want with Gainsight is exactly that we know it's something that be able to feed into this. So for us as well, the important part is AI, what all of this is important, but it's a trigger layer, not just a detection layer. So we can use AI a bit more to just say, hey, this is what's happening.

It can go, let's get involved. So for example, we could have a Renewal Probability Score trigger be feature adoption is dropped. The AI can then go, well, you set up your audience as the end user.

We need an in-app prompt to show them how similar companies are using that feature. So in the app itself, boom, there you go. This is your, we're obviously not saying you're using the tool less, but let's go, hey, we noticed that other customers like you find success at A, B, and C.

Would you like to see what it's like? We then have our, maybe there's a negative sentiment from a decision maker. Our audience there is the champion or our executive.

We could then have the AI go personalize email on ROI reporting and business impact of the tool. Deliver true email. We have the CSM is saying like our champion is doing what we call end of project, which is a polite way of saying the champion has left the business.

So they've moved on. So we need to go, okay, how do we present this tool to the successor? So creating an overview of here's all the wins they've had.

Here's everything we go with. And it's a mixture of an email flow or an in-app flow. Login frequency has gone below baseline.

We need to target our power users. AI does a re-engagement sequence for a quick win use samples delivered by email. And then I think the big one, if it says less than 50% chance, CSM gets an alert, they get a Slack notification with a full pre-built account brief and a draft outreach tailored to their industry.

It makes sure that the AI is not just a reporting layer for us. It is something which actually gets us to be something as a trigger and gets us right in front of those customers, going right back to that one full quarter where a customer is sitting there going, is this right for me before a human even gets involved? So if you wanted to do this and you said, right, you know what, I'm convinced.

And if you're sick of me talking about this on LinkedIn and you want me to move on to a new topic, this is how you can help me do that. Go to your teams on Monday, all your signal sources, map every system with customer health data and identify what you could automate and where qualitative context is being lost. Then define your renewal drivers.

So what specifically makes a customer more or less likely to renew? Build your inputs on this, not just generic templates. And you don't need fancy tooling to do this stage either.

You don't need to say, I need to invest a lot of time in Claude, I don't need to invest in here. You can literally do this in a Google Sheet. Then start with what it says is your highest risk cohort and reach out to them.

If there's anyone who's say below 50%, let's just talk to them. But the biggest thing with this is you never accept the score is a one and done. The score should be learning from your human conversations.

Hence why, for example, that video I showed you, our score resets on Sunday night. And it takes what the CSM has been inputting to make itself more advanced and to learn from the CSM's input. And then that's why it's important that you invest the time and enablement in your CSMs. Don't just say, here's our new probability score, go be more strategic. It's actually taking the time for them to understand, you are Sherlock Holmes in this instance, you investigate, you find out what's going on with your customers, and you help make this model be as strong as it can be. And if there was one thing I would train every CSM on, it's note taking. I know we have great tools that record all of our calls, we use them ourselves, but at the end of the day, I still want a CSM to go da-da-da-da-da, make their own notes and log it.

Make it part of the process, not an afterthought. So, if I could do the eulogy for health scoring. Health scoring, we barely knew you, but you were great at being able to tell us how a customer feels today.

Now want us to move to renewable probability scoring, and you will tell us where your customers are going to next, and most importantly, gives us a reason to have something that everyone in your company can be involved with and have a reason to help. And that takes me up, bang on, to time. Does anyone have any questions for me?

And thank you for your attention.

[Speaker 2]
Thank you, Sean. I can help reading them out so that you can start answering them. So, wonderful presentation, and we'll take your request, and I do the build work for them, so I'll take that personally.

So, who built the model? What's underneath it?

[Speaker 1]
So, I started building the model myself, but we actually do have our wonderful ops team as well who've been able to move it to advance. So, we have a data scientist, his name is William. Everyone should hire William for their companies.

Please don't steal him from us, but just find him and clone him. But essentially, William was able to go through and analyze all of our churn for the past year. And he would present, here is the moment a customer said they were going to cancel, and we would challenge him to go, great, now go three months before.

Tell us everything that customer was doing three months beforehand. What was their support tickets like? What parts of the tool were they using?

And just build from there. So, a combo of us leading the direction, but leaning into our ops team to help make the data be as beautiful as it could be.

[Speaker 2]
Gotcha. So, find William. That's the key message here.

[Speaker 1]
Don't poach William from me, please. Unless you have that DVD.

[Speaker 2]
So, next question for you, Sean. So, for long tail low touch accounts, how do you collect the qualitative human sentiment signals at scale?

[Speaker 1]
So, for that, we're doing kind of a combination of two bits. We are looking at how they engage with the rest of our ecosystem. Because I think sometimes we fall into a trap with scaled of, because there's not a human, it's going to be really hard for us to gauge things.

But our customers are still using us in different ways. It just might not be talking to a person. So, it's important for the scale team to go, and what we do, really advanced into going, how often are they going to our knowledge base?

How often are they going to our academy? What are their support tickets like? How much time do they spend in particular parts of the tool and going back and forth and trying to find those baselines?

That's how we find what we can assume is a qualitative human sentiment. But most importantly with that, that flies us accounts that our scale team should talk to. Because a scale team doesn't mean that you have CSMs who never talk to a human.

They just need to talk to the right human. So, they go in, they have a one and done call. But with that, they're able to validate, yes, this was the right signal for us.

It got us to something we can work with. Or, hey, you know what, these type of customers, they're actually fine. Not someone we need to spend time with.

[Speaker 2]
Yeah, that makes sense. And finding that right human might be a little difficult, but that's usually the right call to do. Perfect.

So, next question is, what signals did you expect to matter for renewal probability that turned out to be less useful than expected?

[Speaker 1]
Oh, that is a really good question. What did I expect to not be as useful as expected? Actually, funny enough, when we did this originally, we looked at the billing page.

I was like, if someone's going on to their billing page, that probably means they're trying to find out how to cancel, right? So, that was something we would have had as a higher weighting. But in actual fact, that by itself normally didn't lead to a cancellation.

It was actually an upsell. We'd find people would go on the billing more so to go, how do I expand? As opposed to, I'm going on the billing because I want to get rid of our tool.

Because if they wanted to cancel, either they were letting their CSM know, or they were going to someone else who was going to be in charge of the tooling piece. So, we thought it would be a big impact for us. Turned out not to be.

It was a great impact for our expansion numbers instead.

[Speaker 2]
Good. I'll move on swiftly with the next question, as we have roughly 10 minutes or so. And of course, any questions that you want to ask, find Sean and ask those questions towards the end.

So, what did you want to ask us for your children? Let's not forget.

[Speaker 1]
I like how this is like the highest amount of votes. It's like, my kids will be great. We will do this at the end.

But my oldest daughter, Vivian, has this big thing where she is clearly a public speaker. And she loves getting opportunities to talk. So, I always try to, at these events, get everyone saying, hello, Vivian.

So, it feels like she got to do it. And literally, she sees a podium and she wants to be at it. And she's been to events like this and has said, when's it my turn?

But my favorite one was she was at a funeral. And when the priest stopped talking, she literally, the whole room was quiet. And she went, when's it my turn to talk?

Not the time. But we actually had to bring her up to the podium and go, lean in. Thankfully, the body had left.

She leaned into the microphone, hi, everyone. I'm Vivian. There we go.

Good girl.

[Speaker 2]
Maybe I should get some tips from Vivian because I was in a nervous wreck when I started this conversation. So, I'll ask you later. So, how do you track human sentiment for long-tail customers?

[Speaker 1]
It's a lot of constant reporting of what exactly is the sentiment over time. So, if we have customers who are just in it for the long haul, we are making sure we establish a good baseline on sentiment. We have some where we would say, like, the sentiment is always a bit iffy.

But that's, I hate to do a generalization, that's just how they are. It might be the serial complainers that, you know, actually always want to renew. So, we just make sure whatever it is the baseline of that sentiment, that is what we're tracking.

And we always bring it back to our CSMs. We have our success planning sessions with them. And whenever they do a success plan, one of it is they have to record the sentiment score of the customer. We're tracking that.

And that success plan is tailored to the customer. Some customers might have, it's a 10-week long project, so that's how long their success plan is. The other 10 weeks, we do a new score.

It could be a 12-month long project they have to work with. So, we just continuously make sure we're getting towards their goal. But constant, constant tracking and beautiful data.

[Speaker 2]
Perfect. A long question. So, how did you go ahead, sorry, how did you go about feature selection?

And how did you weight them? For example, usage, NPS, et cetera, might look good. But on a sponsor level, a strategic choice is made to go for a build over buy.

[Speaker 1]
Preview tomorrow for the keynote where I'll be on stage talking about build versus buy. And exactly this topic. When we look at it, we look at what we see as the features that have the most amount of renewal successful associated with them.

And what does it fit in with that particular ICP. So, really breaking down and going, this ICP has value out of this, this, and this in general. This is why they buy Supermetrics.

So, we measure against those tools having a higher weighting than, say, a customer who would never use those features. And it's not something that they really want to go with. But also making sure that when we sell, we have our strategic buyer on, we have our decision maker.

And we understand you bought it because you want to accomplish X. How close are we to achieving X? And that's more what we're measuring.

The build versus buy piece, like I said, if anyone's around tomorrow, come along to the session tomorrow. But definitely, that's like an hour long answer in itself.

[Speaker 2]
Perfect. So, we'll go through them quickly. Basically, one of the biggest impacts to the quality of our renewal forecasting is both stakeholder change and or apathy.

So, lack of response. And not really understanding value in the eyes of the customer. How does the RPS accurately reflect that?

[Speaker 1]
Great question. So, how we tailor that is, sure enough, it goes back into our sentiment scoring and our multi-threading. So, what we will always do is we make sure that we understand that at each level, what exactly success looks like.

What is the measurement you're looking to put this against? And we track how we're doing against that. Consumption-based model is definitely going to make this so much easier.

It's why I was excited about this morning's keynote. I'm really excited about now in CS because of all this. Because it's tied to what a stakeholder wants.

But that whole piece of making sure we understand that customer's metrics, understand what good looks like to a customer, that's what we're tracking. And by the way, if you have a customer who when you meet them and say, what does success look like for you? And they go, I don't know.

Because they exist. And they're more common than I think we like to admit. Just ask them, well, in that case, when you have your performance review, what are you measured by?

You're measured by this, this, and this. Great. Let's figure out how we can help you towards that.

Make you look awesome in front of your boss, in front of your champions. And that's what we're looking to get. So if we can see we're trending towards their own goals, RPS is nice and high.

If we can see that we're not getting that way and it feels like there's more barriers, we have an alert going that we might need to meet with our decision maker to make sure that we're still on path for what's important today.

[Speaker 2]
Perfect. So next question is, is your finance team taking your GRR forecast as a target for next year?

[Speaker 1]
Yes. I got a Slack confirmation today saying yes.

[Speaker 2]
Nice and short. And then do you use gainsight CS or staircase? I can answer that.

Yes. If yes, is this data surfaced somewhere?

[Speaker 1]
I'm curious about is this data surfaced somewhere because we would use a lot of different data points and we use a lot of places and this is where we're analyzing everything. And a lot of it goes into, you're sick of hearing this, our BigQuery instance. But what we do with everything is we are adamant of that's not where CSMs live.

Our CSMs live in our CSP or CRM. So everything we do needs to make sure it feeds into there. So things might surface elsewhere but the data is passed on to go, this is where your source of truth is.

And our goal has always been gainsight is your source of truth as a CSM.

[Speaker 2]
Yeah. And hopefully we're doing a great job with that.

[Speaker 1]
Yes.

[Speaker 2]
Next question, we've got time for a couple more. How has your RPS actually compared to real renewal rates? How predictable was it?

[Speaker 1]
We went up by 80% this year. That's it. It works.

[Speaker 2]
Perfect. How many different data points did you use to build this model?

[Speaker 1]
I heard someone say like eight. Yes, eight because some of our numbers were not great with some of our teams. So we were able to bring it up to go, yes, it was quite a big jump.

There were some teams where the renewal rate was around the 70s and we had gotten them closer to the 90.

[Speaker 2]
Perfect. I'll start that question again. So how many different data points did you use to build this model?

Do you have a single source of truth that feeds into this? I'd imagine there is a lot of data needed to get this set up.

[Speaker 1]
Yeah, we have a lot of data points. We're a data company at the end of the day, so we hire data people and that's what we obsess over. So there's a lot of places we get to.

Like I say, we take everything but into BigQuery as where all of our data should point towards. But you don't need to have a huge scope of data points. You can start off with something relatively simple.

Like I've seen some companies roll out a renewal probability score, and their data points might be like five pieces that they're really looking at, but they still see immediate impact. They're still seeing themselves get in front of the right customers. So if you want to go super complex like we have and have about 100 data points, go for it.

Don't start off with it, though. Have that as an ambitious goal or a big, hairy, audacious goal for ourselves.

[Speaker 2]
Perfect. Where was the model built? Is it through Claude or any existing CSM platforms like Gainside, Planhead?

[Speaker 1]
The first time we'd done this was in a Google Sheet. Very ugly. Took a lot of time.

The one you saw there was built with Claude. But it's, as I said, designed to go. It needs to feed into a CSP.

It needs to feed into a CRM. You would have saw references in there to Salesforce, for example, because that's getting it in front of our sales team so that you can see these details as well.

[Speaker 2]
Let's squeeze in a few more. Have you considered how to tie CSM incentives into RPS versus only incentivizing NRR, et cetera? If so, would love to hear your thoughts.

[Speaker 1]
Thanks, Janet. I do actually think about that from time to time. It's something that we haven't fully gotten in yet.

I would actually love to use this as a way to measure the quality of input data from a CSM. So it could be, for example, if a CTA gets appeared in Gainside, how quickly did it meet the deadlines we had set up of you need to action this within 48 hours? How well are they doing that?

That's where I see it towards. But I don't know if I want to tie that towards a financial incentive just yet. I think that's more of a you're going to get promoted to senior style incentive.

[Speaker 2]
We'll do one last question. Given this is like a regression model and non-technical wouldn't be able to audit signals or feature leakage like a DS would, how do you handle model degradation and knowing when the probability result becomes less trustworthy?

[Speaker 1]
Yeah, it's why I always set out the most important thing here is the CSM's input. It's why for our own weighting, 60% of the weighting comes from what the CSM is actually logging and putting towards that sentiment score. If we just have it as something that's just purely the product, it's not going to work.

Then you're just back to health scoring with a new name. We need to be able to go, how do we make sure that it's learning what the CSM has said? Like I said, my model runs every Sunday.

It updates based on what we see in the sentiment.

[Speaker 2]
Perfect. I do sadly have to wrap it up here. I know there are a few more questions just because we are approaching time.

I learned a lot, Sean, to just look beyond. Anything that you've just said is very useful for my own customers. I hope it was a very productive session for all of you.

Any questions, like I said, you can ask Sean during break or anytime you catch him. I know he's a very busy man. Also, join your session tomorrow, right?

The keynote. Feel free to join. Thank you so much for being a lovely audience.

Let's give Sean a very, very big round of applause.

[Speaker 1]
Thank you.