The Health Score Revamp Nobody Asked For (But Everyone Needed)

[Speaker 1]
Hello, everyone. I'm Angela. I'm accompanied today with Megan.

First off, a big round of applause for Megan. It's her first Pulse and her first time speaking at Pulse. So, big round of applause for her.

I am the Senior CS Systems Manager and Megan is the Team Leader of the Customer Success Department. And today, as it was mentioned, we are going to talk about the health score. So, before we start, I'd like to actually do a brief mention.

Yesterday, in this same room, actually, I joined a session by Sean. It was a really good session. Raise your hands if you saw yesterday's session about the health score.

Yes. So, if you are here, you heard that in this same room, Sean put the health score to die, and saying that basically health score is dead. I'd like to offer a different perspective today.

I'd like to pick that health score back up and give it some love because I feel like the health score in itself, it was never meant to be a predictive model. So, the health score is not, by its nature, is not supposed to be predictive. It's supposed to give us a check-in of what the situation is right now and what we should be focusing on right now.

So, it still has its place, I believe, in the operations, but if we then put it together with all the other metrics, then it's going to give us the big overview, the big picture. So, yeah. Today, we are going to go through our story of how we changed the health score.

We used to have a health score. We will see what were the gaps in there. We will see how the new health score looks like and how we got there.

The lessons that we learned along the way. And then Megan will talk about how the team is adopting the health score, all the strategies that we put in place so that the team adopts the health score, what were the wins, the success stories, and what are the next ones, and then the key takeaways that we got, and, of course, your questions. So, first, to give you a little bit of context, who are we?

We represent today Shiji. Shiji is a company that works in the hospitality industry. They produce software and hardware for the day-to-day operations of hotels and restaurants.

With the health score today, we will be focusing on one of the products of Shiji, which is ReviewPro Reputation. ReviewPro Reputation is a guest experience platform for hotels and restaurants specifically. It's basically a platform where hotels and restaurants can see the reviews and what the guests are saying about their stay.

And there are all the modules that help them actually improve the guest experience. Both Shiji and ReviewPro Reputation share a mission, which is to help their customers provide the best, most amazing and outstanding guest experience there is. So, now, with the context, let's answer the question, why is it that nobody wanted a new health score?

So, we had one health score, right? And the good thing about that health score was that it was always green. And on paper, green was good.

On paper, those numbers were very reassuring. They were very good. Then, because it was the same for many, many years, the CSMs had gotten very familiar with it.

They knew how to use it. They knew how to read it. It also meant that, because we had a health score, then we also knew what a healthy customer was.

We didn't need to sit down and ask the question again. And then, in its origins, it was very simple, and it was built for everyone, which, in the long term, that's going to be one of its weaknesses, as we will see later. So, just to show you how we got to the first health score, like a little bit of history.

So, when ReviewPro Reputation started, we were a one-product startup, and at that point, we had no health score. Then, a new product was added to the family, that was the surveys module, and that's when sales went out and wildly tried to sell it, big style, to all the customers that we had. And that is when the first reality check came in, because sales was trying to sell the product, but when they were reaching out to customers, they found out that the customers didn't know what ReviewPro Reputation was.

They didn't know how to use the tool, and they wanted to churn. That's when the first CS team actually came on board, with the mission, of course, to reduce churn and bring an increase that value realization. But because both the CS team and the sales team were going in blind, we needed something fast that would help them identify risk, so for the CSM team to identify where they need to go and help those customers reduce that churn and get that value realization, and for the sales team to understand where those opportunities were lying.

So, that's how the first prototype of the health score was built. Because it was an immediate need, it was built with something that we already had available. At that time, it was the touch points, so the volume of touch points, the type of touch points, support tickets, and then later came usage, that back then was only page visits, NPS and Coolio cases, which is what internally we call the success stories of the customers.

So, the first original health score was built for everybody. It was simple, it was straightforward, but it was because it was resolving an immediate need. Then, the strategy, and by the way, that health score was used for years.

And then, internally, the strategy changed. When we tried to look more at retention and at value realization, of course, we looked back at the health score. Tell us, where is it that we need to focus on?

And that is when we realized that customers had very strong health score numbers, but they were turning anyways. So, we had customers that had 80, 90, 100 health score, green numbers, and they were leaving us. So, we went back to the lab with three main questions.

First one was, why the metrics that we have right now, why are they failing us? The second one, what is it that we can add on top of those metrics to capture the whole picture? And the third one, what can we learn from the others around us, everybody in the business, everybody who already had a health score and went through the journey of tweaking it, improving it, what are the current best practices for health scores?

So, first questions, why old metrics aren't working anymore? Well, when we looked at them a little bit more closely, we realized that 50% of health score contributions was coming from CSM or internal work. We are talking about the number of touchpoints that CSMs were making, the cooler cases that they were logging, and how many success plans they were closing.

Then, the usage was there, but we were only looking at page visits. That means that somebody that was logging in the application one day and they were going through the whole application, visiting all the pages, and then never returning to the application was still going to have great usage numbers. Another thing, the simplicity, remember?

So we had no segmentations and no time limits. The second question was, what are we missing to capture the whole picture? So, when we looked at usage, again, page visits, but that's not the whole picture.

We also want to look at how frequently users are coming back. Are they coming back at all over a period of time? What is the depth of the adoption of the platform?

Are they using those features that keep them coming back to the platform? And is the platform or the application ingrained in their operations? We had NPS, but we were also sending out other surveys, but we were not capturing them.

So, for example, we were sending out surveys at onboarding phase. How is your onboarding going? We were sending out implementations surveys, and those were still giving us a piece of the customer journey.

How was it going? But we were not capturing them. We were not looking at upsells, a signal that customers wanted to stay with us and wanted to expand the partnership with us.

And we were not looking at customer goals. Were the customers actually getting, were we helping the customer get those business outcomes? Were they getting value out of what we were offering them?

And then we looked around. What were, when we did the analysis, what were the current best practices? How should we build a great health score?

The recommendations there were we should be looking at four areas. The first one is product adoption. So, are users actually using the tool?

Then, business outcomes. Are they getting value out of it? Relationship health.

Do they trust us? What is the health of the relationship that we have together? And commercial signals, as in, are they renewing?

And are they upselling with us? Another mention of best practice is that a health score should be put into a framework. In this case, it was the DEAR framework.

DEAR standing for Deployment, Engagement, Adoption and Return of Investment. Why? Because in this way, so deployment as in onboarding phase and the return of investment as value realization, if you put the health score and the metrics into a framework that naturally replicates the natural cycle of the customer, then you will know exactly where is the friction.

Are they having issues at onboarding phase? Are they having issues at adoption phase? And of course, use segmentations because the same metrics, so your user base, depending on size, maturity, how they behave, they will, again, behave differently.

So the same metrics that you can apply to small customers, for example, will not be the same that you can apply to the enterprise customers. So based on all this analysis, what is it that we changed in health score 2.0? So we applied the DEAR framework and when we rebuilt health score 2.0, we built it around four main questions. So are they set up? We wanted to know out of the health score. Are they set up?

Are they engaged? Are they using the product? And are they getting value?

Then beyond visit, we wanted to look further. So what is their usage habit? What are they saying in all the surveys that we are sending out?

Not only NPS. We divided the health score into segments and not only that, but all the metrics or wherever possible, we are comparing the average of the, so one customer is always compared with the average of the segmentation it belongs to, to understand, compared to their peers, how are they doing? And by segment, but by segment also means that every segment has its own metrics and different benchmarks.

We put a time limit on signals because in the health score, the original one, a survey that was given two years ago was still impacting the health score today. But that doesn't make sense because if I give a survey two years ago, I might have a different opinion today. So what we did is that we put time limits on any kind of old activity.

And one big improvement that we added into the health score was that every single metric and every single score that's given is explained. So whomever is going to come and look at the health score today, CSM, whomever, can see precisely why a score was given and how is the metric calculated. So I'll show you now how the health score looks like today.

As you can see, we have scorecards by segment. This one, for example, is the enterprise. So every segmentation has its own scorecard.

And the measurements are divided into the DR framework. I'll walk you quickly through every aspect. So all the DR, quickly through the DR framework and all the metrics that are there.

So for example, this is deployment. Here we are looking at, again, are the customers set up? So we are looking at onboarding completion, close success plans, and average time of completion.

Then we are looking at the survey. So onboarding calls, implementation. So here we are looking at two aspects of the deployment.

So are they set up? And what is the customer experience of the deployment phase? With engagement, we are looking at the quality of the relationship between the CSMs and the company and the customers.

So here we will see how often are we engaging with them? What is the quality? What is the sentiment of the calls?

Are we having any face-to-face meeting with the customers? What is the support tickets? Are there any support tickets Are they attending to the live training sessions?

What is the NPS? But as you can see, the NPS is not the only one. It's in a context.

So for example, briefly here, if I could see the engagement, we can see that yes, here we are engaging with them a lot. There is a huge volume, but also there are issues in the tickets. So I might want to go and check what is the issue there?

In adoption, we are looking at the usage and the growth of adoption. We are looking at the sticky features, which again I was mentioning are those features that keep the customers coming back. Are they using them?

Are they setting them up? And for usage and healthy users, for the healthy users, we are looking at the growth of adoption, how often they are coming back. So we have included where before not available and it's no longer only page visits.

For return of investment, we want to focus on are the customers getting value out of this? So we've got the cool cases, which is what we call the success stories. Upsells and customer goals.

So are they actually getting those business outcomes? Are they getting value out of this? The big change, as I was mentioning, now every component is explainable.

So this is exactly what the CSMs see when they go and check. So for example, for usage, here I can see that the explanation is that well, compared to its segmentation, this specific company actually has a ratio of 1.6, which means that it is actually doing better compared to the peers and every measure has also a description of precisely how it is being calculated. So what did we learn rebuilding the whole health score?

A busy CSM can have a big volume of internal work and still have unhealthy customers. Internal CSM work does not necessarily equal to customer health. Page visits only tell a part of the story.

We need to go a little deeper. So consider frequency and depth of adoption, depth of use. Are they returning?

Are they frequent users? Customer segmentation areas and data gives the score context. So because we are using segmentation, now every metric is in its own context and it's relatable to the customer it is applied to.

And also because we are using defined time windows, as I was mentioning before, we make sure that these scores are as accurate as possible. And a useful score helps us decide what to do next. So the CSMs need to understand what does this score mean?

Why was it given? Where is it coming from? Because once they understand that, they can go there and say, oh, I understand what the issue is, I need to go and check specifically this thing.

But this is just part of the story. So this is how we build the new health score. Now Megan is going to explain to you how did we make it so that the people adopted the health score for CSMs. Perfect.

[Speaker 2]
Thank you very much, Angela. So yeah, as she just explained, this is how we build the health score. So I'm going to tell you the other part of the story.

How did we actually incorporate that or made it into a daily habit for our CSM. That's me. Okay, so of course, the first thing that we need to do is to build something in the back, but now it's time to explain it to them.

So align with the team. The first thing we had to do is explain the why. So everything that Angela just explained, we gave it to the team.

Okay, the number were good, clients looked healthy, however, they were still churning. So we had to explain to them that we needed to incorporate more elements into the health score in order to reflect and build trust. In order for them to act on it, in order for them to trust it, they need to understand it.

So what did we do? We put the person who built it in front of them, right? So create internal documentation explaining how we are calculating the weighting, the segments, that average that Angela was just explaining based on the different touch points, face-to-face and so on.

So explain them so that they can trust it. And the last thing that we have done also, well, also in the trust, so the scorecard, of course, in Gainsight, everything that she just saw is very well detailed, so they have access to that information, they can read it, they can really understand each of the metrics, so that was also very key. And in order to drive adoption, we made it a KPI.

So this was new for the year. We made it a personal KPI on their portfolio, and we also made it a departmental KPI. Good.

Okay, so it was a little bit bumpy at the beginning, if I may say. It's all new. There was a lot of questions.

It was also new for us, for the leadership team, so okay, some of the score went down, they had questions, why, how do I do? Okay, let's look at it together. No, this is new for you, this is also new for us.

Angela was, of course, here to support, running more Q&A sessions, like really understand it, but yeah, having that attitude of let's do it together, because we have that as a KPI, as a department, right? So once we have it kind of understood, how did we make it really, how did we make the CSM to really adopt it, right? In order for them to adopt it, you need to make it visible.

Gainsight is one of the tools that they opened, right, at the beginning of the day, this is something that they check, so it was very important to make it there, visible for them, right? So we rearranged a little bit the homepage, right? They had access to the health score of their portfolio, of each account, right?

We added an evolution chart to say how's that evolution since last year, since the past few months, so they can see, they can see the signal, the risk, no? Which clients do I need to engage, where should my focus go, right? And I think by understanding that, so let's focus on one client at a time, one thing at a time, having those small improvements will just, well yeah, drive that health score of their own portfolio up.

So that health score became everyone's goal, right? And this is something that we saw as a team, so each CSM focusing on their portfolio, one client at a time, made their own health score go up, and this had an impact on our departmental health score, right? We also made that visible for the team, right?

So on the monthly KPI, on the monthly KPI presentation that we have, we started showing the evolution, it's going up, okay? We also decided to spotlight some accounts that were experiencing a good growth, right? And that created between the CSM synergy, no?

So hey, how did you do that? How did you identify an opportunity here? And by sharing those best practices between them, they were able to each go to their own objectives, and I think that's something that we really saw as a success from the leadership team, that they were helping each other, and that had a positive impact on our department as a whole.

I'm going to share with you a client success story. So one of my team members, Paulina, was assigned a new account that had, let's say, a relatively low health score, right? So of course she saw that engagement was needed.

But what should be the focus? Okay, engagement is one thing, but how do we engage the right thing in order to, well, in the end this is what we want to do, no? Get that customer value.

So okay, she went to analyze the account, what it is. She saw that there was opportunity, or let's say, a need for better configuration, the sticky feature that Angela mentioned. She spotted that this was not in place, right?

So she thought, okay, maybe this is something that I can explore with the client. Through her first call, she discussed with the client, remember, we have a tool that helps hotels in their operation, right? And one of that is managing their online reputation, and that one metric that, for example, our clients might monitor is their management response, right?

So Paulina asked them, hey, how do you manage that? Do you have processes? Do you have a goal for that?

And this is when the client said, well, actually, yes, we do, but we might, well, need your help with that, because in the end, we offer the right tool for that. So what did we do? What did Paulina do?

She, together with the client, they defined a goal. And if you have the right user to use that, it's going to help you achieve your goal or improving your metrics, which for us would be seeing the customer value of the tool, right? So after those integration, after product training, after a few more calls, the numbers started to grow.

The health score of that account started to grow, which was a positive impact, right? And at that moment, we started to upsell. We also have a product, a functionality that leverages AI for our clients to use to manage that online reputation.

So all of that turned into an upsell opportunity that was close one. So overall, that return on investment part of the health score, I think was well, let's say, achieved, and I think the results from May to July, the health score went from 39 to 88, which is a 126% increase. So shout out to my pauline, she did good.

So yeah, that's just one example. We did have different kind of success stories like that. So yeah, I think this was showing the positive impact and adoption of the team on really understanding that health score, how do they apply it in order to raise what else did we, let's say, gain, what else can we take away from that health score, and what's, let's say, what's next, what's ahead of us.

I would say two things. The mindset shift from the team. I think from the leadership perspective, again, we really supported them in understanding that health score, in going into understanding which client is at risk, where can you engage from the health score and reduce the churn, right?

In the end, that proactive engagement, this is the core of what we do as customer success, ensure product adoption, usage, so that the customer gets the value of our tool in order to reduce the churn. So we really saw that shifting, changing within the team. And yeah, I think this is something we are quite proud of.

And slowly this year, our health score on a departmental level grew a little bit, but still I think we should celebrate the small win. It's going up. We have a goal to achieve by the end of the year.

We have a few more months, but I'm sure we're going to get there. And what's the focus for next year? Different things here.

Sorry. As Angela was explaining also, well, we have different segments, right? We still notice that our small and independent segments were the ones that were still churning the most, right?

So what did we do? Through a business case, we created a new role, a client service manager, which will be focusing on these segments with the main goal of improving that health score and reducing churn for those small segments. Another thing that we are currently working on also, so we mentioned the importance of the segmentation, right?

So we noticed throughout this year that with feedback from the CSM and with our own analysis that one segment was still a little bit bumpy and was not again reflecting a reality. So we sent it back to the lab again with Angela. It's been a work in progress.

So we hope that by next year we will have also a bit more sorted because it's a segment that is important to us still. So that's also going on. And yeah, I think there's many things to explore.

What if we use that health score beyond customer success also? I don't know if there is a possibility to share it with pre-sales. I think there are different things that we can think about.

But that's a little bit what has been changed. So what should you take home? Check whether healthy matches reality.

Yeah, question your green again. As Angela explained, it was green, it was good, but it is not. So understand that if you still have high churn, then maybe you need to review that health score, right?

Which is what we did and what was much needed. Segments. Of course, probably in this room we have people from different industries with different business size, different kind of products.

So know your customers, segment them in the right way and that will allow you to achieve your goals, I would say. Make it explainable and actionable. Again, transparency.

If your team needs to, if the CSM needs to act on it, they need to understand, they need to trust it and to trust it, they need to understand it. So I think that transparency, explaining how it's calculating the why behind is very important so that everyone can act in the right direction. Make it a daily habit.

Again, we link that to a KPI. This was the first year that we actually had that. So far we're seeing a quite positive impact.

So, yeah, if you want to make it a habit, make sure that it is also visible and that you perhaps tie it to a KPI so that the team really acts upon it. So now, we know that, well, after everything we explained, we went from 1.0 to 2.0, no? I think after today's session, after all the sessions over the past two days, we understand that we have a landscape that is constantly changing, evolving.

We have a product that keeps on changing. We have customer behavior, consumer behavior that is changing. So health code 3.0, maybe nobody will want it but maybe everybody will need it. Thank you very much.

[Speaker 3]
That was great. Thank you, ladies. All right, so some questions for the audience.

What happens when the score says the account is healthy but the CSM disagrees?

[Speaker 2]
Hmm, that's a good one. Well, this is, I think, something that we are sometimes still facing. I mean, again, I think that segmentation is important but we do face some situation where, I know, wait, sorry, let me read it again.

What happens when the score says an account is healthy but the CSM strongly disagrees? Hmm, that's a good one. I would say, well, understand why the CSM disagrees.

What does it make him, like, disagree? And if it says that it's healthy but the CSM believes there is an opportunity to, I don't know, to make it better because the client is unhappy, I guess they can still make that, make the steps or make the, whatever is needed that they believe will make the client healthy. Even though the score is there, I think it's also feedback that we can also take, that we can always share, understanding the why in those scenarios and they can take the right action that they believe will make the client healthy in reality, not just in the score and gain side, I guess.

[Speaker 3]
I'm going to, how much of the score is automated versus subjective to the CSM?

[Speaker 1]
So currently, the health score is 100% automated but we are looking at adding manual inputs where there are situations where, you know, based on the data points, the health is good but because the CSM had a very important touchpoint with the customer and they picked up that the situation is going bad but we are not seeing it in the numbers so that they can add it manually to say like, okay, this is actually bad.

[Speaker 3]
Just a gain side CSM perspective, one way that we handle that is through our risk process. So if I log a risk CTA, especially if I flag it, it's now a star but I still call it a flag, it would override the health score. So one way to handle it or other inputs.

All right, so do you evolve the scorecard based on lifecycle stage? So in your current model, do you evolve the health score positively, permanently? Should it?

Have you evaluated whether a positive onboarding loses influence over time? Something like that.

[Speaker 1]
So we actually have it. So the deployment phase will only be permanent during the deployment phase. So it will only impact during the onboarding phase.

So if you've seen like the example before that was an older company, that means that they were already deployed and their metrics were NA. That means that the onboarding will no longer impact them if they already went out of deployment.

[Speaker 3]
Awesome. So what are your CSM's KPIs that are linked to the health scores?

[Speaker 2]
Well, they have a portfolio KPIs that is related to their own portfolio. And yeah, that's just the one KPI. It's not only based on that.

Of course, there are different KPIs but part of, let's say, the strategic management of the account and so on, the health score is to improve. Let's say we looked at let's say the numbers, the evolution and we said, okay, increase by X number that portfolio health score by the end of the year. So that's the KPI.

[Speaker 3]
Is there one metric within your health score that is a greater predictor of churn than others or is it aggregate?

[Speaker 1]
I tell you why. It's because two reasons. Again, the health score is not predictive so it just gives you what is it right now, what is it that I'm supposed to focus on right now.

And the second is it depends on the context. So if you look at the health score you might have usage for one specific company that's really bad but in the context everything else is fine and then when you look at it it's because, or actually this company has more usage because they don't have that many users but actually everything else is fine. So we expect that.

While in another company where normally they would be really, really, really engaged and suddenly the usage is going down that could be a bad signal. So everything, all the metrics need to be read in a context so there is not one metric that predicts churn.

[Speaker 3]
All right, so the next two we can probably mark as answered. All right, so have you guys thought about how often you want to revisit or recalibrate the model? You sort of touched on it towards the end.

[Speaker 1]
Yes, so ideally yearly. Right now we just did it so we are waiting for the year to conclude but the idea is that always look back and see okay, so for example one thing that we already noticed about this new model that we sent out is that it tends to be really gray areas kind of measure so we already know that one improvement that we want to do is that to look more like at those gray areas to stabilize these metrics a little bit more but yeah, ideally I would say yearly.

[Speaker 3]
What is the portfolio size of your CSMs?

[Speaker 1]
Oh sorry, just to mention yearly or if there is a big change within the company so suddenly if you are launching out a lot of new products or the strategy changes or for any given reason the definition of a healthy customer changes within the company that should also be your hint to it's better to go and check the house code to make sure that we are aligned.

[Speaker 3]
To take it back to the lab.

[Speaker 1]
Yeah.

[Speaker 2]
What is your CSMs portfolio size? This I will answer. It depends but I would say on average is about 25.

[Speaker 3]
All right. Do you now see a better correlation between health score and retention?

[Speaker 2]
That's a good one. I think that I don't know if we have made a clear analysis yet but I guess this is going to be the whole point. I mean this is what we rebuild it so of course it's just been implemented this year so I think again as Angela said we would need to let's say complete the full year which is coming soon and maybe compare it to 25 and see actually if that had a positive impact so that's still a work in progress let's say.

[Speaker 1]
And right now what we are seeing is that although we don't have the retention yet what we are seeing is that compared to the previous health score it is we have a confirmation that it's actually reflecting reality so if a customer success manager goes and sees like the usage is low then there is a confirmation that indeed the usage is low and it's reflecting reality.

[Speaker 3]
Influence retention. Perfect. So for customers or companies that have multiple products upwards of 15 how would you approach that?

So updating the health score if you had to start from scratch with this where would you start?

[Speaker 1]
Okay so depending on your context but again what's really really important is to start from the basic. Again the health score should be a representation of ideally what does it mean for your customers to be healthy? What does it mean for them?

So if you're for example if you're adopting the DEAR framework the end goal is for your customers to get return of investment. So if you have multiple products and each of them has like do those product bring the users at the value realization or each one the product needs to be looked at separately? So I think it depends on the context but ideally it should always end up answering two questions which is are my customers healthy?

And therefore what does it mean for my company that a customer is healthy? And then ideally are they getting value out of tools? So out of all this product out of all the things that we are doing what does it mean for us that they are getting value out of it?

And then starting from those questions I think it's really important whenever we choose metrics that we don't start from the metrics we start from the questions. So what are those metrics meant to tell us? Which questions are they supposed to answer?

[Speaker 3]
You could probably also approach it from your highest ARR product or wherever you're having the most challenges or the most churn a lot of different places you can start but I really like starting with the questions versus the metrics themselves. I'm going to do one here. How do you prevent a health score from becoming too complicated over time?

I won't show you gain sites right now because it is.

[Speaker 1]
I think two things. Transparency. So it tends to be less complicated if people understand how are the metrics calculated and what is the magic behind the scene but also always pulling it back to again, what does it mean to be healthy?

Value realization. Not every metric actually tells you are the users going to succeed? Are they getting value out of the tool?

So focus on those first. Makes sense.

[Speaker 3]
Alright, last question. So when you incorporate support tickets into the health score is this because more support tickets means more likely to churn or less support tickets is more likely to churn?

[Speaker 1]
How do you think about support tickets and how do you define? So in the way that we deal with support tickets there are specific case reasons in the support tickets that tell us people are complaining here either because they are logging a bug so we are logging a Jira ticket for that or there is a cancellation request or they are complaining in some way. So we have all these data points that tell us these specific tickets are actually the ones that are causing trouble or that there is something going on here and you should be checking.

[Speaker 3]
Yeah, versus volume or what the tickets are about. It's about what's going on in there. That makes a lot of sense.

Well thank you so much. I really appreciate it. You guys did a wonderful job.

Let's give them a round of applause. Thank you.