The New Math of Community: Proving Value When AI Changes The Rules

[Speaker 1]
So I'm going to talk to you today about how we measure a community and particularly how we're thinking about it today in the current era where everything is going a little bit crazy. But before I do that, I want to talk to you about my very first real job. So my very first real job was working in public relations.

I was an intern at a PR firm. And how many of you worked in public relations? Okay, I feel sorry for you.

But public relations is very interesting in that like many fields, there's a hierarchy to it. At the very, very top, you have the creative director. They will meet with the client and decide what the key messages we're going to get out there, what publications we're going to be looking at, what is the branding and the theming that we need.

Below them, you have like the planner. So that's going to decide, okay, which publications do we target? What kind of messages and which publications make sense?

And then way at the very bottom was me. And my job was essentially to get the press coverage for our clients. And that meant I had to write the press release, I had to have the client tell me why that press release was wrong, rewrite it as basically an advert, send it to a newspaper or a magazine.

And then I'd have to phone them up and be like, hey, did you get my press release? They'd be like, yeah. You're going to use it?

Maybe. Please can you use it? And that was kind of the job.

But what was really interesting about this is how they measured my results. Like how do you measure the work that I do? And what they did is whenever I got any coverage in a newspaper or magazine, they would clip that out.

Literally they'd get a copy of the magazine, they'd cut it out, they'd put it in a brand book, and then we'd go to meet the client maybe once a month, and then we'd go through it. Yep, we've got your coverage there, there, like a scrapbook, but for clients. And this was how it was measured.

One week, I didn't have time to do the cutting or requesting the magazines, and so instead of doing that, we just had all the publications at once. All the publications in a massive stack. And I remember bringing this big stack of publications in to meet the client, and he was like, wow, that's amazing.

That's so much bigger than last month. It was, it's just the entire newspaper instead of just that one clip we used to take. So then after that, we started thinking, well, let's just have the biggest magazines that we possibly can, have the biggest power we can, and that is how we are measuring results.

And we weren't alone, like press clippings was a very common way of measuring the effectiveness of any campaign in public relations at the time. And in fact, what's really fascinating about it is that this goes back to how PR was invented in the first place. Public relation, as you may or may not know, was generally invented by a guy named Edward Barnaes.

Edward Barnaes is someone who called himself the father of public relations, so you know he's good at his job. But what's really interesting about him is that he was a relative of Sigmund Freud. And Sigmund Freud, as you know, a lot of associations and psychological principles, and he was the first person that began using these principles in a PR campaign.

So in 1929, he has Lucky Strike as a client, the tobacco company. And at the time, Lucky Strike were doing tremendously well. However, for men, they were great.

For women, smoking was still seen as a taboo thing. So the very first thing he did is that he did a campaign. 1929, he did the Tortures of Freedom campaign, which is he paid attractive women during the St. Macy's Day Parade to walk down the parade and smoke. And this attracted a huge amount of attention, and he measured that by the number of press clippings that he received. And this was a common thing that he did again and again. A short while after that, he had to promote bacon.

He was hired by a company called Beechnut to promote bacon. So what he did, he went to a lot of doctors in the area, and he paid them to say that bacon was a healthy thing that you should have for breakfast every single day. The common thing you'll notice is that at the moment, this is borderline unethical, probably illegal today, but this is what he did, and it worked tremendously well.

Again, because the press clippings were really, really high. Or my favorite one. He was later hired by United Fruits, a Guatemalan organization to sell bananas.

And bananas at the time in the USA weren't that popular, and he decided the best way to sell as many bananas as you possibly can was to make people think of it as like an exotic fruit, to get them to know more about Guatemala. So he set up the Middle America Information Bureau. And now, this presented itself as a place where journalists could get independent information about what was happening in Guatemala and Panama.

In reality, it was just a front to sell as many bananas as possible. But again, the press clippings were really, really high. And this is how PR was measured for a really long time.

And the challenge was always, the more press clippings you got was good, low press clippings was very, very bad. But halfway through my internship at a PR firm, we decided to use a new technique, a technique that was called advertising value equivalent. Any of you heard of this?

Okay, AVE is a very simple thing. Essentially, you look at the size of the article that you got, that coverage that you achieved, you look at how much it would cost to advertise in that publication, and then based upon the size and the value, you calculate how much each article is worth. So if I get an article in the New York Times or a magazine or something like that, you'd estimate how much that was worth, and then you'd add them all up, and then we could tell clients, we got you 300,000 pounds in advertising or in public relations this month.

But what I noticed with this, every time we told a client this, they weren't that impressed. Even though we had a precise dollar metric that we could use, they never seemed that impressed. In fact, I think they missed the massive pile of publications that we used to show to them.

And the issue with all of these is that there's certain issues with how you calculate this. I mean, I spent so much time measuring these things and cutting them out and trying to estimate what the rate cards were and all that, but fundamentally, the issues are this. One, it confuses just exposure in a publication with the impact of it.

Two, it ignores if it's the right fit for the audience. You could have coverage in publications that are desperate for news, and then you could have some that aren't. Three, the message might not even be what you wanted to get across.

Or my favorite one, I had a client in the UK called the American Museum in Britain. And one week, they told me, this weekend, we're gonna host an experiment where we're gonna recreate Benjamin Franklin's flying a kite in a storm experiment to demonstrate electrical charge. Then they added, unless it is actually a storm, and then we're not gonna do it because it'd be very dangerous.

And I knew when I was writing that, that would be an awkward thing to say. And that's exactly what happened. A journalist did pick that up.

They wrote a negative article about my client. And I remember telling my boss, like, I think I screwed up here. And she said, well, just measure it the same way.

So even though the coverage was negative, we still counted that as part of the value that we gave to a client. So there's a whole range of issues like this. And the key point is that advertising value equivalent was never a really good fit.

No one even paid these rates in the first place. This all changed in 2010. In 2010, a lot of the top PR representatives in the world went to Barcelona to participate in the, let me get this right, the Association of Measurement of Electronic Communications, a very unique niche association, I think.

But they decided, we are not gonna use this anymore. We know that this method is bunk. We are not gonna use the advertising value equivalent ever again.

And they stopped using it. And gradually over time, PR firms stopped using it as well. But at the time, there was outrage about this.

Organizations were very upset. They were worried they weren't gonna get as much budget anymore. They were worried that if they couldn't show an exact, precise dollar value of what their work was worth, the entire profession would collapse.

And do you know what happened? It didn't collapse. In fact, what happened, if you look at the revenue of public relations firms since, it's skyrocketed since.

I can't say this is because of it, but it certainly hasn't hurt it. And I think the key lesson of all of this, and I think one of the key themes I wanna get across today is that bad metrics are worse than no metrics at all. And I think for a very long time in the community space and how we measure a community, we've been using metrics that look very precise and they seem like they are intuitive and they make sense.

But I think they've been bad metrics and there's an opportunity, I think right now, to do something much, much better. So let me give you an example. How many of you use community to measure retention?

Just one person. If you can raise your hands high, that would really help in this kind of question. Thank you, thank you.

Okay, hardly anyone, interesting. So the challenge with retention is that it's very hard to prove the community caused the retention, right? You can always prove that people in a community do more than people not in a community, but it's very hard to prove that was the community itself that caused that thing.

And time and time again, I had this argument with so many people, both online, offline, elsewhere. I'm a lot of fun. But what you realize is that if you like the product and you're satisfied with it, yes, your level of engagement is going to increase.

And yes, you're gonna have a high level retention. But most of the time, we're just measuring this, the correlation between these two things and pretending it is a causational relationship. And very often it isn't.

And I think we like this because it gives us a number that's really, really high and we like telling our boss and CFOs look at how high the number is. But if the number is invalid, if it's a bad metric, it doesn't really matter. And also with retention, with retention especially, there's a lot of issues with it.

Whenever you try to get into the weeds of how you measure that retention, you end up with so many challenges to resolve. So for example, if you want to prove that community members do more than something else, renew more, the very first challenge you'll have is that the people that participate in a community often aren't the people that make the decision. They aren't the decision makers of that community.

And so time and time again, you've got to try and match these accounts up but often you get match rates of like 28% and it becomes a real challenge. So the first thing that you do is that you realize, oh, we need to refine this question. So you get something a lot more precise the second time, which is something like along the lines of, which I've seen here this week, do organizations with a community member renew more than organizations without one?

This is a question I've seen over and over again. What's the problem with this? Take a second just to think about it.

What if that person is registered in the community and they haven't visited in a year? What if they had never participated? What if their only post is being of a negative one?

It doesn't account for any of that. So you realize that, oh, we can't just count the registrations, we have to count some measure of engagement activity. And then you go even further to a question that looks like this, which is do organizations with at least one member who has visited the community more than three times in the past 12 months have a higher probability of renewing their subscription, their annual subscription than those that don't?

So you end up with this really narrow, precise question, which seems like it's legit. And it's better than nothing for sure. It's better than a lot of what we've got at the moment.

But when you begin to think about it, even a question as specific as this, which you can measure, isn't that accurate at all. Because it doesn't tell you what you actually think it does. Because the problem is how you define the question decides whether you can prove retention or not.

In 2018, there was an experiment I absolutely love, where they got 29 data teams together, and they asked one question, which is do, let me try and phrase it the right way, do dark-skinned soccer players get more red cards than light-skinned ones? Now, I don't wanna go into the social aspects of that question, which are obviously important, but how do you even begin to define that? They had 29 different teams that had the same question, and the answers ranged all over the place.

Some said it was a lot more, some said it was a little more, some said it was about the same, and one said it was slightly less. Because how you define the question completely changes the outcome. If we look at retention, for example, why just three times?

Why not once, two, five? We can completely change that. It's an arbitrary measure that we're using.

And so how you define a question completely changes the level of retention that you're going to show. You can double or triple the result just by changing how you define the question itself. So, I think right now, community as a profession, we're having that Barcelona situation, where we're starting to realize that a lot of the metrics and methods we've been using are nowhere near appropriate for what we want to show.

I'll give you a common one. Core deflection. How many of you measure core deflection?

Okay, some of you might hate me by then, just talk. So, core deflection, there's various ways you can calculate it, but essentially it's some combination of this, which you look at the number of visitors to questions that are being answered in your community. You look at the percentage who said it solved their question, usually by a pop-up poll.

Then you multiply that by the average cost of a support call or ticket, and then you get an answer. Do any of you have a formula that's like this? I love how now you're just like, yeah, maybe, I'm not going to raise my hand now.

So, here's a challenge with this. Oh yeah, and you can make it more complicated as well, with would have contacted support otherwise, or out of warranty. You can make it more complicated if you want.

The challenge with this, that yes, you can get a formula, which looks quite precise, which looks like a legit formula that you should be using, but the challenge is often here. We did an experiment a while ago, where if it's a pop-up response, did it solve your question or not? What happens if we delay how long it takes for that pop-up to appear?

And what we found is a curvy linear relationship, which is if it pops up straight away, then the negatives are a lot higher. If you delay it to around 12 to 17 seconds, then the positives are a lot higher. And we found just by changing how long it takes the pop-up to appear, we can double or triple your call deflection rate.

So, how legit can this actually be? And this has been the challenge for a long, long time, but I think the only benefit is we've been kind of aware of this challenge, been kind of aware that it exists. But now the challenge is here.

Now the challenge is how many people are actually visiting an answer question in our community? And this is going down, and it's going up at the same time for reasons that are going to annoy us like crazy. The first challenge is this.

How many of you are seeing engagement decline in your community over the last year or so? Quick show of hands, I won't insult you this time. Thank you.

So, generally what we're seeing is because Google have changed how they show the search results, your answers used to appear here, and now they appear here. And this doesn't seem like a big deal unless you do what we do for a living, and then it's a monumental thing that has had a huge impact on the level of engagement. We've been tracking this for a number of organizations, and what we're seeing, a lot of communities have seen a decline over the last couple of years of anything between 40 to 90%.

40 to 90% of engagement. Think about how that changes everything, how that changes that core deflection method that you're using. And also, they're still getting the answer from the community itself.

The value is still there. Your community is still delivering them the value that they want, but they don't have to visit the community to get it. So how do you begin to measure that?

Another challenge are bots. What we've seen a lot over the last couple of years of a lot of our clients is that human traffic is doing okay, going down a little. Declared bots like ChatGPT, Claude, and the like are going up.

And then you get these folks. These are the dark bots. These are the bots that pretend to be human to extract something of use from your community.

And this is an issue for two reasons. One, you can be charged for it. I'm sure that GainSight doesn't, but if you're worried, then Julian is right there.

I'm just kidding. But generally, you have to be very careful you're not paying for this. But what we're seeing more and more is that dark bots are representing a higher proportion of traffic.

And very often, they're getting ingested into the calculations that we make. So we want to make sure that's not occurring here. But we've seen this with Microsoft for a while.

We did a lot of work on how they measure a community. Again, what you're seeing, they have, I think, a huge decline in the number of participations. But the number of people that are consuming rich snippets are really, really high.

So this is having an absolutely huge impact on any calculation we make on call deflection. This is a staggering decline to think about. And when you go from, say, 1.2 million to 250,000, it completely changes the value of the community, even though it's still as valuable as it was before. So we have a real big issue with call deflection at the moment. And I think we have to be prepared for this Google zero world, which is, well, what happens if we get no traffic from search? What happens if our community just gets displayed on search engines and that's it, and we don't have any visibility?

We don't have any data on any of that? I was telling someone this a couple of months ago, and they said, that will never happen. Google needs us.

Google would not do that to us. Google likes us. Let me share this.

This is my website, Feverbee, September 2021, and had 15,700 clicks that month. What do you think I got last month? Closest answer wins my respect.

You can guess. One and a half K? Anyone else?

Three K? You're gonna have to shout really loud at the same time. Just kidding.

Someone at the back? No? Okay.

Google does not need us. Or it does, it just doesn't care. That's a 96% drop in search traffic, and I'm still doing exactly the same things.

I'm still writing up, perhaps I shouldn't, but I'm still doing exactly the same things as I was before, right? But this is what we're seeing. If it happens to me, it could happen to you, it could happen to anyone else.

So search traffic is in a huge decline right now, and I actually think this is a good thing. I actually think we benefit from this, because I think we have this opportunity right now today to get rid of all the bad metrics we've been using for so long, and finally start using some metrics that are reliable. Finally, we can get rid of this illusory pursuit of more precision, because for years and years, we've had more data, more data, we've been integrating our databases more and more to try and make it more accurate, more accurate, and I think we've been getting further and further away from the truth.

So what I want to do is go through a couple of methods that I think work well now, and things that we've tested with many of our clients, and then some of the things that you could be thinking about as well. So the very first thing is that you don't need a bigger number. When we measure the value of a community, for the best part is to get more support, to get more internal support, more budget, or just justify the budget that we have.

And the temptation, and the reason why these methods have persisted for so long, is that we think the bigger the number is, and these numbers give us big numbers, the more budget we're going to get, the more respect we're going to get. I don't think we need a bigger number. We need a more trusted number, a number we can rely upon.

A few years ago, I sat next to a CFO on a train of like, let's say a mid-tier European firm. Some of you might have heard of it, but I won't say the name. And I asked him like this question, like, how do I actually get more support for community?

And I asked him a bunch of questions, and he said something really interesting, which is, the size of the number doesn't matter. What does matter is, can you trust that number? And there's actually a lot of data on this as well.

There's studies that show what CFOs support and what they don't support. And you can go through each of these. And time and time again, what comes up is uncertainty in the number, or if the projections seem overly optimistic, that's a bad thing.

And we've tried so hard to have these numbers that are really, really high, when really what we need are numbers that, A, we can understand how the calculation makes sense, it's not so complex, and it doesn't have to be precise. It just has to be a very strong indicator that, oh, if we support this, this is where we'll be. And so I'm gonna go through a couple of methods that I think work, make sense, are easy to use, and I think we should be considering as well.

And some of you will agree, some won't, that's great, but let's go through them. The first method that I think all of us should be using in some form is the ask method. This is very simply where we ask people.

This is a method that gets a lot of criticism, but it's widely used pretty much everywhere. We all use some measure of this. For example, if you add, where did you hear about us, to your other fields, and add a link for community, it feeds right into the attribute of what you have.

This is the most basic thing you can do. But there's actually more advanced versions of this. And for all the criticism it gets, it's still widely used pretty much everywhere.

We've all seen the net promoter score, we've all seen the CSAT score, and the recall rate as well. These are all methods that we can try and adapt for our situation, but I do think there's ways of doing this good and bad. And what we found when we do it for clients is that we've used methods like this to prove the things we really wanted to show.

What happens too often is that when we ask members to poll or CSAT score a net promoter, we give one metric, there's one baseline, and that's it. And we say, well, that's because of the community. The problem with this is that, let's say you're measuring customer satisfaction.

We had a client, I think 2018, 2019, and what they found is that of all the functions in the organization, with the documentation, knowledge, support tickets, the community had the lowest CSAT of anything by a long way. And we couldn't work out why until we phoned up the customer support and pretended to be a customer to see what would happen. And what we found is that if the customer support line couldn't solve what the problem was, they'd tell those people to ask the community instead.

So what's happening is the community is this net which catches the most angry customers you can possibly have. But if you only measure that one thing, then the CSAT number is what it will show. So here's what we need to do.

All of us need to be doing this, by the way. Measure people when they join the community, measure them a year after. When they join the community, a year later.

And compare it to the people that didn't join. This is known as a difference in differences approach. Very simple, very easy.

It'll give you some idea of what the difference is you can attribute to the community. If you do absolutely nothing else, you do this, you'll be doing better than most. And you can even do it in a whole range of things.

We worked with a client, I think 2022, and had a survey when they joined, a survey a year later, and looked at the differences in their attitudes in a whole range of things. And we found out, oh, there's actually a lot that's going on here. There's a lot of things we otherwise wouldn't have thought about here.

So asking people when they join, asking a year later, keeping those cohort consistence isn't that hard to do, but gives you great data that you can use right now today. And my favorite thing is what we call the community-driven impact score. This is a question we've developed based on the Net Promoter Score, which makes the causation of community a lot easier.

So essentially, we ask a question, how much did visiting or participating in our community influence your decision to renew your subscription this year? Very easy question to use, and you can change that to anything you like, for product adoption or any outcome that you want, but it lets people assign the attribution themselves. And then based upon that, we assign a value to the different answers that they can give, and then we can perform a very easy calculation to have some baseline, some idea of what that community might be worth.

Now, this is just an example. You can adapt it to your situation, but we found this to work tremendously well with our clients over the last couple of years, and doesn't require that much data from the community itself. Everyone got the photo they wanted?

Cool. Okay, and based upon this, then you can assign what the value is. I've only edited this slightly from an actual client, but you can see exactly where the pools of revenue are from each of the clients.

You can see by the visitors and how it all contributes to what you want to show. Easy to understand, easy for everyone to use, and you can deploy it very, very fast. That's method one.

There are some issues with this. One, as you know, when you ask people, there's a recall bias. People don't always recall why they did something.

They might over or underestimate things. I want to be honest, there are issues with every method that I'll present today. Three, there's an attribution bias, social desirability.

People might give you the answer that they think you want, and then there's a non-response bias where you might get most of the best responses from people that are most likely to give you the best answer, so you might want to think about weighting your answers. Does that make sense? Awesome.

Okay. Next method. What if we compare different groups in the community?

Remember when I had that whole rant before how you don't compare members to non-members and all that? It was five minutes ago, but okay. But the difference here is that you can actually change how you do this.

So we had a cybersecurity client back in 2022, or 2021, I think we set this up, where they wanted to know if they're going to host a big annual event for the first time, how do they measure what the impact of that is? There's a whole range of complicated ways of doing that, but if you have a forecast to begin with, it becomes a lot easier to show what the impact of that event was. And we can say, yeah, there is a bump right at the beginning, but it sustains over the long term.

We can see, again, just a forecast, and there's a forecast function in Excel and Google Sheets you can use if you want, but we can see in the forecast itself whether this works for you. And then we can even do it by different groups. We can compare the retention rate pre and post event in different groups of product adoption rate, and really compare different groups of attendees against non-attend attendees, and see what the difference is between, say, California, Seattle, New York, et cetera.

And the data you often find is a bit messy, isn't always clear, but it gives you some very clear things that you can use and show where the biggest impact is. The really hard thing, and I won't go through every part of this, the really ideal situation to do here is what's called matched pairs. Do any of you have any background in statistics at all?

You, okay. So, matched pairs, you don't have to remember all of this, you can get the slides later on, but essentially what you want to do is filter out the impact of people liking the product or using the product to begin with. So what you do is that you figure out what are the things that really matter that impact the level of participation, engagement, like the usage of the product, and you want to match them with an organization that's similar.

So here we have web visits before the period that we're going to measure, we have the pipeline at the start, and then we can measure the impact over time by matching them with a neighbor that's not engaged in the community. This is a technique that's used in academics and statistics and a lot of other fields, and it's something we can easily deploy in this, but the key thing is to figure out what is the metric you're gonna use to match people in a community or not, or engaging in the community or not, and then compare the differences between them over a period of time. This isn't as complicated as what it initially sounds.

With AI tools now you can pull a lot of this really easily and get Claude or something to do it quite easily, but then you pair them up and then you match it in your database, Salesforce, GameSite, whatever you're using, and then you have a control group and you can measure the impact. A quick aside here is that very often when we try to prove the value of a community, we try to prove the absolute value. We generalize the result by all the people in a community.

What we found with CFOs and most organizations we've worked with is that if you just do a sample of customers and show the impact of that, that's usually enough because then you can roll it out in a bigger and bigger way. Based on this you can scale to the accounts gauge. I won't go through every part of this, but we get the point, right?

You find a way of matching groups up in a community without the community, and there's a method if you want to write it down called propensity score matching, which is the gold standard here, but do that, match up people to one another and then compare the differences over time, and then you can say, hey, the only thing different between these two groups is the community that we can see. Does that make sense? They get a little bit more complicated each one.

Okay, have enough time. Oh, we're good. Okay, the test method.

So what we've looked at before is what happened historically. Like we're comparing what's happened in the past. What if we run a test?

I mean, have you ever run a test in your community to prove the impact? No one has run a test. You should run tests.

You really, really should run tests. They're a lot of fun. In 1971, a franchise owner in California created the Quarter Pounder with cheese.

And in 1973, McDonald's finally realized this was a good idea, and they decided to roll it out across the whole of the USA. And they ran billboards like this. But how do you measure the impact of that?

How do you measure the impact of the billboard compared to anything else? How do you know if it's working or not? What happens is that you run a billboard in one part of town and not another, and you look at the impact on sales against the projected sales.

Make sense? And this is a similar method that we can use as well. And it's a method that's time-tested in a whole variety of different situations.

Have a fast food theme today, but. In 2023, Wendy's did exactly the same thing, on Krispy Fry's Cam campaign. But how do you measure the impact?

Well, what they do is the same thing again. They had the identity of people that are exposed to that campaign. They build a lookalike group of people not exposed to it, and then they survey and measure the differences between them.

You can combine these methods together if you want to. And then you can see what the impact is. And maybe you can't actually order that to the campaign that you're running, but it shows a big impact.

And we could do a similar thing in our space as well. I think one of the things that's always really upset me is that we never do tests. Like, we try to do it with our clients and about half of them are open to it.

The other half, really, really not. I think they're scared. But we try to really make sure we have tests to definitively prove what the value of a community is.

And some tests, honestly, are easier than what we think. A while ago, we noindexed a community from Search for a couple of months to show what the impact of that community was. And for all the talk about support deflection and how big the impact is, you can't argue with this.

We were seeing that Search traffic dropped to almost nothing, as expected. The visitors dropped to almost nothing, as expected. Support tickets, way up.

It's hard to really argue with an impact like this. And so doing a test makes a lot of sense. And sometimes you get a natural test that you can do.

If your community is migrating and goes down for a day or week, measure the impact of that because there's such a golden opportunity to prove what the impact of that community is. I read this a while ago. Just to recap, both Virgin Media and O2 took their separate forms offline for new postings in January 2026.

This was written in May 20... Yeah, May the 26th. So these communities have been down for four months.

I would do anything to get the data and support tickets in that time. Imagine the amazing data they have right now. And I bet they're not measuring it at all.

Someone here from O2, right? Yeah, hi. You're not from this one, are you?

No, I'm from Germany. Oh, we like that. Anyway, if you can talk to your friends, there's great data right here that you can measure today.

Anyway, so how do we run a test that actually works? Running a test seems like it's something that's really hard to do, but it's nowhere near as complicated as what you might think. The first thing we have to do is to have a way of splitting groups into separate groups.

I know you can't read this at the back, but essentially you split groups. You split two groups at random. You tag both groups, whether they're in the engaged group or not.

And then you invite them to do something specific. Don't invite them to join the community or not. Invite them to do specific activities within that community.

Maybe it's to engage in events, maybe it's to attend a user group, whatever it is, but invite them to do very specific things. And then you can measure the impact at the end, and it gives you great data that you can use. And really, with any CRM, I know you can't read to the back, but in any CRM, you can really do this quite easily.

And let's see if they renewed, if they churned, you can see what the impact is. And over time, you get great data that you can use. So, the final thing is, once we have this data, let's say we've done all three tests, what do we do with it?

So we've tried so many things to get more support for communities over the years, different ways of presenting the data, trying to show how do we tell the story. And we've done emotional appeals, we've done non-emotional, we just shared the numbers. This is what we found works best.

One, let me go to the next slide first. One is to present a range, not a fixed number range. So what we do with that ask method, we do that as the high end of the range.

We do the match comparison or the test at the low end. And so if you're a CFO or someone that looks at this, you're like, well, I don't know if the number is the high or low end, but this gets way more trust than just one number. Way more trust.

So having a way of presenting the data like this, makes it a lot easier. Because you're showing like, no, we're not gonna pretend we know exactly what the impact is, but this is what the data shows at the moment. The other thing is to show which direction it's going in.

When we do a dashboard like this, we try to show everything, but also is it going up or down? Because what people want to know isn't just does the community generate lots of revenue for the business of retention or pipeline or core deflection, but which direction is that going? And we found the direction to be as important, if not more important than the number itself.

So knowing what direction it's going in has a huge impact on how much people want to support it. And this is where I think we need to be. Combining these two things together, having that range that people can look at and say, hey, I don't know if it's the high and low end, but I know it's in that range.

That's a good thing to look at. And then knowing which direction it's going in, and then you can evaluate everything based upon that. This is where I think we need to be at.

I think this is all data we can understand. It's not always the easiest thing to do, but it's not that hard either, especially with all the data and tools we have access to right now. It's nowhere near that difficult.

And if you want help, you can reach out to me. We're happy to help you. But really, do the ask method first, then look at the comparison groups, and then do the test as well if you can.

If you do these three things, we've found this to be so much more effective at getting support, at getting buy-in and everything else than any other technique or method we've used. And I think ultimately, like I've been saying, we don't need the highest number we possibly can. We've reached a point where we can reconsider what metrics do we want to use?

How do we best get those metrics? And how do we present it to build the most powerful case we possibly can? Thank you so much.

I appreciate your time.

[Speaker 2]
Thank you, Rich. Okay, we have some questions. When a senior leader is skeptical that community still matters, what's the single most convincing thing you've seen work?

[Speaker 1]
Okay, I've got two answers. No, don't all leave. I've got two answers.

I'm just kidding, you've got trains to your thing. I've got two answers for this. There's the answer that I think should work, then the answer I found actually does work.

The thing that should work is showing what that number is in that range, like I've just been saying. The most persuasive thing I've found of getting support is to show them what your competitor is doing. We're behind our competitors.

We have to invest more. Or we're ahead of our competitors. The argument works both ways.

And so I found that emotional appeal works way more than the metrics. But I think generally, if it's skeptical, you have to figure out why. Because your interpersonal relationship has such a big impact on that.

And if they're skeptical, is it because they're scared? Is it that they don't believe that the number is right? Because you have to figure out what the source is, and then you can figure out what kind of story you need to tell.

So there's depths to that, but hopefully whoever asked that, that helps.

[Speaker 2]
Thank you. How could we measure the impact of community on our customers who are getting help in LLM, bleh, bleh, bleh, bleh, bleh, bleh, bleh, bleh, it's the wrong time of day, LLMs from content that originated in the community?

[Speaker 1]
Yeah, so this is a technique that's similar to core deflection, but there's a more advanced version of it. So I don't want to self-promote, but if you look at ghost deflections, I've written some stuff on my blog, it's more complex, and you need to Google search console access, but you can measure that. I think John Wishart has written something on it as well at some point, so look that up on the Gainside blog, but it's too complicated to go into here, but generally if you look up those two sources of information you'll get a great technique you can use to measure that impact.

But I would say any company that's saying, we can show you the impact of any activity in chat, GPT, whatever, it's not that accurate at all. There's a better way of doing it that isn't accountable here, but it's not relying on data from chat GPT itself.

[Speaker 2]
Okay. Of the three measurement methods you've shared, which do you see teams getting wrong most often, and why?

[Speaker 1]
Overall, the ask approach. I think when we ask members, we don't do the before and after. So we get one static point in time, and then you get the challenge of that customer support issue where people just end up in the community because they're out of options.

So that's always a concern. I think that's the one I see most common. I think the other one, like I said before, is when you compare people in a community to those that aren't, you're always going to have a number that shows that people in a community do X more than people who aren't.

I remember years ago, there was a company called SignCaps, I think, and they did a study on Facebook fans, this is like 2010 or something, and they realized that fans of a Facebook fan page, or sorry, fans of a Facebook page were worth $73 more than non-fans. Right, yeah, they're fans. That's how that works.

And so often these kind of things aren't really thought about. So I would say that, yeah.

[Speaker 2]
Okay. What metrics would you track before a community has been created so that you can then re-compare after launch and then after adoption?

[Speaker 1]
Great question. I think this is something that we don't do enough. We should look at the ticket deflection rate per customer before we launch a community and look at that over a period of time.

We can look at the retention rate and measure that over a period of time. Most of us have a community already, and so we don't get to do that before and after test. If you do, then get all the metrics you care about in place, product adoption, retention rate, all these things, and then measure it after you launch the community, and then you can correlate that with community activity over time and see what that relationship is.

I think that's generally a pretty good win. And I think it's something that we don't do enough. Thanks for the question.

[Speaker 2]
That's a great question. Does dark bot traffic have any value, or should it be regarded as completely negative?

[Speaker 1]
It's a tough one, because we don't know exactly what it is. And so we know that a lot of clients of one vendor, not Gainsight recently, had huge issues with this. And they were getting charged for it, and it's difficult to say what the traffic was.

And some were getting a million visits from Bangladesh. You're like, I don't think there's a million people in Bangladesh that suddenly became interested. We don't know.

And so I think generally it's hard to know. I would say at the moment, no, because otherwise they would be declaring as bots. So I assume there's something nefarious in what they're doing.

But I don't have a lot of data in that. I do think that there's a great question in whether you should open your community to ChatGPT and Claude and the like. And I think generally most of us should, because there's value in how they surface those answers to your customers, and I think that makes a lot of sense.

Otherwise, no, I don't think so. But I could be wrong on that, very wrong on that.

[Speaker 2]
Okay, I think we've got time for two more. What if leadership does not accept a more trustworthy KPI, but expects stacks of magazines?

[Speaker 1]
Yeah, you should change jobs. I don't actually have a good answer. I do think, it's the same thing again.

If the KPI is not accepted, you have to figure out why. And I remember with a client years ago where the number for call deflection was three times higher than the entire customer support cost of the organization. Like, okay, the number is too high.

So then you revise it down. But how does that make sense? You're drawing the bullseye around the arrow.

That doesn't make sense. And so I think for these situations, you have to find out why they don't believe it. And usually it's because the number is too high, or it could be because they don't want to believe it, as in they don't like the community and they're predisposed not to like it.

Or the third one is they don't like you. And I don't want to be mean, but if someone doesn't like you, they're not going to trust anything that you say. And maybe it's not they don't like you, but you haven't built that credibility in their eyes would be a nice way of saying it.

And maybe you need someone else to give them that number, or maybe you need to find the peer benchmarks to give them that number. But it's never a case that they just don't accept the number. There's a reason why they don't.

And it could be it's a threat to them. It could be it's a personal relationship issue. It could be something like that, along those lines.

But you have to work that out. And like, I know we focus a lot on the data, but how you personally present the information in the room has a huge impact. That's a whole other thing, but like it does have a huge impact.

[Speaker 2]
Okay, quickly, let's make this a quick fire one. Which one do you want to go for?

[Speaker 1]
I'm not going to do the AI one just because it's not the talk, but I'm sorry for that. Let's do the top one. What triggers a community survey to be sent to a customer and how do they know if they've performed any action?

We measured it in the database. When they join the community, there's a survey that they get three days. I don't remember exactly, but I think it's three days after they join or after they make their first visit, have to look that up.

And then a year later, they get that survey again. The second survey, we do incentivize the responses just to make sure we have a good number of response rate. Otherwise, the response rate can get really, really low.

And let me just tackle the question from Matt Smith as well just because he put his name to it and everyone else. Who is Matt, by the way? Hi, Matt.

Thank you. The only one that put his name to it. How do you effectively identify and measure community qualified leads?

I don't know, to be honest. Like, I feel like with community qualified leads, it depends how you define the term. The problem with building a community to generate leads is that people that aren't customers don't tend to participate in communities they don't buy from already.

For like Reddit, it's fantastic. You know, people go, they ask questions. Generally, we look at whether they make a number of pre-purchase questions or attend events that are demos or things like that.

But this ties into more how you're scoring leads elsewhere and how we sync it up with that. It's very unique to each organization we've found. The AI question, if someone wants to reach out to you after this, I'm happy to answer that.

Thank you very much. Thank you so much, everyone. Thank you.