Why Telemetry Data Is Your Churn Prediction Superpower: Lessons from Boomi

All right, so three o'clock in the afternoon on day one, we're gonna be talking about telemetry data, all right? Really exciting, really awesome. It's the superpower though. We will convince you of that by the end of today, okay?

I am very fortunate to be up here with my esteemed colleague over here, Arun. He's gonna tell you how all of this actually works. And so we wanna get as much time in front of Arun as possible and get your questions ready. He knows everything about CS there is to know.

So first, raise of hands, who's experienced surprise churn in your business over the past 12 months? Yeah, surprise churn, all right? Our mission today, right? And the superpower that you're gonna learn today is how to get rid of that surprise churn through telemetry data.

And we're gonna prove it to you today. Now, this problem of surprise churn is the problem that we talk about here at Gainsight Pulse. It's actually the problem that is becoming the most important problem at the board level of every software company in the world right now, right? So I would argue that Gainsight Pulse is actually the place to be if you are trying to solve the most important problem in the entire software industry, how do you maximize NRR?

Right, how do you find growth? How do you retain your customers? It is the most important thing that CEOs and board members are thinking about right now. As they're matriculating new technology, new AI capabilities into their tech stacks, they're trying to keep as many of their customers as possible to make that happen and innovate with all of this new AI tech that we have coming our way.

And so super important problem, and a ruin is at the forefront of how that's being solved within the industry right now. The thing that we've been using to solve this problem, I would argue is only 20% of the customer signal that is existing in your stack today, right? You're using all sorts of different things that your customers are telling you, right? But you're not talking to all the customers, every single user of your software all the time, every single day about how they use your product and what they think about your product.

But they are telling you what they're doing and how they're doing it, how they're interacting with every single person in the organization through the way they use your product. We call that data telemetry data. And there's multiple ways that your customers are using your product today. They're physically using your UI, right?

We all know that. And we're getting that data through the product analytics capabilities like Gainsight PX, right? But more and more, they're also connecting other systems together with your products and driving data through your systems, through APIs. And as we've heard about a lot here at Gainsight Pulse through MCP servers, right?

Data is flowing between your platforms through system to system interactions and user to system interactions that we have to understand deeply, right? That's where the signal is. 80% of the signal that your customers are giving you are through those interactions of your product. And when we marry that up with how that looks from a revenue perspective, we actually start getting to price to value awareness and predictive power that was never even possible until the AI revolution came aboard, right?

Super exciting time. And so we're solving the biggest problem, NRR, through some of the most advanced capabilities, AI capabilities that's ever been in market to date. How we're doing that is trying to be simplified as possible, right? The first thing we're doing is we're ingesting all this data together.

But with the beauty about AI is that you don't have to do anything. It can learn and understand just from your data. So for any product that we have in this room right now, regardless of how the product works, the machine can learn about how your product works through the interactions of the product with the people and the system interactions that we were talking about through APIs and MCP servers. What that's able to do is predict at a massively predictive situation.

It gives us the ability to predict over 90% of growth and churn of your business, 12 months and more in advance. So Rune's gonna talk about how that looks at Boomi and what they're doing with that. But more importantly, knowing is just part of the problem. Knowing where that churn is going to be, where that growth is going to be is part of the problem.

How do you act on it? What do you do to actually change the future and make those customers stay? That's the next wave of really interesting intelligence that's coming online. And so at QuadSci, we're looking at the action problem through AI, through the ability to converse with this data, understand this data by having conversations with the AI about how customers are using the product, what APIs are being used, what MCP servers are being used.

What does that mean to the growth trajectory, to the churn trajectory of a customer? All of those questions can be answered in-app, but also through the MCP server. We hear about all MCP this week. And that gives us the opportunity to ultimately power any type of use of AI that you're deploying within your organization.

And Boomi knows that more than anybody. The ability to drive AI through an integrated framework is what Boomi does. And so I want to hand it over to Arun to talk about how this actually happened for Boomi, and what it looks like today, and what they're thinking about in the future. Thanks, Sean.

I've got my mic. All right. I'm gonna walk you through, walk you through our experience with QuadSci, but more importantly, the problem we were solving, the scale at which we're operating, and the business case we built around buying another software, and measuring the result of the software. So once more, my name is Arun Parikh.

I lead customer success for Boomi, joined by some of my colleagues who run the similar business in Boomi. And a little bit about Boomi. How many of you have heard about Boomi? Great, perfect.

So look, I think if you look at us, the number one independent software company, we compete with companies like Salesforce, SAP, all this big weight. So what we do is, we're very passionate about what we do. What we have built is not have a hobby. We want to make sure our customers are massively successful.

They get maximum value of our product. We've been a leader in the Gartner MQ for I-PASS for 12 consecutive years. And then we are also challenges and leaders of other quadrants. We have 30,000 plus customers globally.

We have OEMR products, we have direct customers, we have partner, we have resale customers. We help customers of all shapes and size. And again, we are an agent AI company. We have 75,000 plus AI agents that our customers are using and they have it in production.

So we started our partnership with Quartzi over a year ago, a year and a half ago. But the start of every partnership is to define the problem statement and the challenge and what are we solving for. And I know every revenue leader today here will talk about why NRO is important, why churn is important. For us, we built a narrative around churn.

So 1% of churn in our business equates to, let's say 1% of churn in our business equates to 5 million. It's a very easy narrative to build. But the narrative we build was, hey, what is the impact of 1% churn? For a business of our size, our cost of acquire is 3x.

So if we lose 1%, if we lose 5 million in churn, it would make us spend 15 million to acquire the 5 million back, that's number one. So build a narrative around churn. The second narrative, which is even more powerful than that of it is about, hey, for a company which operates in our space, what is our impact to valuation for a 5 million churn? It is 10x to 20x.

So if we lose 5 million, our valuation goes up by 100 million. That stuff, everyone cares about. So the board cares about that stuff, the CFO cares about that stuff, and everyone cares about stuff. So that was a challenge we're trying to solve.

But how do you solve that challenge for a business as complex as booming? We have 5,000 plus direct customers, we have 10 different product lines, our customers use our product in so many different ways. So we have to make sure they adopt our product, make sure we have to make sure they are value aligned to our product, they get measurable returns from our product to make sure we can eliminate that churn. And the other thing that we were doing was we had 2,000 renewals that we were actively processing every year.

And 50% of them are very complex renewals with multiple product line. So the state, the first crawl attempt was how do we get a sense of what is happening with our renewal? So we put in a practice where CSMs were responsible for forecasting renewal six months out. So two years ago, CSMs were not forecasting renewals.

And we started to approach it, hey, how about we, CSMs are the most close to the accounts, we have stability with our CSMs, how about we let them forecast renewals six months out? And we started to do that. And the problem we saw was when we identified that this account is gonna churn six months out, we activate our safe playbooks. Hey, let's kind of align behind this account, try to save, but it's too far too gone, too far too lost.

Customers who make a decision to move out of a platform, they make a decision 18 months out, not six months out. So we were very reactive in terms of trying to go all in on a customer who have already decided to move out the platform. That was problem number one, if you're gonna solve. And like I said, surprise churn, when we did the synthesis with QuartzEye, we understood, hey, what is our CSMs forecasting?

And what churn without us forecasting as churn? And the value was five million. So it was a problem we're gonna solve. So, and then the, we all know that we have a lot of software.

We have a lot of software. We use Gong, we use GainSight, we use Pender, we have Salesforce, we have a lot of systems for quantitative data, quantitative data. We were tracking everything, we were asking CSMs to put a lot of this thing. So the question was, why buy another software?

Why buy another software? For two reasons, one is we wanted to change the frequency of, we want to change the timeline in terms of how early are we forecasting renewables. So six months was not good enough. We wanted to forecast renewables 18 months out, 12 months out, and human intelligence won't scale when we want to forecast 18 or 12 months out.

We needed to build system intelligence, a lot of ingestion, data ingestion, and Sean's gonna talk about that if he has time. We wanted to kind of get all signals to say, hey, there is a risk, there is a potential risk, 18 months out, 12 months out. And then the other problem we're gonna solve was, we wanted to provide accuracy that, hey, we are forecasting a churn, and we are forecasting this retention, and we have to commit that number. That number cannot have a variance of more than 10%.

So if we say we're gonna commit 88% or 89%, we better commit that percent, and not lower than that. So that's why we've kind of added quasi as a layer of predictive intelligence over all the systems we have. And I'm gonna tell you why. So with that, our current state right now is we have every renewal today scored in five scores.

Hey, is this customer gonna expand? Are they gonna be stable? Are they gonna churn and downgrade? So we have that scoring for every of our customer.

And I'm gonna tell you why that scoring is super important for us. We have not been able to get to 12 months yet, but we are getting to 12 months. We've been able to accurately predict in that 10% accuracy eight months out. And we're trying to get to the 10% accuracy 12 months out.

And then we'll probably do eight months out. And again, like I said, we are trying to get a 12-month head start in terms of risk, risk to signal to an account, so we can put all of our save motions or all the fun stuff that we do to save a customer. And I don't give this math. I give this math to everyone.

A lot of people talk about long tail, high touch, low touch, all that stuff. But when a customer churns, the amount of effort it takes is the same. Whether it's a 100k customer or a $1 million customer, when they say they're moving away from your platform, the amount of effort capital you put in is the same. So with this state, where are we right now?

And this is a funny story. So buying every software is not just a software purchase. It's a change mindset. It's buying into a change.

People are doing things in a certain way, and we have to tell them, hey, now you have a better way of doing things or a different way of doing things. How does it land? So the 12 months before, our operating model was very simple. We had health scores.

They will spit out, hey, this account is green, yellow, or red, and then we have a CSM sentiment. A CSM would go and say, you know what I spoke to this customer last month? Everything is great. And the health score would say yellow, green, and the CSM would say green, yellow, whatever.

So we had these two things. And when we came up with the first iteration of, hey, what is Quatsyde projecting, we said the sentiments were completely off. The health sentiment was off, the CSM sentiment were off, and the Quatsyde was saying this account is gonna turn on a contract. And we took this data, applied to four years worth of posture.

Hey, what was happening in our business three years ago? What was, we did not forecast it, but we at least had health scores. Can we get a delta of what was green versus what was projected to be churn and contract? And we found some amazing insights in terms of, hey, green is not always green.

When the CSM says sentiment is great, it is not always great. And I don't wanna say this, but customers lie to us all the time. Or a better phrase is, hey, we are not talking to the right person because we're making the right decisions. And there's a lot of economic impact as well that goes into things.

So what we did was, we were always, our behavior was this, health score says green, we always override a health score with a CSM sentiment score. If a CSM said, hey, this is green, we would override our health score. And we take the CSM's word for this, and then we had surprise churn. That's the problem we were solving for.

But now, our operating model is completely different. If our platform says, this is gonna contract or churn, we stress it, we stress test it. We don't believe in health score. We don't believe, I mean, we do believe in health scores, but we don't believe in what the health score is spitting out and what the CSM sentiment is spitting out.

They have to prove it out that, hey, this software, whatever signals we are ingesting, and we are ingesting 70 billion plus signals to come at a score in terms of this account is gonna contract or churn. And you say, prove it up, prove us out, based on all the history of your conversations, the telemetry of your customer in terms of why you feel like this customer's gonna renew or not renew. And we're doing it 12 months out. So that's a mindset change and behavior change that we have.

And even today, we have people saying, you know what? This is a false positive. Great, but prove us out. Why is it a false positive?

So the final thing I wanna say was, we had CSM pressure testing our health scores, and now we have, we are pressure testing our CSMs based on the score that the software is generating for us. And look, there's no transformation without some sort of a scorecard. We have been partners with them for the last one, one and a half year. We put them through an extensive evaluation, completely fact-based evaluation, based on, hey, prove us out, look at our historical numbers of churn and what we predicted.

And this is what happened, right? So we were, we are now able to predict churn accurately for the last 12 months, or at least have a view of where the customer's gonna land in terms of renewal. We surfaced around 27 million of historical churn, like pre-forecasting. Pre-forecasting, when we were saying the accounts are green, yellow, and red, we surfaced, hey, there's 27 million of accounts where actually the health score was not as the health score that came out of your systems was.

And in the last one year, we have saved 18% of our churn. And these are big numbers, and let me simplify it in a very simple way. So when we did this rollout to all of our CSMs and our frontline managers, we made it very simple. A CSM in any company owns up from 10 to 20, 20 to 40 accounts on an average, right?

Depending upon how big the accounts and whatnot. So we took this value proposition to everyone. You have 40 accounts. What if you say one?

What is the impact of that? Let's say you save 100k, 100k of churn and downgrade. In your book of business, what will it lead to? Like, you do the math, we have 100 CSMs, and every CSM saves 100k of surprise churn because of this predictive intelligence.

The economic impact is 10 million. That's the problem we are solving for at the grassroots level. With that, pass it over back to Sean. (audience applauds) All right, so first of all, I wanna say thank you to Arun, but also the Boomi team, which might just stand up.

(audience applauds) The Boomi team is one of the most intellectually curious group of people I've ever worked with, right? They're looking at their business in ways that most organizations just are not. And so we're super fortunate to be partnering with them and working through this journey together because telemetry is tricky. Like Arun said, 10 different products, global business, $500 million.

These are tough situations to put yourself in. For every single CSM, they're working through that complexity every single day. How can we make their lives easier at the end of the day? And what are the signals that are most important to help them do their jobs better?

That's what we're thinking about every single day, how to make those people's lives better. How do they engage their customer with more context and have better conversations faster every single day? So what we just asked you to do as you're starting to think about this, every single one of your organizations has telemetry data that you're not using today. It's housed in product management organizations through tools called product analytics tools.

GainSight PX is a great example of that. Then you have your observability tools that we use to find the API calls, the MCP calls. That data's housed with your engineering organization. This is your data.

You're absolutely allowed to use it. Every single company in here has this data. So what I would ask you to do on Monday when you go back to work is find the people that have this data. They have the keys to this car.

They have the keys to this unbelievable trove of information about your customers. Find out who it is and then try to sample it. See if you can get an extract for a customer and see what it looks like. The data is hard to understand, but it's important for you to understand what it looks like because it's so valuable to customer success, to sales, to marketing.

So that's what I'd ask you to do coming out of this session. Our goal at the end of the day is to put all of those hands down. Eliminate Surprise Churn. And if we're able to combine in the ability of how your customers use your product with how they're talking to you every single day in your entire organization, the truth is there and that Surprise Churn goes away.

So our goal is to ultimately put those hands down by next year. And so we can open it up to questions. I saw a bunch of them coming through. But I will open it up here.

Since you guys like my voice so much, I'll come back up. Okay, thank you guys. That was awesome. First question, are there commonly tracked signals that you find are not effective predictors of churn?

What can we stop doing? Okay, this is a question for me. Sean? I think you start.

I'll start. So there are two things. Every software has some way of getting telemetry. But how are you comparing against?

That's the number one thing. So we are compared, hey, this customer is using this product. This is a product telemetry. But can you give us a sense of, hey, the customers in the past seven years who had similar product telemetry and what was the behavior?

That's what we're tracking towards. So static product telemetry you will get, but historically analysis in terms of, hey, customers who were in this state, 12 months out, what was the outcome? So that is very difficult to track. And that's what we're trying to track with Quartz Eye.

Because they have built as a view, hey, X number of customers were in this shape, 80 months out. And for those X, Y led to renewal, X led to surprise, Sean. So that's where we were trying to get to. I hope that answered the question.

Yep. Right. Well, and then what can we stop doing? AI can do this research for you, right?

It can learn from your data and understand your business faster than any set of rules you create for your business. So I would say what you would stop doing is coming up with rules that may or may not be predictive or reflective of your business because the AI can figure it out automatically. Yep, if I may add one more thing. So a lot of people track product adoption as a driver for churn.

Hey, they are 80% adopted, or whether you're selling seatbit licenses or consumption-based, that is a big red herring, in my opinion, because that is what leads to surprise churn. Basically, we say, hey, this customer is adopted at XX or Y, so we are good. That is a big misnomer. I think the platform activity, hey, are they logging up?

Are they waking up every day and doing X amount of work in the platform? It's a bigger driver of ensuring that the customer cares about the platform, any platform. I have a question. So now you're able to predict renewal accuracy with 90%.

How often do you have to keep doing this type of analysis, if I'm understanding correctly? Oh, every day. I mean, we do this analysis in a lot of different ways. And we are one of the most, what do you call, trying to find a better word of it, very skeptical about everything we try to renewal.

So if a customer expands, we are skeptical. Believe it or not, if a customer expands, we say, oh my God, they expanded to 200X. Are they gonna renew in full? We have to really get value aligned with them very quickly so that, hey, they were spending 500K.

Now they're spending a million dollar. Do we now add million dollar worth of value for them or not? So we're that skeptical. So we do that analysis every day.

We do that analysis on if the customer expands, if the customer is flat. Emily, my colleague, she's running a project in terms of customers who are stalled, who are not building anything on the platform. So we have cohorts and we do analysis by cohorts every day. Wow.

But this is also the beauty of AI, right? So the AI is doing the hard lifting work behind the scenes that allows for that analysis to happen. And do new factors emerge all the time or is it pretty? Well, it does.

Like I said, as Sean said, I'm sure a lot of companies have synthetic good expansion, which is basically expanding product lines or customers buying more of what you have. But we always know that buying more of what you have always goes through some level of right sizing. So that is, and that's analysis we can see, hey, where is our downgrade? Is our churn basically on right sizing?

Or customers not utilizing the platform they bought? Which is very difficult to track. Like, you know, hey, you can track telemetry for one part of our platform very well. How do you do it for a platform where there are eight or nine different products and all of the telemetry is different?

Very true. With all those signals, how did you figure out which one should hold the most weight? Is this now replaced your gain site health score? So the way that it works is revenue outcomes is what matters.

And so the AI is trained on for the history of all of the product usage that you have going back years. What were the revenue outcomes? Did they grow? Did they stabilize?

Did they contract? Did they churn? And then like Arun said, the AI is able to figure out lookalike accounts that did the same thing, had the same patterns of behavior, and it determines it that way. It's not a human exercise.

The machine's figuring all of this stuff out. We had 70 billion events that ran through the AI that was able to figure this all out. That's not a human-led problem. And so, yeah, we're thinking about this differently.

And right now we're dancing with the gain site health scores because there's so much activity and so much effort that's gone into that. And so we look at it as a additive now, but yes, we're looking at what does this mean for health scores going forward. I'll give another example. I'm gonna give you another example.

So if you ask any CFO or anyone in finance, and I think we're a very, very easy example, you go to a restaurant, you order food, you get your food, and then you look at receipts. So this is a three-point check, not a two-point check. So doing a three-point check, make sure what I ordered is what was delivered, and the bill that I have is what I, is right. That's exactly what we do.

We have a CSM sentiment, and we have health scores, and we have a lot of systems which gives health score, but we're adding a third layer. And all of this thing has to reconcile. For us to kind of be say, yeah, absolutely, now I understand. If one of these doesn't reconcile, then we go deeper.

Something is wrong somewhere. I have another hard question that just popped in my brain. So with the renewal forecast accuracy being 90%, what do you think is holding you back from achieving 100%? Customers lie to us all the time.

(laughs) No, obviously there's always gonna be, I always say, churn is like a gravity. You always hold it up. No software company will ever be able to eliminate churn. And obviously there's always gonna be some level of churn that we cannot predict.

And mostly, we also wanna do right by our customers. We calculate churn in terms of NDR, not dollar attention. So when a customer logo goes off, it's things a lot. But some level of churn we even allow.

The customers want to right size or want to swap or want to go down because if the business is going down, we allow that. That stuff is difficult to forecast and we allow that. What solution does Boomi use for product usage analytics? Gainsight, Pendo, Salesforce, WhatSight, many.

But our main PX is Pendo and Gainsight. And then obviously we build layers over the top. And then they have an observability platform as well. Yeah.

That we saw on top of. Very cool. A 12 month churn model usually needs more than 12 months of history. What's your standard renewal term that you ask your teams to position?

We always position three year. We have a very good renewal management process. We only provide three year and anything under three year is subject to exception. And obviously we have auto renewals for a lot of our long tail.

But for a spend threshold, we only offer three. And then the CSMs and the account managers have to kind of build a case in terms of why we're given anything that's in three. And sorry if I may. And we also, a lot of this is also driven by, incentives drives human behavior.

We have incentivized so much in terms of securing multi-year renewal. Not just because multi-year renewal is important, but the amount of labor that we have to spend, that the customer has to spend, year on year on renewing, on building a business case, and doing the assessment and whatnot is too time consuming. And inexpensive. I work for a company whose data teams prefer to build, what?

We skip them. I have been skipping around. This is my job, thank you. (audience laughing) I work for a company whose data teams prefer to build before buying.

How does CS advocate for a tool like this without alienating or getting blocked by our data teams? We get Chuck and Ori to answer this question. They basically answered this question at the keynote today. But-- They're not here.

Yeah, look, they're not here. So I would probably do a very bad job. Do a very bad job at being a mini Chuck or a mini Ori. Yeah, I think it's about building a business case.

It is always about building a business case. Our job is to be, the job of a core CSM is to make sure we build partnership, we build relationship, and we have a valuable line. The software part, I don't know, Matt. I mean, I'm not an expert in tooling strategy, but we can take that question.

I think Matt leads our tooling strategy and he can always answer the question. Yeah, I mean, what we're seeing out there is that many of our customers are being super focused on the products that they bring to market. And bringing the data and AI capabilities within their own products. They're not serving necessarily the operational needs of your customer success team, of your sales team, of your marketing team.

We think about that every single day, and we're innovating every single day. So the data team might come up with a model that works for a given point in time, but then they go off and do something else. And they're not maintaining that model, and it's a new problem that they're trying to solve. And just like any product, you're buying a product because there's innovation that comes with that product and that subscription means that you're getting more and more innovation every single year, right?

Without having to do anything in your organization. And so that's classic build versus buy mentality, but yeah, that's how I frame it up to the data team. And where do you want your company to spend their energy? Is it building products that maybe they're not the best suited to build, or do you want them to invest in their own product?

Look, for us, I would say our fundamental decision is we build a lot of stuff internally. We're gonna be talking about in our session tomorrow. We do a lot of build ourselves, but it's always based on what are we trying to solve for? So we have to align ourselves in terms of what are we solving for and how are we measuring it?

And if you can't do that, then I think the build versus buy argument goes away. Is there a question on there that you wanna ask? That you wanna go for? The next one looked like the one that we skipped, right?

Perfect. Are using telemetry to monitor new customer adoption, what metrics are you watching closely? Yeah, we do. We do.

The biggest metric we are looking at is time to activation, which is how fast our customers go like with the first use case. That's a metric that's an odd start for us, but we know with the stack that we have, it's a very prescriptive journey. Customers have to buy the platform, they have to do something, they have to build things, they have to set out, they have to pour into production. So we map the whole journey in terms of, and then we have state gates.

The customer have to get on the tool, get building on the tool in the first two weeks, and if you don't, we launch a lot of digital motions and human intervention, but that's a metric we're tracking. So the north star is time to activation, but then we have stage it out in terms of, hey, time to activation is 30 days. What the customer would have to do in terms of the day before, days before. And is the telemetry automatically giving you those signals when something is incomplete?

Okay. Did you use churn reasons to help you find signals and telemetry data to test and compare? If so, did you have challenges with getting churn reasons? I don't understand the question.

I think I understand. So when QuadSci makes a prediction, the prediction comes with the reasons driving that prediction, right? And so at a customer level, we can see for every customer, what is the reason that someone is growing, stabilizing churning, right? And then what we can do is roll that up and be able to identify whether the things at a macro level or a regional level that are ultimately driving churn through those parts of the organization.

So I'm wondering Arun, how have you incorporated that into your playbook? I think if I get the question wrong, apologies, but we don't have great consistency in terms of logging a churn reason for every downgrade and customer churning. That's the number one problem. We have 30,000 customers.

If they downgrade, if they move away, we can't expect our CS Simpson log while the customers churn. So the thing we do is, hey, what was the behavior of this customer 12 months before they downgrade or churn? That was where we're tracking. So it's less about human-ended reasons, more around the patterns of the customer 12 months out when they decided they were gonna right-size or go away.

What sort of 70 billion pieces of data are you ingesting to make this work? Please list them all. Well, list them all, yes. All right, great.

I think Dan is not here, but I think I may have said 70 billion, which is like underscore them. I think they may have said better billion, but I went with billion, but all sorts of data. Like we use extensively every time anything hits our platform. User logging into the platform, going into a screen, doing an action, dropping a thing in the canvas, to deploying things, to changing things, everything.

So we love each and every action for the user. So today we know, if we go and open our customers dashboard, we would know in the last 10 days what the customer has been up to. How many times they have logged into our platform, done changes, do configurations, build things, modify things, deploy things, change things. The whole adoption has changed and decreased all of that.

So we look at everything, like a whole leading metric as well as a logging metric, in terms of, hey, once we start to see a meaningful difference in terms of adoption, do what they're doing every day on the platform. And then you also have a by role. By role, yeah. What is the administrator doing, what is the developer doing, what is the analyst doing?

So did QuadZy remove the need for manual CSM renewal? Well, no, we, you know, no, we just, we just test every CSM's renewal forecast, what's X scores. And if, so we had this practice, hey, this is a false positive, this score is false positive. And we used to take that as answer.

Now we're like, no, tell us why it's a false positive. This is what the data is saying. Why do you think it's a false positive? So are they still going in there and they're marking what they think their renewal?

And then in a lot of cases, we, when we find surprise, then we adjust our renewal forecast. And then we work on a problem. So the way our dashboard is, this is our rolled up CSM forecast. So let's say, hope I can give you a picture.

We, by four quarters, we look at, hey, this is our bottoms up CS forecast. And this is, let's say 88%. And what I say is, hey, based on our telemetry, our understanding, your forecast is 84%. Then the 4% is we go after.

That's the thing we don't know, or there's a misalignment. So it doesn't, that is the need for CSM to enter forecast manually, but the 4% is what we are stress testing every day. Okay. What is the data available with engineering when you don't have PX or other similar tools?

So there's a category of products called observability products. A to dog is probably the most well-known of all of them, but New Relic, Elastic, these are all owned by your engineering organization. And what they do is they capture logs from your platforms and store those logs. That's the data source that the engineering team has.

Perfect. This is a nice long one. About 70% of our churn is deemed non-controllable, like closed businesses, mergers and acquisitions and liquidation. Do you find that telemetry can be useful in the prediction of this sort of churn?

Or are your numbers based on churn that can be mitigated when you can predict it? Very, very useful. I love this question because this was a challenge I gave to Quartzite, to Dan specifically six months ago. Dan, there is controlled churn, there's uncontrolled churn.

There's events that happen in the platform, but there's events that happen out of the platform. As in people leaving the business, people acquiring businesses and whatnot, how can I do this? And Dan showed me. The V3 that I talked about, now I can say, "Hey, this customer, give me everything." It tells us now, people changes, role changes, M&As.

I tell them, "Hey, Dan, tell me what is, "Boomie is a customer of yours. "What is happening with Boom?" Dan ran that query and he told me, "Hey, you have launched a partnership with OpenShift." He did this thing, the Quartzite platform did something that we announced one week ago in our company conference. Now we have also intelligence about what is happening in the platform and also out of the platform. Okay, does anybody have any other questions they wanna submit in the last few minutes that we have here?

But I love this question because that is true. A lot of this is external. We're just gonna stand up here awkward if you guys don't submit more questions. I'm just kidding.

Okay, well, you guys were awesome. Thank you so much for giving us extra time to ask you questions. Good morning. (audience applauding)