From Sandbox to Scale: A Framework for Smarter AI Experimentation in CS Ops

42 min.
2023


Session Abstract

CS Ops teams are being asked to innovate with AI while proving that new ideas create measurable value, which makes experimentation harder: teams need room to test, but also a disciplined way to evaluate what works before it becomes part of the operating model. In this session, Polly Sims shares her Sandbox > Scoreboard > Stewardship > Scale framework for moving AI ideas from controlled experiments to repeatable practices. You’ll learn how to structure AI experiments using the framework and what to define upfront to measure business value, establish ownership, and put practical guardrails in place. You’ll also see how to make evidence-based decisions about when an experiment should stop, evolve, or scale into a repeatable CS practice.


[Speaker 1]
Thanks, Emily. Thank you. Good morning.

Well, so I should probably start by saying that in Dublin at Pulse last year, I said to my boss, oh, I think that I can stand up and give a presentation to a room full of my peers. And it's possible that I had had too many Guinnesses because because the closer we've come to this, the more nervous I felt. So I am pretty nervous.

But I know that you'll put your arms around me metaphorically and keep me going. So it's nice to be with you all today. It's nice for me to be here.

And the very blonde version of me that appears on that slide as well. But we are, in fact, both the same person. So from sandbox to scale.

How do we get from everything that we've heard over the last 24 hours to where we need to be as we move forward into 2027? I think the problem is this. Your organization doesn't trust AI.

And I think I mean, it wants to, of course, but it doesn't. Not yet. And I think as individuals, we're not really trusting of it either.

And why can I say that? Well, how about the cost of it going wrong? Has everyone seen that that post on LinkedIn of the newspaper article that's been written?

And it's made it all the way through to publication. And at the bottom of the article, it's the prompts been left in. Would you like me to make this sound warmer, more professional?

And we laugh and laugh because we think that would never happen to me. I would never be that silly. But that's the fear, isn't it?

The accountability if something goes wrong. Who's to blame if I write an agent and it does something unexpected? Am I the one to blame?

Am I in the firing line? And probably most critically, what are the financial and legal costs and the human costs of something going wrong? Because a summary that's been generated by AI can be corrected by a human.

But if it autonomously takes a payment. Unauthorized, that's a more difficult problem to solve and a costly one to fix. But what do we do?

Because it's clear that we need to move beyond the basics of AI assistance and get agents to earn their place in our operating model. But to do that, we need AI to earn our trust. And so that's what I want to talk to you about today and present to you a framework of how we can get agents to be trusted in our operating models.

So someone has an idea and the idea is promising and there's energy around it. Someone builds a prototype and a few people try it. And the first reactions are encouraging.

And the experiment then enters this framework. This middle ground, the owner is still answering questions. The process for using that agent is not fully documented.

The data is fine, but it's not quite clean enough. Nobody really knows which metrics matter. The cost isn't visible.

And eventually somebody asks, so are we scaling this? And the answer comes back normally as something like, we're not quite there yet. And the gap between a promising sandbox and a repeatable process is where a lot of our ideas get stuck.

The problem is not a lack of ability or the lack of creativity or even a lack of support for the original idea. It's that the experiment itself has to do more work now because it has to be safe enough to try. It has to be measurable enough to evaluate, governed enough to trust and useful to survive beyond the person who originally created it.

And so today I'll share a framework for moving AI led initiatives from controlled experiments to repeatable practices. Sandbox, scorecard, stewardship and scale. And I'd like to make the outcome very practical for you all.

By the end, you should be able to answer three questions. What are we testing and where are we safe to test it? What evidence would tell us that it's created value?

What would make us stop, refine or continue the experiment? By means of a show of hands, how many in this room have an AI experiment that's made it all the way from pilot to a repeatable operational practice? Yeah, about half.

And of those people, how many of you got it right the first time? Yeah, that's the middle ground, right? But it's not a failure.

It's just a signal that the experiment needs a better operating structure. But let's zoom out for a moment. Ops teams are being asked to innovate with AI, but also prove that ideas create measurable value.

Those two expectations are connected, but they're not the same because innovation asks what's possible. Your business is asking you, well, what did it change? What did it cost?

And can we rely on it going forwards? And there's another shift happening alongside that. The planning cycle in your organization is getting shorter.

Expectations on individual performances are expanding. We're taking on more work to stay differentiated in an increasingly noisy market. And so that creates a particular kind of pressure because it becomes difficult to build a program with longevity when the definition of success or the priorities or even the strategy itself changes quarterly.

That leads to feelings of anxiety and overwhelm for people in customer success, not because there's no opportunity to make an impact, but because it can feel as though everything is urgent, but nothing's given enough time to become durable. And that's one reason why I myself need a framework not to add process for its own sake, but to be able to feel that I can create continuity in adjusting time operating model. And of course, don't get me wrong, leadership teams have the right to ask where can we optimize spend?

How can we make more use of AI? What can be automated? And then alongside the ever present, how can customer success be a more profitable cost center?

Right. We hear it every year. And if you work in customer success, you feel that pressure immediately because you're asked to handle more, more products, more complexity and prove that impact in hard numbers and not just in compelling anecdotes.

And at the same time, the financial case for AI is still developing. PwC's 2026 global CEO survey found that only a minority of CEOs report both cost and revenue benefits from AI. But many of them reported no financial benefit at all.

And that is important context because the pressure to adopt is high, but the return is not automatic. So an important question is probably not is probably not are we using AI, but is our data and our operating model, our governance and people ready for the work that we're asking these agents to do? Everyone in this room has a has a role to play in making that happen, because in ops, we sit close enough to the work to see where the friction is.

We understand the systems and the handoffs, the data definitions and so on. And we see where agents might remove toil and we can see where apparently efficient changes might create new risk, new work or worse of all, a worse customer experience. So that makes everyone in this room experimenters and translators, because we can translate between the promise of the tool and the reality of our operating models.

It's a valuable position to be in, but it's also a responsibility. So I'm not presenting the latitude to experiment as as a perk, but as a responsibility, if we're given room to test, we need to use that room carefully and produce learnings that help other people. Think of it this way.

Imagine that you hired an exceptionally capable CSM. On day one, you hand them a complex customer portfolio and they read it quickly. They summarize what they find.

They give you they recognize patterns and they give you initial insights. They tell you what needs attention. And that's all genuinely useful.

But you wouldn't give them full autonomy on day one. You would still need to explain the context of your customer book. Check the data that they're working on.

Define the decisions that they make and show them how your processes work and give them a clear escalation path. You would review what they do. Before it shapes a conversation or a business decision.

AI and agents are exactly the same. Because they're very powerful, but they can't be completely plug and play. The capability is only the beginning.

The operating model that will make it trustworthy. And all serious deployments are going to ask the same thing that you would ask from a new team member. You'd ask them for clean, connected data.

You'd ask them to redesign a process, not just slap a bot over the top of one, like we heard Kate Neal talking about in this room yesterday. An honest understanding of the impact of the changes that they make. Allowing human validation checkpoints into the workflow and, of course, clear ownership.

And so, the defense of our role in ops is this, and it's not defensive at all. AI is a very capable teammate, but it still needs context, boundaries and reviews set by you. Everything that it does needs to be integrated, validated and made safe.

Doing that work is not overhead on top of AI. It's what turns agents into agents. It's what turns agents into value for your business.

And so, the way that I've learned to think about my role or the role of people in this room is that we can turn those experiments into something that other people can rely on every day. So, I'm not trying to prove that every idea that I have deserves to win. Most of my ideas do not deserve to win, but I'm trying to create a fair and disciplined way for each idea that we have to teach us something valuable.

Some experiments should stop, some should change completely, some should scale, but the job of us is to know which is which. So, that's where the framework comes in. The sandbox asks, where can we play and where can we do it safely?

Scorecard asks, when will we know if it helped? In stewardship, we look at what we owe the wider organisation and the people impacted by the change that the experiment will make. And of course, scale.

Has it earned the right to take a place in our operating model? And then we return to the sandbox with better information to conduct the next experiment. And before we dive in, I want to add a practical tool for you.

This is the experiment charter. It can be a one-pager, but it should answer questions like, what problems are we trying to solve? What is the time box?

What will we measure? What would make us stop, refine and scale? And the charter is not a bureaucracy for the sake of bureaucracy.

It helps the experiment to not change halfway through to suit the purpose, because the original output is not what we had hoped for. It makes our assumptions visible. As I move through the framework, I'm going to use an example, a practical example of how a CS Ops team might design an AI-assisted renewal risk triage workflow.

So just have that in mind as we go into our first area, sandbox. Designing a sandbox can be split into these six areas. The first, state the problem.

Don't try to solve for the technology first. Start with your operational problem. Are CSMs spending too much time gathering signals before a renewal conversation?

Are the risk reviews inconsistent? We need to use AI more is not a problem statement. Reduce the time it takes to prepare consistent evidence-based renewal reviews is closer to a proper problem statement.

So define that first. Then look at your smallest useful cohort. Don't try to begin with every customer or every segment or every workflow.

That way, madness lies. Choose a cohort where the learning will be useful, but the risk is limited. It might be one segment or one region or one book of business or one workflow where the owner is clearly defined.

A limited cohort is going to give you enough room to observe what the process actually does before the consequences of that process become difficult to reverse. Non-goals, clearly define non-goals. And this is a good one because this will see you through as you move back from scale into sandbox.

A good experiment says what it's not trying to do. Okay, so in this instance, it might be that the agent that you've created will not contact customers. It will not make an autonomous renewal decision.

The agent will not override human judgment. Non-goals are valuable because they prevent the scope from expanding for the sake of your own convenience. Number four, a named owner.

The experiment should have an owner. The outcome should have an owner. They might be the same person, but they don't have to be.

The outcome owner will care whether or not the business or the customer result improves. The experiment owner will coordinate all of the testing. Without ownership, an experiment can be everyone's idea and nobody's responsibility.

Time box and a rollback plan. You have to give yourself a line in the sand of when you are going to evaluate and take action on the results of your experiment so far. In addition, the rollback plan is going to answer the more uncomfortable question of what if this was just a bad idea and it's making things worse.

If you can't describe how to turn off the workflow or remove the output or return people to the previous operating process, you haven't finished designing your sandbox. And six, human impact guardrails. Ask yourself who's going to be affected and what must remain unaffected by your experiment.

I have made this mistake. I was interested to hear Cara from Silverfort yesterday talk about how she had, there she is, hi, talk about how she had created something and the feedback from her CSMs was it's interesting, but it's not actionable. Think about that when you're thinking about who will be impacted by what you're about to do.

Don't experiment on people you can't afford to lose. Seems obvious, but it's worth repeating. It means don't experiment on your most at-risk customers, your most fragile processes, or on your most exhausted teams.

The purpose of a sandbox is not to make your experiment look small, it's to make the learning safe enough to be honest. And now let's talk about how we would score this. What value is bought by what you're about to do?

Adoption and usability. Are the CSMs actually going to use it? When you've set it live, are they using it?

Do they understand what it's for and why it will make their lives better? Can they find the evidence in the summary that's been output? I heard somebody, and I can't remember who, talking about this yesterday, being able to look for the evidence and cite the sources of what the agent has produced.

It's very important. Efficiency. Does the workflow reduce preparation time?

If the agent saves you five minutes, but it requires 10 minutes of human review to validate whether or not it was correct, that is not efficient. So think about those things as well as you're building out your scorecard and how you might score your experiment. Quality and accuracy.

Does the workflow surface useful signals? How often are the summaries materially incomplete? How often does the CSM reject what the agent has told them?

For an AI-assisted process, accuracy is not only a model question, it's also a data quality and workflow question. Customer and business outcomes. Can we connect the intervention to something that actually matters?

Can we say that by using our agent that we've got an experiment that we will have earlier identification of renewal risk, more consistent success planning, fewer avoidable escalations? Don't promise that a small experiment will prove a lagging revenue outcome immediately, but define the chain of evidence to the people who are interested. And finally, cost and economics.

Track the economics of your experiment knowing that if your experiment is successful, it will scale and that cost will become a more defining factor. The tool or the usage cost, implementation or effort, data preparation. Whatever you do, establish your baseline, use a comparison if you can, and define the decision date upfront.

If an experiment has no scorecard, it's not really innovation, is it? And a scorecard that only measures activity is not yet a business case. Stewardship.

In ops, this is a topic that's very close to all of our hearts. More governance feels good. More stewardship over ideas feels good.

Split this into four. Document the pattern. For renewal risk workflow, stewardship means recording things such as, it's not an exhaustive list, the problem that the workflow is going to solve, data sources it relies on, common failure modes, what happens if the data is missing or contradicts itself, the metrics and the decision thresholds.

It doesn't need to be a 40-page manual, this document, but it does need to be enough to be understood by another operator who might come in over the top and take ownership of your really wonderful idea that does actually make it all the way to the operating model. Make that ownership explicit. Who owns the workflow?

Who owns the data? Those questions could be answered differently. A CS ops leader might own the workflow, the CS revenue leader might own the outcome.

But if those relationships are not explicit, problems will sit between teams and that's never a good place to be in. Preserve that human judgment. Use AI to remove toil.

Do not outsource thinking. That means automating the boring parts, the data pulls, the summaries and the preparation so that humans can focus on doing their high judgment work that only they can do. In the example of a renewal risk example, the CSM should be able to challenge the summary, add context and explain why the interpretation from the agent is wrong.

And of course, socialize the learning. Stewardship means that the learning is going to go down to the people who are affected by the change. So that could be CS ops, it could be rev ops, it could be product.

But whatever happens, make that conversation not look at this thing that we built. It should be, here's the problem that we were trying to solve, the evidence that we gathered, the trade-offs we found, and here are the conditions under which this experiment is useful. That language makes it easier to work cross collaboratively between teams.

Stewardship is what stops CS ops from becoming a little sort of hobby place and a dumping ground for people's ideas, right? Our experiments should eventually make somebody else's day easier, not make our own lives more interesting. And finally, scale.

At the end of an experiment, you will have three legitimate outcomes. Stop, refine, or scale. Stop when the experiment no longer produces a meaningful signal.

Or if the workflow creates unacceptable risk. Or how about the problem was not important enough to solve. Stopping is not failure.

It means that the framework has served its purpose. Refine your experiment, refine that agent when there's promising signals, but maybe some incomplete data, that's okay, we can work with that. Adoption might be low, not because the idea doesn't have support, but because the data's incomplete, or the workflow itself, the agent is challenging to use.

The output's useful, but maybe not reliable enough yet. Or the economics might be wrong. These are all real scenarios that you should be prepared to answer.

Refinement of your experiment should produce a new, sharper experiment with better inputs, not an indefinite extension of the old one. And then scale when the value can be measured, when the controls are understood, ownership is clear, practice is repeatable, the economics are defensible, and the people affected by the change care, and they support what you've done. The decision to scale is not in and of itself a reward for having a really good idea, but it's a judgment about whether or not the organization is ready to rely on what you've created.

By means of a short reflection exercise, I'll ask you this. Think of one AI experiment that you're running, or that you want to run in your own organization, and think about these three things. What would make you stop doing it?

What would tell you it needs refinement? What evidence would you require before you scale it? If those answers are difficult to write down, then this is a useful exercise for you, because it probably means the experiment needs a clearer charter or a scorecard before you layer on more technology.

The goal of every experiment is not to make it succeed, but to make it useful. I do not have the perfect system for balancing speed, value, and responsibility in customer success operations. I still spend more time than I would like sitting with half-baked frameworks, incomplete data, and experiments that don't land.

But I've learned that the goal is not to make every experiment succeed, but to make every experiment useful to me and to the business. Useful because it creates measurable value. Useful because it teaches us what doesn't work for us, and useful because someone else can understand the learning and make a better decision next time.

The framework works because it's a loop. It's not a one-way street, and it's a practical way to define where you can safely play, how value will be measured, when the learning into something, turn the learning into something shared and governed, and expand only what is useful. So my invitation to all of you is simple.

Don't ask, how do we scale this AI idea? Start with, what's the smallest safe experiment that could teach us something really important about our business? Then define that scorecard before the excitement either takes over or the excitement completely wanes.

Name the owner before the work starts. Write the guardrails down before the launch plan, and be willing to stop it. Be willing to refine it.

Be willing to scale it based on evidence rather than just enthusiasm. That's how we move from sandbox to scale. That's how we turn experiments into shared assets, and that's how people who work like me in CSOps can help make agents work better and safer for the people around us.

[Speaker 2]
That was great, Polly. Thank you. So we'll go over some Q&A.

If you would like to ask a question, you can put that in the, I think, polls and questions section from the homepage of the app, and choose track three. All right, so what was the most viable experiment up to date that you have deployed?

[Speaker 1]
Valuable to me was a loss analysis agent. One of the areas that we were trying to overcome was to be able to do a blameless loss analysis when we did have a customer that churned, and that was very difficult because we found that there was a lot of emotion and a lot of, of course, investment in customers that did churn, and it was difficult for everybody involved to sit down and say, what could we have done better next time without everyone taking it personally?

So I would say that's the most viable experiment and valuable experiment to date that I've deployed for the business, for, sorry, for personally, but then there was also another one that I created an agent that scoured all of our case studies, looked at why those customers gave us a case study, and then reverse engineered it back into our product usage to look for other customers that were finding similar results, and then be able to create those outcomes for the next transfer customers.

[Speaker 2]
That sounds really good as well. So what does the escalation matrix look like in case a successfully deployed workflow goes wrong?

[Speaker 1]
Yeah, so, so important, and that goes back to clear ownership of your, of anything that you do. It, you need to have an escalation path. That's one of the things that, that an agent in and of itself won't necessarily know what to do.

The escalation matrix would look different in lots of companies, but for us, it's very clear. We're able to say who the owner is. They take the, they take the first, they take the first pass at a fix, but we're also able to identify who are the people that can help us to roll back what we've done.

[Speaker 2]
Perfect. How much evidence do you guys look for before turning a pilot into kind of a standard workflow?

[Speaker 1]
We tried to answer those questions. Is it valuable? Does it make a material difference?

Is it defensible? Can somebody, is some, can somebody pick it up if I was to go under a bus tomorrow?

[Speaker 2]
I like to use won the lottery, and buys a desert island, not hit by a bus. Though, as an American here, that is a likely thing if I look the wrong way.

[Speaker 1]
Yeah, do be careful when crossing the street.

[Speaker 2]
I look all the ways.

[Speaker 1]
Yeah, so how much evidence is up to you, and really very much up to how, how much trust there is in your department, and the idea that you're working on. But evidence, evidence in and of itself is an important thing to have. How much is entirely up to you.

[Speaker 2]
I'm gonna skip for a second, because I think this is a little bit of a follow-on. But what advice would you give to applying your framework retroactively? So for folks in this room who might have projects already in flight, but they're stuck in that refinement phase.

[Speaker 1]
Yeah, that's interesting. You can apply the framework to projects that are already in flight, simply by going through and answering some of those questions. But if it's stuck in refinement, ask yourself, do I have the right stakeholders at the table?

Do I have the buy-in to make this more achievable? Do I have the executive support? Does my manager, or whoever it might be, understand that what I'm trying to get to as a large concept is to make somebody's day better, to save us money, to make us more efficient.

And look for that executive buy-in. But apply those questions. And if it's stuck in refinement, maybe the question you should be asking yourself is, should I actually just stop?

[Speaker 2]
Yeah, that makes sense. So what exactly is your sandbox? Is it test customers, or test environment, or real accounts, that small cohort you were mentioning?

[Speaker 1]
My personal sandbox is, it's real accounts, but it's not live, right? So we will take like a download of customers, we'll take like an output, and we'll say, right, this is how things are at this moment in time, and use those as our sandbox.

[Speaker 2]
How do you frame the time and effort spent on these experiments to leadership? Leadership when there's pressure to really deliver value with AI in a short amount of time.

[Speaker 1]
How do you frame the time and effort spent on experiments to leadership?

[Speaker 2]
Basically tell leadership to slow their roll when it comes to AI, I think is the question. My leadership's in the room, so I need to be. What advice would you give others?

Not necessarily to your leadership, but if someone is struggling, and I'm sure you're not.

[Speaker 3]
I'm not.

[Speaker 1]
Look, I think every organization out there is ultimately trying to answer the question, how can we be more efficient by using AI, and how can we deploy agents to do some of the heavy work? Ultimately, your leadership doesn't want you to mess about with things that are really real and essential to the company. I would recommend having an honest leadership conversation about how these experiments are not an add-on.

I heard somebody talking in the keynote this morning about we'll just vibe code at the weekends and at night. No, in ops, we need to be able to carve out that time to do the important work. Right?

I think more and more companies are understanding now that by saying we need to make efficiencies, and we need to deploy agents, and we need to make as much use of AI as possible, that comes with the trade-off of, great, everybody's learning at the same time. We're all learning together. There is no AI expert out there, unless you're in here, in which case, raise your hand.

There's no one person out there that has all the answers. You all have the opportunity to make yourself that expert in your own organization. If necessary, make it a personal goal.

Make it a career goal. Tell your leadership that this is the thing that you want to work on this year. They'll support it.

They'll 100% support it.

[Speaker 2]
I think, also, you mentioned in your presentation about identifying what the risks are. I think if leadership is pushing for speed, you have to be honest about what are the risks if this is not done correctly or fully vetted.

[Speaker 1]
Absolutely, but whoever asked that question, come and find me afterwards, because I'd really like to dig into what it is that you're actually trying to solve and see if we can have a more in-depth conversation.

[Speaker 2]
It sounds like this group would like some more examples of AI agents or workflows that you've tested through this framework. Sounds like the ones you mentioned earlier were of interest. Any others you want to share?

[Speaker 1]
Most of them didn't get past the... Tell us which ones didn't get passed.

[Speaker 2]
Yeah, I think that's valuable as well.

[Speaker 1]
Well, now, let me think here. Oh, yes, there was one that I tried to see if I could bring everybody. I wanted to be...

Not everybody in our company has access to GainSight. That's probably the best place to frame this. I wanted to be able to pull health scoring data into Glean, which is, we use Glean, you might use something else, we use Glean, and be able to ask Glean what is the customer health of XYZ.

What I ended up creating was that nonsense output that we heard about in the keynote this morning, because the data was massively out of date. The product data was not... Yeah, it was just, it wasn't there.

The stakeholders were two years out of date. And so I thought going in that I was gonna be creating something really valuable for everybody, bring CS to everybody in the business, I thought. And what it did is just create confusion and upset and more anxiety and more overwhelm, because what they saw was, well, this customer's fine, but they've told us they're churning.

Really challenging.

[Speaker 2]
That's obviously a problem. Yeah. Makes sense.

I think that's helpful to understand or to know what people try that doesn't work.

[Speaker 1]
Lots of things that I tried don't work. I mean, that's just, I mean, you could apply that to my whole life, actually, but like in this concept, in this context. I think if everyone's being honest, yes.

Most of the things that I tried don't work. And I've come unstuck. Like, there's a reason why I'm presenting this framework to you, because I would just sit down, crack open Glean, and think, right, what can we do today?

And you end up spending hours with no structure and nothing really valuable to show from it. So that's why the framework works.

[Speaker 2]
Love that. So how do you think about the potential costs of tokens, usage, whatever it may be, before scaling things too widely?

[Speaker 1]
Yeah, it's really interesting. When we first started to deploy AI and particularly Glean, we were fortunate to be on a, we were grandfathered into a contract where we sort of had like unlimited amount of credits. People are nodding, right?

So, okay, I guess we're not the only ones. And when I realized what the cost, how many tokens it took just to answer one simple question, I was horrified. So, but that is the way that we're moving forwards.

We have to bake those costs into our operating models. That has to be part of the P&L in every business now. And we should ask ourselves, is the risk worth the reward?

Is the juice worth the squeeze? What we get out of this experiment and the agent that will sit in our operating model in the future, is that going to save us money, put more money in our pockets ultimately? Consider it.

You have to consider the economics. That's in the framework.

[Speaker 2]
Basically answered the next question. Have you, I think this is interesting. Have you seen a successful experiment create unintended work kind of somewhere else in the business?

Oh, I love this.

[Speaker 1]
Yes. Yes, about two weeks ago, actually.

[Speaker 2]
Wonderful, tell us more.

[Speaker 1]
Well, okay, so I connected three systems and it went, she's laughing.

[Speaker 2]
We need a guest up here.

[Speaker 1]
And I thought that I had done a brilliant thing. I thought that this was great. And I had, and it had been, hey, it had been switched on for like three months and nothing had gone wrong.

Until another part of our business turned Gemini Notes in Google Meets to on by default. Skipped to the end. Internal documents ended up being posted to the timeline of one of our customers in GameSite.

Right? Nothing, like we're not talking like data breach or anything, but it was just in the wrong place. We were like, what's that doing over there?

How has that got there? And I got a phone call. I'm sorry that this is a fire drill, but we might actually have a fire.

Actually, as it turned out, I'd connected the wrong system via GameSite, via Glean, via. And so there was a lot of that, what I mentioned earlier, like how do you remove things from one place and put them back where they're supposed to be? Not only did it create unintended work for me, it created unintended work for three other people in the business.

So I'm fortunate at Mediafly, we're a very forgiving group of people. But yeah, it does happen. I don't wanna say these are the trade-offs because they're not supposed to be the trade-offs, but don't be too hard on yourselves if those things happen.

Rely on the framework, keep going back to it. And hope that your boss is really supportive when things go wrong.

[Speaker 2]
So if you have an experiment and it succeeds, the results are kind of what you expected or better than, but the owner that you've identified doesn't get buy-in from either their team or others, how do you handle that?

[Speaker 1]
I mean, I generally work very closely with our CS teams. And so I don't try to start something until I think I know who will be the owner and who will eventually benefit from it. And again, Kara was talking yesterday about working collaboratively with the CSMs. And that works across the business. But ultimately, if the eventual owner won't buy into it, you are allowed to retire it. You can say, we tried that, no one was interested in the uptake, we'll put it in the parking lot and we can come back to it again.

[Speaker 2]
Makes sense. Final question, just due to time. What does your tool set look like for your experiments and agents?

So you mentioned Glean, sounds like Google.

[Speaker 1]
Glean, Google, Claude. But honestly, everything that I'm doing in my role somehow ties into Gainsight. Like, you know, small plug, but I'm really excited about the agent studio scene.

Like that's gonna make things, like that's game changing stuff. That's gonna make my life much easier. I will use anything that is available and is realistically going to, that I think is gonna help solve my problems.

But it's entirely up to you. But I think everybody that's here has got access to something, even if it's Google Gemini or something basic. Even if you don't have Claude, that's still fine.

Like you can still do an awful lot with your day-to-day tool set.

[Speaker 2]
That makes sense. We're gonna go ahead and leave it at that. So there's plenty of time to get to the next session.

Polly, thank you so much. That was wonderful.

[Speaker 1]
I'm so grateful to all of you for indulging me. Thank you so much.