How to Build a Defensible Certification Program That Won’t Break The Budget

[Speaker 1]
First of all, thank you. I know it's the end of the day. I appreciate you sticking around and coming to this final session, which is about, I guess, a very specific area of customer engagement, enablement, and education in that sense.

So my name's Joel. I work for a British software company called Hornbill. We are a B2B company.

We sell a SaaS solution. I won't bore you with exactly what we do, but if you've ever been told by a colleague at any company where you've asked for help or you've requested something, and they've said, no, no, go and raise a ticket over there, that's the kind of system that we build. So IT service management.

That's the type of technical solutions we offer. And a couple of years ago, I was at Pulse, presenting at Pulse with a colleague of mine. Back then, we were kind of right at the beginning of our journey for customer education and customer enablement.

And the topic of that session was all around how to build a customer education function from scratch and on a budget. And fast forward to two years today, this is kind of almost like one of the next steps we've taken in that journey. We are still a small team.

Granted, there's more of us now. We are definitely still on a budget. But what we are doing, and I guess one of my main passions about my mandate for customer enablement is really finding ways that we can drive value for our customers.

So with our initial, I guess, academy enablement content community, I guess, areas of focus, we really dived into some of the low-hanging fruit, some of the things that the business and customers were screaming out for. So support for onboarding, support for our partners, support for sales enablement. But one of the areas that I was really interested in seeking value for was from our customer perspective, not just for their onboarding, not just to help them in their day-to-day jobs.

We run monthly virtual instructor-led hands-on sessions. Our customers love it. They get to turn up, forget about their day-to-day.

They learn about a topic, and they get to break an instance that doesn't have their customer data in while we give them activities and challenges. And we run events like this twice a year, where we bring customers together, and we have days of, we call it a day of learning and connecting. We have more hands-on practical sessions, theoretical sessions, customers presenting to each other.

But one of the areas of value that our customers were kind of not necessarily screaming out for, but they were lacking, was how can we help our customers in their careers? How can we help them demonstrate their value with their managers and in their companies, where they're often being asked to do more with less, automate things? How can they then turn around to say to their, I don't know, employers or prospective employers, if they have to move on into the job market?

So this idea of certification was on my mind for a while. So today I'm going to talk about how we built a defensible, and I'll get to what that means shortly, certification program for our customers. So just a little bit of background about me.

I've worked in the tech space for a number of years. I've overseen or been accountable for technical professional certification programs for all of the companies on here. And in terms of size and scale, if I put Hornbill's logo to match the scale of those other companies, you probably won't be able to see it.

So one of the big things that happens when you build certification programs is in most companies, you offshore that. You outsource it to psychometric agencies, to companies that specialize in that, and with good reason. So I have bought the expensive versions many times, and often that was a case of just signing the invoices and rolling my eyes at how much those invoices would cost.

But the beauty of working in smaller companies, where you don't have the budget for that, is that you get to take what you learn from other places and cherry-pick the things that could work for you, and you get to deploy them in a way that fits your customer base. So the things I'm going to talk about today, it's not a new process, it's not a new way of working, but it's what I've learned from years of having worked with different vendors, building certification programs, overseeing them and setting them up. And I've tried to just distill it for what could work for a very small team, but still is a rigorous, defensible certification program.

As an example, I worked at one of those companies where we would spend 150k a year on one certification program, on one exam. And that's not uncommon, and that excluded external costs like our internal program managers, subject matter expert time, working with customers to validate what we're doing. If I were to turn around in my current company at Hornbill and ask for that to build one exam, I'd be laughed out the door.

But there's a reason why it costs money, and it's not all just smoke and mirrors and something that happens in a black box, although it very often is. And it's quite hard to find out how these certification programs are actually built, what rigorous examination actually contains, because they are run by experts. They are done in a programmatic way.

And I'm going to kind of walk through a little bit about that today. So we're going to start with a quick poll here. We've got that up on the subject of this.

But this first question is really, hands up, quick show of hands, who here is working exclusively in a customer education role? Oh, brilliant, we've got a few. Who's working in an area where customer education was in their remit, but maybe not exclusively their role?

Great, others? Not sure? Depends on what you get told on Monday.

Okay, so question here. What is stopping you, if you haven't already, from building a certification program? That's the one.

Thank you. Great. Okay, some answers coming in.

Time, time cost. Yeah, where to start exactly? Not sure which skills to include and what level to test it exactly.

Again, where to start? There is not currently, to my knowledge, an off-the-shelf guide of how to build a certification program. What I've done today is I'm not going to talk about every step that we did at all.

We only have half an hour of me talking before some questions. I've actually put together what I've done into a guide that you can download and access. I'll share a link at the end.

I do not ask for any information. You do not have to give me your email address. I won't be using any data to train any models.

But yeah, I love this. Imagination. The slow match of time.

Skill jar. Knowledge. Scalability with a lean team.

Scalability. Yeah, this is basically all the same questions that I had when I started this journey at Hornbill. I'd like to point out that we use at Hornbill, Hornbill, sorry, Gainsight Customer Education.

We started on that journey when it was North Pass, and we're still using it. And we deliver our certification exam through that platform. And I'll talk a bit about how we do that shortly.

Okay. Before we look at where to begin, I want to just give a bit of an overall definition of a couple of terms here that are going to help us go forward. This idea of a certificate and certification.

Now, these two terms get used kind of interchangeably a lot, but for purposes of today, I want you to distinguish between the two based on a certificate being something that you get at the end of a learning experience. So, we've all done this. We may have signed up to use a new tool.

We may have gone to an academy. We've done a course. Oh, wow.

I've got a certificate at the end. Brilliant. Asked a little quiz that tested if I remembered the stuff they told me two minutes beforehand.

That is a certificate. It's a validation that you've completed something. Certification is kind of at the other end of the spectrum.

This is something where we are validating someone's competence and we're certifying their ability to do their job. Now, if you were to think about it in a couple of different ways, on one end of the spectrum, we call it low stakes. What are the stakes involved for someone to complete this exam, this test, this experience?

So, something like I said a minute ago, but it could be more of a marketing venture. There's nothing wrong with that at all. It could be a bit of using this initial exam, learning experience test for some lead generation.

What's riding on it for the customer is, well, not much. It's easy to attain. Their job's not at stake if they fail it, perhaps.

When we talk about high stakes certification, at the other end of the spectrum, this could be where someone's job, promotion, or profession is on the line. This is very apparent in highly regulated industries, finance, airline safety, for example. You don't send your pilots on a course where they just have to take a 20-minute e-learning course and answer a few questions.

It has to really be validating their proficiency to do their job. So, this, essentially, with that, going back to that kind of 150k a year for an exam, this was roughly the breakdown of what that would actually cost. The majority of that was really on something called item writing.

Item writing is the question and answer writing. Actually, writing the questions and answers. This is a very time-consuming process.

And what is called in the certification world, JTA facilitation, Job Task Analysis Facilitation. I'm going to come back to that one in a minute. Some psychometric consultancy and standard setting, nice, very ambiguous term there, is applying a bit of rigor to the questions that are written to grade them and standardize them.

Evaluating the exam and monitoring the performance of the questions, because this needs to happen after you just release the exam out into the wild. You've got to see how it performs. Does it match your expectations?

Is everyone passing it? And going back to that low stakes, high stakes perspective, a good way to determine, a really easy way, if you're not sure which end of the spectrum your certificate might land or certification might land, is that ask yourself, do you want everyone to pass it? And if your answer is yes, that's a low stakes certification.

And if the answer is, well, absolutely not. We can't have everyone passing this. It's meant to test some skills, some knowledge, some rigor, then that's a high stakes.

Outsourcing your certification programs doesn't necessarily mean those exams are going to be hosted on a platform. You may still have to do that yourself. But proctoring is something that needs to be taken into account.

Proctoring is the idea of, remember, in an exam hall, if you ever went to college or school, university, you write your exam answers on a piece of paper and there is someone watching that it's you writing them on a piece of paper. How do we do that in an online world? How do we ensure that no one's going, yep, yep, put that into Google, put that into Claude.

What's the answer? Yeah, our LMSs out of the box don't really have that built in. You can buy proctoring software.

You can even buy AI-based proctoring software. But again, that's quite expensive. And another thing to set up and manage and integrate with your systems and solutions.

So we came up with a really kind of cheap and cheerful way of embedding that to add some rigor. And I think with our Hornble exam, we wanted to start off at what we called our professional level exam. And we determined on that kind of spectrum, this is a medium stakes exam.

Someone's job is not on the line if they fail it. They may, it may contribute to a promotional cause. But we wanted to add value to our customers who are saying, all the stuff you do is helping us do our jobs.

But when our jobs are under threat, when we need to show value to our managers that we are continually learning, what have you got for us? We have quite an even split between our public sector customers and our private sector customers, which really informed the design and the approach we took with this. We don't charge for our certification.

Many organizations do, and there's value to doing that. It puts a value literally on it. When I was at Amazon, our AWS technical certifications were, they're considered the most, probably the most valuable technical certification program in the world.

And it's heavily invested in. Even the LMS that has the practice materials on Amazon, we would spend two million a year on just to host that, just to license that LMS. So it depends on what your motivations are as a company as to why would you even want to go down this route, because it is quite involved. This is very true.

And this comes back to my earlier point of, when you are paying all that money and you are paying for all of these things, you're paying for a reason. And it's not just paying for someone to blame, it's paying for the accountability of all of those things put together. And sometimes, particularly in a heavily regulated industry, that's not a bad thing.

In fact, that can be a very good thing. Okay. So this is the nine-stage process that we undertook to build our certification program.

A colleague of mine, some genius, recently said to me when I showed him this, you know, different guys, that's a lot of steps, Joel. Why did you not think about using AI to automate a lot of this? Which is a valid question.

But my answer was, you know that line that we keep getting fed about AI all the time, to get it to automate the easy stuff, so you can actually get on with the strategic stuff? This is the stuff where you get to use your brain and to really dive deep on the different areas. There's things within here that you can use AI to help speed things up.

I'm not going to talk about all nine of these stages today. That's far too much. And some of them are a little bit self-explanatory.

But I'm going to pick a few of these. I'm going to start with these first two. Because getting these right and spending the time on these underpins everything else.

The rest is a bit like painting by numbers, once you do these two. Not only that, the flip side of getting that first one right, unlock the door for our entire academy roadmap going forward. And I'll come back to that.

Okay. So, the first thing we are asking is, right, who are we testing? I spoke a little bit about the motivations of why we were doing this.

But we really need to understand who we were testing. And I think someone put that in their initial challenges to make sure that we were actually testing them correctly. So, a job task analysis is the exercise of trying to determine what someone in the job that uses your product or does the things that you promote them to do and enable them to do, what do they do in their job?

And it really is a breakdown of these four things. We start by identifying what are the objectives? What do they have to do in their job roles?

And they can be quite broad. Manage system security. Okay, right.

Great. What does that mean for this person in this job role? So, every objective has a series of sub-objectives.

And in order to do the sub-objectives, there needs to be a combination of, okay, what tasks do they need to be able to do? And what do they need to be able to know to enable them to do those tasks correctly? And I think the guys earlier on in the last presentation talked about this idea of, you know, where the difference between learning, the difference between telling people how to do stuff.

This really maps it out and breaks it down into those areas. Because with education and learning, we're teaching concepts, not just clicks. And once we've mapped this out, the next stage of following on, once this has been validated, to be able to write questions that test these things is actually really easy.

The process kind of does itself. It's kind of telling you where to start with a question once you break it down. And for example, a sub-objective, it may have more than one task, may need more than one bits of knowledge.

And that allows you to write the questions and the answers to determine that someone knows or can do that. So, this job task analysis process probably took the longest of any of those nine stages. And it started out from us, our team, from building out a structure, a framework.

And we worked with internal subject matter experts first. And then we brought in a select few of our customers to validate it. Now, typically, these JTA sessions, I've seen these happening over the series of a week.

Getting customers together, putting them up in hotels, spending a whole week to define what does the perfect candidate do across all of your companies that we can genericize. We didn't have the luxury of the time or the engagement to do that. And our customers, I would have had to spend five days just explaining what we were trying to do before we got them to do it.

Because this would have been so alien to some of them. So, it started from us. We validated it internally.

But there had to be flexibility because we wanted it. The value came from having it validated with our customers. So, we had a small steering group of customers who contributed and fed back on what we were doing.

But it really threw up some interesting questions and debates. I think if we even automated some of these steps, it would have gone a bit missed. So, one of the things that really came up internally from one of our subject matter experts was, right, we need to be able to validate, we need to be able to test people that they know how to engage with our support services properly.

Because we know a lot of them don't. So, this is definitely a part of their role that they should be able to do, regardless of what company they're at. But particularly for Hornbill, how do they engage with Hornbill?

The problem with that was, when I looked into that, I couldn't find this anywhere for customers to actually determine. The reason it was hard is that we hadn't communicated it properly to customers. And this showed that we didn't have to have every answer to every question and every task written down in a document or in our knowledge base or in the academy, but we couldn't be tripping our customers up.

And that really helped me find something that we should be doing better as a company to enable our customers in the first place. To go back to my earlier point to reinforce this, the last thing we wanted to do was to write an exam or write a certification on the basis that tests the product. We're not here to test the product.

We're here to test people's jobs. So, it's that click versus concept again. So, again, this was another question that may have come up when we got to the item writing section from one of our subject matter experts.

And it's the case of, no, no, this is not what we're testing at all. Because we all know that could change next week without us knowing. We could have someone doing an exam going, well, hang on, this isn't right.

Last week it was when I checked today, this was something else. So, having that mindset of we are not here to test the product, we are testing the people, really helps you distinguish between the motivation for doing it. Okay.

We've talked about the job task analysis. Talking about what does someone need to know and do to function in their job. But on the flip side of that is who is the minimum qualified candidate to take this exam?

And this does two things. It helps you determine and push this every time you need to in front of a customer, in front of a subject matter expert when you're going through those item writing exercises. But it also allows you to be up front with your customers and say, should you be doing this exam?

Should you be really entering into this? However, two hour long exam we're going to put you in because you're going to get quite annoyed if you realize it wasn't for you. So, for example, what can they do unaided?

What can they do but they need support and what are completely out of scope? And this allows you to, even when you do that job task analysis and identify things that they might need to be able to do, well, actually for a minimum qualified candidate and for our professional level exam, we recommended it was for people with a minimum 12 months hands-on experience. We recommended that they finished the foundational training.

Didn't give them all the answers because a job competency based exam is based on the things that they learn and can do in their job, not just based on the stuff they do in our academy. Okay, we've jumped a bit forward in the map here, but this is the interesting point. Okay, we've identified what people should be able to do in their job.

We've broken it down into the tasks they need to do, the knowledge they need to do. How on earth do we turn this into an exam and how do we determine the pass mark of the exam? And again, I've looked.

There's nothing easy and obvious online that tells you how companies can do this. We did two things with our subject matter experts. Once we'd written the questions without any thought of, okay, is this a really hard question?

Should we be asking this? Are we testing the right level? We wrote the questions that mapped the tasks and the knowledge that they needed to know.

And then we asked our subject matter experts to give each question two scores. The first one is, how important is this question for this person in their job? Is it essential?

They need to do this. If they can't do it, they're screwed. Is it important?

Yeah, they should be doing this. They won't need to do it every day, but it's important. Is it a nice-to-have?

Yeah, that'd be great. They're probably a bit more advanced maybe, but they won't be doing it all the time. It's a bit niche.

The other thing is, how difficult is this question? Not in terms of how advanced is the user. It's just, is this easy to understand?

Have we made it a bit hard? Is it an easy, a medium, or a hard question? And that gave us two scores per question.

So this was our kind of phase one for getting our subject matter experts to validate the questions and that we put together against that job task analysis. So for example, we weren't judging the scope of this, these questions over the tasks and knowledge. We'd done that.

We were confident in that. We had buckets of questions based on their importance and difficulty. What this allowed us to do much later on, is when we were monitoring how well the exam was performing, if we looked at a question and said, oh, everybody has got this right.

Is this an easy question? Is this a hard question? What was it marked at?

So we weren't going back to rewrite the questions. We were going back to see if it needed regrading. So likewise, this was more about the comprehension of the way the questions were written and the importance to their job.

The second stage, there's no Slido poll, just so you don't want to get tripped up again. So we got our subject matter experts to grade each question on those two things. And then we asked them to do a third score as well once we'd done that exercise.

What score would you give for our customers to get that right? What's the probability from a scale of 0 to 4, from 0 to 100 percent, do you think the minimum qualified candidate will get that right? So a good example here is, I'll put a question.

How many out of 10 candidates with, I don't know, three to six months experience get this right? So we would have this on one hand and the two questions we asked. And that's all we asked for our subject matter experts to validate.

Yeah, obviously we had to check the technical accuracy of the questions, but we managed to do that ourselves internally. This was to take a more objective stance that helped us put together the questions and determine how the exam was structured. So this was how we determined our exam pass mark.

We had those banks of questions in the matrix, in the different groups, in the different bands, and we took the average predicted score per question. So if it's 75 percent, 0 percent that we just saw, times it by the number of questions in that band, average that overall, or waited for the different number of questions in each band. And that was the expected points for the minimum qualified candidate that we thought could get right.

And that was our exam pass mark. So we didn't have to sit there going, well, should it be 60 percent? Because we think we put some easy ones in there.

We think we put some hard ones in there. It allowed us to choose. Look, for this professional level exam, we decided that we wanted, let's say, 10 questions from this band, 15 questions from this band, more questions from the middle ground than the outliers.

And this gave us the pass mark, not debating that. Well, this also meant that we had a bank of questions where no two exams are the same. So on the exam form, so every time someone loaded up the exam, we had these nine different banks of questions, and it would take 12 at random from easy essential, 15 from random, from easy useful, and so on.

So this, again, made it defensible. One of the key things about a defensible and slightly more higher stakes exam is that it is repeatable, but it is the same every time. Okay, I talked about proctoring earlier.

How can we be sure that no one is cheating on our test? How can you be sure that if you, well, anything we put online, anything you put out there, we know that somehow or another, someone can take it. You can take a screenshot.

You can put a camera down here. You could, people can cheat wherever you are. So it's trying to gauge that risk management, risk-managed approach of how deep do you need to go.

When I was at AWS, Amazon, for example, in AWS, we had a whole team, and their job was to scour the web to find out where people were posing questions and answers from our exams. Maybe that's probably done by AI these days. But this helps ensure fairness as well.

We want someone to come to our exam, and we want them to think that this is fair, and we want to, that no one else is cheating, even though they are spending a lot of time and effort into the preparation for their exam. So we applied quite a simple rule to this. We would host our exams in Gainsight Customer Education.

That's where the actual questions and answers were hosted. It would create that randomized exam form from the different bands of questions every time. But we would say in our materials that this is a proctored exam.

You may be, or you will be tested. And we caveated our exams. We would host them online with a group of people on a virtual call.

We would have about 10, 15 minutes of introduction, a couple of example questions, setting the scene, telling them what is going to happen. And we would say, right, during this exam at some point, someone might jump on a call with you for 60 seconds, and we're going to start asking you about the question you're on, and we're going to start getting you to talk through your answer as you do it. Okay, that's not checking everyone's not cheating at certain points of certain times, but it's giving people an understanding that they are getting checked on.

And there is a barrier to, okay, well, maybe I won't be having a separate screen open at all times. And we would ask them to share their screen while they did that as well. And that was an expectation.

Okay, so this process looks great internally because it cost us nothing. Yeah, I could say to my CEO, I've delivered you an exam program that would have cost you 150K for zero. Obviously, that is a lie.

Yeah, it cost my time. It cost the time of our subject matter experts, the customers in our steering group, the time taken to build the exam in the academy, test it, roll it out, the comms, the marketing. It's not a cost.

That's zero. It took in total from start to finish, I think four months. And it wasn't my full-time job.

But it was more a case of building out that job task analysis. That was the biggest, I would say, task in a sense. And that's something, unless you have it in place already, you can't get AI to build that for you.

Well, you can ask it to start, but where's it gonna get that baseline information from? But what those days actually brought, what all that time it took, all the time that we put into it, this is kind of what it gave us. So obviously, it gave us an exam.

We've been running this exam now for almost a year and a half. So we are still, I think we've had one round of updates to it because again, we're not just testing clicks. It gave us what we call a competency matrix.

And this is something that we'd started a bit when we started launching our academy, but we didn't really have time because we were too busy doing onboarding, we're too busy supporting sales enablement. But that job task analysis gave us what in learning, the learning world is known as a competency matrix. It is a breakdown of all the different tasks across domains for job roles.

And we were able to, from our job task analysis, to break down, well, actually, what are these tasks? Where do they sit on a level basis? And this is a beginner task, an advanced task.

And that competency matrix has kind of underpinned our content strategy ever since. I say to my team, if you're building content for the academy and it doesn't relate to our competency matrix, then something's gone wrong. The other benefit of this is now we are building our more specialized advanced certification.

So we kind of did quite a generic Hornbill administrator professional level certification. But because we, I guess, our product goes in different directions at the different, more advanced levels, we've been able to use this. And it's been the blueprint for what we've done for our advanced certification.

And that is a more hands-on, practical, assessment-based certification. And it feeds off nicely from our initial one. Okay.

It didn't all go smoothly. There was one area that we didn't pay attention to as much, or I didn't pay attention too much. And this is the one thing that we've done differently.

From now. We had one candidate, I found out, sat it three times. And this only came up in an MBR.

And I started digging into what had gone on. They failed three times in a row. And we have a certain amount of period where we say, well, if you fail once, you have to wait a certain amount of months till you can do it again because you want to go back and study.

But what we had done, we'd never given them the feedback to help them succeed. We never told them what to focus on. Because the customer education platform is not set up for exams.

What we could do is tell them the questions they got wrong. But we didn't want to do that because they were just, okay, well, I'll just answer it the right way next time. All we ever gave them was a number.

This was your score. What we should have done and what we do now is we can go back and look and say, well, these are the areas that you need to go and focus on. Because this is where you're lacking.

So that was a blind spot on our side. We were more looking at the performance of the questions. Remember I said, we don't have to go back and write the questions again.

We know they test the skills and the knowledge needed. But maybe we got them slightly wrong in terms of how many of our customers would get them right. And we regrade the questions.

We didn't look at the performance of the individuals. So where we're at today. This is our results from that one exam.

We've been live for a year. We launched it last summer. So a year and a bit.

We have a 74% pass rate of all our customers who've taken it to pass. The actual exam itself has an 80% pass rate based on that simple equation from earlier on. Between 70 and 80 is the sweet spot.

And I was unbelievably surprised that it came out at 74. It resounds quite nicely. I think it was at 70% is, you know, a distinction or a first in our education days.

One of the best things that we've done is we have managed to get 20% of our customer base of our administrators taking this exam. And this has not been an easy feat. We've done a lot of work engaging them, trying to get them to see the value.

Also worried what happens when they fail. We want people to fail, but it's not nice to have that conversation when they do. This to me is one of the, I don't know, the telltale times of a successful program.

20% of those customers have sent someone else. 27% have sent someone else from their organization to come and take the exam. And it's not always been, oh, the next time we do the exam.

They've said, look, I've worked towards this. This is something that you can do as well. And it's that aspirational, adding value to people's careers and their workplace.

And that's kind of almost the intangible things outside their day-to-day. What I did get asked, right, Joel, how's this helping retention? I didn't even look.

I haven't looked because this was never the intention of this. This was never to help our customers stay with us. We know already, and we can show it, that well-trained customers, well-engaged customers stay with us for longer.

But there's also so many forces at play at the moment. So many, we've seen churn with customers who are so well-engaged, have such great training coverage, engagement with our professional services, and these decisions have nothing to do with that. So I wanted to focus on the value that we wanted to add in the first place.

Okay, so going back to what we started with, with the poll, where we finally got to the right question, let's think about what are those things holding you back from certification? Time, yeah, it did take time to do this. There was a lot of time spent.

I know that exam inside out. To be honest, I don't. It's been a year.

It's all gone from my head. But at the time, I knew that exam inside out. I did not want to trust AI doing anything with that.

Knowing where to start. Okay, again, I was exactly there. Doing that job task analysis, working with a couple of customers, a couple of subject matter experts to start with, and identifying what people need to be able to do and to be able to know in their job roles.

And we weren't just testing what to do in Hornbill. It was how to think strategically in certain aspects of their job. I'm hoping there were things in that session that may have addressed some of those things today.

So my last thing is if you want to get the map, get a bit more information, there's a QR code. You can download it. I don't ask for any details.

You're more than welcome to steal it, use it, hack it, or do what I do, take the bits that work for you and apply it to the way you need. Great, thank you very much.

[Speaker 2]
Thanks, Joelle. Maybe also good to know, so after this whole Pulse event and within a month or so, we're going to also publish it in our library together with also your presentation, so you will have this at hand yourself. We do have four, four and a half minutes left for Q&A.

I saw a couple of them trickling in, so we will run through them. Let's start with the first one. How would you link certification to customer outcomes like renewal without over-claiming?

[Speaker 1]
Yeah, I think I answered that at the end, where we kind of don't, but we do link our academy. We link it to HubSpot, so our customer success teams can see who is certified, who isn't, what training has been done, what training hasn't.

[Speaker 2]
Yeah, and a couple of them speak for themselves also in the last couple of slides, like 27% of the organization said like, let's send someone else also for the certification program. The second one, how do you keep a certification current as the product changes? Very good one.

Without rebuilding the exam each time, do you use MCP? Would love to know more about your MCP use cases as it relates to the certifications.

[Speaker 1]
Great question. We haven't used an MCP in relation to our certification yet. We have used AI in relation to our certification.

We have pointed AI at our release notes and our documentation over the last however window and pointed it also to our exam questions and asked, where are the gaps? That was a starting point. We have also manually audited those two questions 205 questions in our bank as well, which doesn't take as long as you'd think, but we have connected the two, not with an MCP though, we haven't got there yet.

[Speaker 2]
All right, thank you. Where do customers track their certification? Is it the community?

Is it the learning platform?

[Speaker 1]
Yeah, great question. In the academy, so in customer education, they can track, they can view the certification. There is a link, and I think we were the customer that requested it, where they can share their certificate on LinkedIn from there.

We also send people out a mug, which people have been all over. They want the mug more than the certificate, I think. It's a very exclusive mug.

Even people internally at Hornbill can't get it unless they did the exam. We've also had requests and we've done it for people wanting email signatures with an official badge as well to prove that they're certified. So yeah.

[Speaker 2]
Yeah, yeah, I can totally get it. I would also go for the mug as well. We want to standardize our pass mark across exams.

Theoretically, we could reverse engineer the formula you shared to achieve that. That would be the benefits and disadvantages of this approach.

[Speaker 1]
Yeah, great question. I think the main benefit of that is it really helps you not have to, when you're writing questions or when you're asking your subject matter experts to think of questions, you're not thinking, oh, is this too hard? Is this too easy?

What's the pass mark going to be? You don't have that in mind. You use that formula once you've graded the questions and it just makes it a very objective approach.

And it also helps when you come back to checking the performance of the exam. So we look at how are the questions doing in relation to how they were graded. We, I don't know, our subject matter experts graded this question that 100% of customers should get this right at this level.

And actually, only 50% have got it right. And therefore, that opens the door for a conversation where, okay, maybe we need to bring this down to match that. We've only really had, I think, two questions that really showed a different aspect.

So what you're doing is you're not having to rewrite the questions. You're not having to dumb the questions down or make them harder. You're regrading the questions.

So it makes it more objective.

[Speaker 2]
Yeah. We do have time for one more question. So can you expand on how you provided the CAMP feedback to those that didn't pass without giving question level feedback?

How did you get them those teams for restudy?

[Speaker 1]
That's a great question. I told my team to go and do it. I think I pointed out that we had people failing and we weren't telling them what they were failing on.

We just said, this was your score. But in customer education, you can actually go and see how they performed in the exam. You can see because they're grouped into the areas.

You can, it's a 20-minute exercise, if that, to go and have a look and just say, okay, these are the themes that came up based on the questions. Map them back to the job task analysis. Okay, here's the objectives that you failed on.

[Speaker 2]
Yeah, perfect. We are at time, but I think Joel, he will still stick around here. So if you have some questions, come forward and ask Joel directly.