Measuring Self-Service Success

The good news: Most service organizations have expanded beyond their agent-assist channels to include self-service support for their customers!

The bad news: While service organizations have robust measures for their cases/incidents, many struggle to measure customer issues resolved via self-service support. Given the comfort zone around case metrics, we often see case deflection as the primary measure for successful self-service resolutions. Keeping the case as the primary “currency” for the support experience has many drawbacks because measuring cases while optimizing self-service does not accurately assess the customer experience:

  • Customers who resolve their issues via self-service don’t always intend to create a case if unsuccessful.
  • As self-service support increases, particularly for known issues, case metrics (cost per case, cases per agent, time to resolve, etc.) move in the “wrong” direction because new issues usually take longer to resolve than known issues.

Consortium Member approach to measuring self-service success

Consortium Members have been working on this measurement challenge for many years and found a better approach: up-level from case to customer issue by measuring issues resolved per service channel. As organizations shift issue resolution into their self-service channel, they find that issue metrics all move in the right direction.

Service Engagement Measures Template
Service Engagement Measures Template

Recording

Consortium Members shared their good, better, and best methods for calculating these metrics in a recent webinar. You can access a PDF of the slides, or watch the recording and see the chat transcript below.

Resources Shared

Chat Highlights

  • Ross Shepherd | HPE: Hi folks! Please go into some depth explaining clickstream analysis and how it can help with these measurements. My current reading is that Clickstream still depends on a bunch of assumptions so how can we ensure it is accurately representing the intents and use cases it’s deployed for.
    • Christina Roosen | Akamai | California: Hi @Ross Shepherd | HPE! There is some level of assumption with all self-service success measures – until we can read the minds of our users to find out if they got an answer. I use the Path Exploration reporting on Google Analytics to look at the path users take through the site so that I can see their start and end points. We have knowledge and case management on a single site, so I can see clearly whether they searched, found an article, and then either did or did not open a case.
    • Sara Feldman | Consortium | Las Vegas: Yes some assumptions are unavoidable. From the Understanding Success by Channel resource: “The most useful strategy is to trend against yourself. Consider how you can establish a baseline for your organization and measure your progress against it.”
  • Dave Stewart (Akamai, Ottawa): How can we differentiate between time spent on a page and a page being abandoned (but not closed) which could mean the viewer left in disgust….or loved it so much they kept the page open.  It would be great if we could capture bookmarked pages, but unless that’s part of the UI itself, it’s a client/browser side activity we don’t have insight into…
    • Jason O’Donnell | Percepta | PDX: There are some good assumptions you can use here… throwing out sessions that last longer than a set amount over the expected amount of time on a page would be one way to filter out abandoned open pages
    • Sara Feldman | Consortium | Las Vegas: Looks like GA4 has tried to address this!
    • Dave Stewart (Akamai, Ottawa): Indeed..it’s almost like I willed that into existence right there
    • Sara Feldman | Consortium | Las Vegas: Well done, Dave!
    • lola CAILLAUD: @Dave Stewart (Akamai, Ottawa) you’d be able to have analytics such as bookmarked topics if you’d decide to expose your knowledge content in a single hub, serving different customer’s touch point thought, and containing your KB articles or any product related documentation. Then you’d have a very precise view on how your content is consumed, if it help deflecting cases and so on.
    • Dave Stewart (Akamai, Ottawa): I love the idea of having everything in (or findable through) a single space – our customers would sure love that.  Any suggestions?
  • Kellie Sullivan: Christina that was a great point to remove the direct traffic.
  • Elena Forrest, Oracle, Arlington Virginia: What is Organic Search? and Organic traffic volume?
    • Jason O’Donnell | Percepta | PDX: Organic search is effectively the non-paid search results. Organic traffic is the volume of visitors coming from organic search
  • Dave Stewart (Akamai, Ottawa): So true….the numbers have to be accepted by stakeholders
  • Ruchi: how can we measure self service if the current tool is not providing the functionality of no. of search hits by the user?
    • Arun Vats: Honestly, that’s a big challenge. Maybe, tool enhancement is needed first
    • Jason O’Donnell | Percepta | PDX: audits of what metrics/measures you DO have is a good place to begin… and depending on what you’re able to connect you may or may not need tooling enhancements. But, starting with what you have will help you iteratively improve your reporting as you learn more about what those measures are telling you about where to focus.
  • Sara Feldman | Consortium | Las Vegas: Find the spreadsheet and all the public guidance from the amazing Member working group here
  • Lynette Ledoux | SearchUnify: So Issues per Engineer includes Issues Solved via Self-Service?
    • Arnfinn Austefjord, Consortium: Yes, Issues per Engineer includes Issues Solved via Self-Service.  Engineers contribute to Self-Service through their KCS activities.  They should also be given credit for these issues.
  • Caralee Harrison: Would Issue Focused Time to Resolution include self-service and agent-assisted (thereby lowering time to resolution in a mature environment)
    • Arnfinn Austefjord, Consortium: Yes, Issue Focused Time to Resolution include self-service and agent-assisted (thereby lowering time to resolution in a mature environment)
  • Rohan Ragoo: We do surveys for Intel – great to hear
  • Callie Cella: Does anyone use outbound phone calls for customer satisfaction measurement?
    • Judith de Jong, RWS (Belgium): Yes, we try to do yearly calls on all our touchpoints with our largest customers in RWS. We call it the Voice of the Customer program and it covers products, PS, Account Management, Support, Hosting, Services, or whatever else customers use from us.
  • Ruchi: What steps should be taken in order to improve customer satisfaction? Even after having a good number of self service articles, end users are raising tickets to service desk and not utilizing KBA’s. How to make change in end user behaviour to contact SD instead of using self service articles?
    • Dave Stewart (Akamai, Ottawa): Industry analysis seems to very strongly indicate that clients prefer to self-service their issues, so it’s possible that they either aren’t aware of how/where to search for knowledge, or the search experience and results is poor so they give up and open a case (since they’re paying for the Support anyway)
    • Arun Vats: At the time of ticket logging, encourage users to first go through self service portal and try issue resolution on their own. Only if, they are unable to resolve the issue, should they be allowed to log tickets.
    • Jason O’Donnell | Percepta | PDX: this is a huge question that could be its own workstream! 🙂 Starting with Customer Journey Mapping is a great way to begin to understand some of the challenges your customers encounter before jumping in to solutioning.
    • Janine Deegan | Smarsh: Take a look at what cases are coming in and make sure your articles are easily findable. Are the titles in customer context? Are key words accounted for?
    • Jason O’Donnell | Percepta | PDX: @Arun Vats I’d warn against gating access to support reps. We’ll never (and shouldn’t) see all customers use self service.
    • Kelly Murray | Consortium | Seattle: Some Members have had success shifting user behavior by including language in their case closure emails along the lines of “this was available on our self-service site” and including the link to the article.
    • I’d warn against gating access to support reps. We’ll never (and shouldn’t) see all customers use self service.
    • Jason O’Donnell | Percepta | PDX: @Arun Vats, it sounds like a voice of the customer initiative to reach out and understand the customer perspective on Self service may be in order.
  • Warren Cook: Wondering if anyone incorporates customer self built channels like Reddit into their Community data analytics and if so, how?
  • Kellie Sullivan: For everyone using GA for views for number of engagements. Is there a length of time that you are using before a session is counted as a new session. Is it hourly, daily, weekly? Meaning, if a user comes into our knowledge pages their session is only counted once for that determined time.
  • Sara Feldman | Consortium | Las Vegas: Love that insight! Why are we measuring? For actionable insights to IMPROVE. Great thing to keep top of mind when deciding what to measure and how.
  • Lynette Ledoux | SearchUnify: Navigating to content is a good point: Would this activity not be captured as Direct in GA?
  • Christina Roosen | Akamai | California: For everyone using GA for views for number of engagements. Is there a length of time that you are using before a session is counted as a new session. Is it hourly, daily, weekly? Meaning, if a user comes into our knowledge pages their session is only counted once for that determined time.
    • Great question! I’m going back to context, because I always do. I’d try to find the average velocity of case creation when making this decision. Do your customers open a case a day? A case a week?
  • Brian Pahl: Thinking of that shift from measuring Case Avoidance to Support Interactions across desired channels, Our Support management is hyper cost focused. What have you found as a compelling argument to management to shift the measurement perspective?
    • Devin Poole: There is a “best of both worlds” argument to be made. Customers would prefer to self-service (on their own time, at their own pace), and it is typically much more cost effective (can be 100x less cost for a self-service resolution). I would encourage you to initially take an “issue centric” approach and focus on the top 3-5 customer issues that customers are trying to self-serve, but are not successful, and put resources against solving those problems. 
    • Dave Stewart (Akamai, Ottawa): There is also a case to be made for concierge-guidance (either human or AI) to ensure customers know where to go to start exploring/researching.  I believe SAP takes this approach somewhat, or at least focuses on triaging incoming issues or questions…
  • Avila, Cecilia: We heavily use slack channels and members of the channel offer solutions to each other.  How is this support measured?
    • Sara Feldman | Consortium | Las Vegas: Have you looked into what type of analytics Slack provides?
    • Avila, Cecilia: Replying to ” Not yet.  But I can partner with our group who owns our enterprise slack.
  • Christina Roosen | Akamai | California: Navigating to content is a good point: Would this activity not be captured as Direct in GA? If the user is navigating from another site (that isn’t organic search) that shows up as Referral traffic. Direct is ‘I wasn’t on another site, and here I am on yours’
  • Kevin Kimathi (Coveo): Has anyone found a way to bucket self-service activity into issues? As a way to identify issues are beings resolved by self-service
  • Noah Kardos-Fein (he/him): Are there any good resources for implementing and leveraging GA4 in a self service context? We have it enabled, but there’s a huge learning curve to getting things properly configured, since it’s traditionally used for tracking marketing stats.
  • Christina Roosen | Akamai | California: Are there any good resources for implementing and leveraging GA4 in a self service context? We have it enabled, but there’s a huge learning curve to getting things properly configured, since it’s traditionally used for tracking marketing stats.
  • Sara Feldman | Consortium | Las Vegas: Thanks for being here – please stay in touch!

Kudos

  • Caralee Harrison: That was great thanks!
  • Anna Murafa: Thank you — great session!
  • Allison Donald: Excellent session thank you all!
  • Warren Cook: Great session. This is such a complex and evolving topic
  • Kevin Kimathi (Coveo): Great session. Thank you!
  • Kellie Sullivan: This session was AWESOME today!! Thank you
  • Faizan Fakir [Coveo]: Thank you
  • Ruchi: Thank you!!
  • Chitra Ramaswami: Thank you!
  • Nick: Thank you! Bye everyone!
  • Tim Harris: Thanks everyone. Great talk!
  • Lana Kosnik, Upland RightAnswers, Michigan (USA): Thank you!
  • Darren Burnell (Quest): Thank you, very informative
  • Dave Stewart (Akamai, Ottawa): Really good session on a really important topic – getting this wrong is a huge drag on overall success
  • Maykol: More sessions like this one, PLEASE!
  • Elena Forrest, Oracle, Arlington Virginia: This is a great session, thank you for covering this topic! I learned a lot!

Leave a Reply

Your email address will not be published. Required fields are marked *