SAP’s Innovative Approach to Knowledge Article Scoring

At a recent KCS in Action, we were fortunate to host Dr. Sean MacNiven, the Head of Research and the Strategic Program Architect for SAP Global Product Support. Sean presented on SAP’s innovative approach to AQI (now referred to in Consortium publications as the Content Standard Checklist). He covered how coaches approach article reviews with the help of both their peers and automation, as well as some details about their AI roadmap.

Sean shared the issues SAP uncovered when asking coaches to score their coachee’s knowledge articles, including:

  • The coach/coachee relationship is a partnership: mutual reliance, confidence, trust, and friendship develop along the way. Can reviews be genuinely fair, impartial, and objective when “personal”?
  • The reputation of the coach is tied to those they coach. The coach has a vested interest in the success of the coachee.
  • The relationship is not one of equals, and individual preferences and biases can be magnified. This could lessen reliability and fairness.

SAP has had great success in randomizing the article review and scoring process. These reviews are done by two separate, random coaches, which encourages alignment in scoring, and allows a coach and their coachee to discuss the output more objectively. Along the way, SAP was surprised to find that coaches disagree on the article score the majority of the time. Sean shared the specific questions they typically disagree on, with “Alignment to the Content Standard” score showing the most significant disagreement.

Sean demonstrated their article scoring tool, including how they are currently using automation to speed up article reviews and improve the accuracy of those reviews. Looking forward, SAP plans to leverage automation and AI to conduct the article review and score without involving humans. They could then perform these reviews on all the articles in the knowledge base.

SAP’s ultimate goal is to leverage AI and automation to create knowledge articles (which align with their content standard and other criteria!) from structured case notes without human assistance. This is a dream scenario for many Consortium Members, and we look forward to seeing how technology and processes develop over the next few years in pursuit of that.


This KCS in Action webinar was packed with a wealth of information! We encourage you to access the PDF of the slides, the chat transcript and view the recording below.

Resources Shared

Chat Highlights

Michael Fisher: The luxury of “350 coaches” to choose from puts this far beyond our reality.
Mel Mathis | ADP | They/Them: Same. We struggle having enough Coaches.
Kristin Hunter: Same. And… I’m wondering about having reviews randomly assigned no matter the coach pool size.
Joe Reifel: Perhaps there are some some “closet coaches” that would happily join in if the assignment was randomized and anonymous.
Shari Wennes – SAP: We can ask Sean around the “pool” to go off of.  But we have set this up that if they have completed a review and they get another one but we are only asking today that the coaches do 1 random review a week.

Michael Fisher: What about using Generative AI tools to do these article evaluations? That would take the subjective personal factors out of the mix entirely.
Mel Mathis | ADP | They/Them: Depending on the company, Generative AI is dangerous unless created in house or bought fully by from another company.
Michael Fisher: Obviously you’d need to train the AI, as in any application of that tool.
Shari Wennes – SAP: We do have automation built into some of these questions but I agree personal review is needed for some of these items.
Anna Busch: That´s our approach at clients, our AI suggests it and the person in the loop confirms

Billie Jean: Most SAP Concur Expense coaches (with travel coming soon) use PAR to do the other piece of KB Cases.  We have to do the AQIs separately.  Is there a way to get all this to be in one place vs. two different applications?
Shari Wennes – SAP: ServiceNow does have AQI and PAR built in on our system.  Coaches do the initial AQI in NOW first and then the AQI Peer Review is done in ISE and feeds the information back into NOW.  Sean’s team will be looking at doing the same thing for PAR in the future as well.

Jason O’Donnell: Are the coaches reviewing content for products within their expertise, or across the company and reviewing content for which they may not have direct product expertise?
Sean MacNiven: No completely random. The AQI should not require subject matter expertise for the product.
Jason O’Donnell: I absolutely agree 🙂 Glad to see others who have the same perspective.

Guest: I found this very interesting to see how things are being done differently between the Concur and SAP Platforms. I have already been involved in some projects related to AI and ChatGPT with KCS and am excited to see how they can potentially impact the program in the near future.


pablo.diaz: Thanks for your time Sean
Kevin Matthews: Thanks , Sean! This was great material and provides a lot of food for thought!  👍
Rick Joslin, Cigna, Pittsburgh, PA, USA: Thanks for sharing your wonderful story and creating a vision for the future.
Cathy R: Danke!
Cressenda Youngs: Great session!!
Joel Sandi (he/him): Fantastic, thank you!
Dave Ross: Thanks for your time, Sean! Great presentation.
Joanna D: Thank you very much!
Anna Busch (avato consulting): Thanks!
Christine Egli: Super interesting, Thank you!
hannah hitchens: Appreciate the presentation!
Maarit Karp: Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *