Observability Dashboard

Observability Dashboard allows you to scrutinize the AI capabilities employed by Copilot, Autopilot Knowledge, and Autopilot. The dashboard provides comprehensive insights into the performance of the generative responses, enabling you to identify areas for improvement and optimize your operations.

Copilot

The Observability Dashboard for Copilot provides information about generative responses, Copilot for Agents Queries, and AutoSummary.

  1. Click the app selector and select Actions.

  2. On Actions, click Observability Dashboard.

    By default, the Copilot dashboard appears. It contains three charts:

  3. If needed, update theDate Range for the dashboard and select the interaction type in Channel. Choose to view details for Voice interactions, Chat interactions, or All.

  4. Click Run Query. The three graphs are updated to reflect your desired dates and channels.

Viewing Data About GenAI Performance

Generative responses are answers automatically generated during calls.

  1. Click the GenAI Performance label to drill down into the statistics. Three graphs appear:

    • Over Time: Shows the percentage of answers that were used, modified, ignored, or resulted in no answer over time.

    • By Category: Displays the details by category.

    • Average Kb Per Interaction: Shows the average number of knowledge base interactions per day.

  2. You can customize data that appears in the graphs:

    • In the Over Time and Category graphs, you can toggle the display of different answer statuses by clicking on the legends.

    • In the Over Time and Category graphs, click Absolute Numbers or Percentage to switch between percentages and absolute numbers.

    • In all three graphs, click Maximize to view the data in a table format.

  3. Scroll down to see the data grouped by categories. The Categories View organizes the knowledge base answers into different categories, offering a structured approach to analysis. Each category presents:

    • Total volume of knowledge base answers

    • Average adherence score

    • Average number of links and images provided

    • Average knowledge score (score assigned by the knowledge base)

  4. Clicking on a category reveals the specific queries and their details, such as:

    • Query sent to the Knowledge Base

    • Suggested knowledge base answer

    • Agent's actual response

    • Number of links and images provided

    • Adherence score (the similarity between suggested and actual response)

    • Offset from the beginning of the interaction

    For each query, you can:

    • Click Play Interaction to listen to the audio of the interaction (if available).

    • Click the Info button for the query. In the Response details panel on the right, click the Create Article button to generate a knowledge article. An AI-generated article is drafted based on the transcript. You can edit the article as needed and then publish it.

      For complete information on editing and publishing an article, see the knowledge generation help.

  5. You can switch the way you view this data. The default view is by category. Click Group By to change the grouping from Category to one of the following:

    • Master Contact

    • Team

    • Skill

    • Agent ID

    The data reappears based on the new grouping.

Viewing Data About Agent Queries

Agent Queries displays information about the knowledge base answers generated on-demand based on agent questions. It provides graphs that show the status of direct queries and adherence scores.

  1. Click the chart heading, Agent Queries, to drill down into the statistics. The first graph shows the percentage of responses and no responses Over Time. The second graph displays the details Over Category.

  2. You can toggle the display of different answer statuses by clicking on the legends.

  3. Click Absolute Numbers or Percentage to switch between percentages and absolute numbers.

  4. Click Maximize to view the graph in full screen.

  5. Scroll down to see the data grouped by categories. The Categories View organizes the knowledge base answers into different categories, offering a structured approach to analysis. Each category presents:

    • Total volume of responses

    • Number and average of no responses

    • Average number of links and images provided

    • Average knowledge score (score assigned by the knowledge base)

  6. Clicking on a category reveals the specific queries and their details, such as:

    • Agent query sent to the Knowledge Base

    • Response to the agent query

    • Number of links and images provided

    • Date and time of the response

    • Average knowledge score

Viewing Data About AutoSummary Queries

AutoSummary provides a comprehensive view of summary performance. You can see graphs that track performance over time, with data grouped by intent and skill. Detailed tables display suggested summaries alongside actual summaries, complete with adherence scores to gauge accuracy. For more comprehensive details, you can play back specific interactions, which will give you a full picture of how summaries are generated and used in real conversations.

  1. Click on the chart heading, AutoSummary to drill down into the statistics.

    • The first graph shows the percentage of summaries that were used over time. Summaries are identified as being used in one of the following fashions: As Is, Revised, with Minor Revisions, and Ignored.

    • The second graph displays the details By Intent.

    • The third graph displays the details By Skill.

    • The fourth graph displays the details By Team.

    You can toggle the display of different answer statuses by clicking on the legends.

  2. Click Absolute Numbers or Percentage to switch between percentages and absolute numbers.

  3. Click Maximize to view the graph in full screen.

  4. Scroll down to see the data grouped by categories. The Categories View organizes the knowledge base answers into different categories, offering a structured approach to analysis. Each category presents:

    • Total volume of responses

    • Number and average of no responses

    • Average number of links and images provided

    • Average knowledge score (score assigned by the knowledge base)

  5. Clicking on a category reveals the specific queries and their details, such as:

    • Agent query sent to the knowledge base

    • Response to the agent query

    • Number of links and images provided

    • Date and time of the response

    • Average knowledge score

As a CX leader, Mark wants to use the Observability Dashboard to identify knowledge gaps in the knowledge base. He notices that many knowledge base suggestions are being modified or ignored. Mark clicks on Knowledge Base Answers to view data by category. He focuses on categories with low average adherence scores, indicating misalignment between suggestions and actual responses. By expanding low-scoring categories like Billing and Payments, Mark sees low adherence to queries about payment plans and refund policies, suggesting knowledge gaps in those areas.

Through the Observability Dashboard, Mark can pinpoint topics needing knowledge base enhancements. He can then work with the knowledge base team to address these gaps, improving the quality of suggestions for better customer interactions.

Autopilot Knowledge

The Observability Dashboard for Autopilot Knowledge shows data about how well your automated system handles customer questions. You'll see a graph that displays performance trends over time. This lets you track changes daily, weekly, or monthly.

  1. Click the app selector and select Actions.

  2. On Actions, click Observability Dashboard.

  3. Click the Autopilot Knowledge tab.

  4. Set the desired Date Range for the dashboard, and click Run Query.

  5. Click the GenAI Performance label to view detailed graphs about responses:

    • Over Time graph: Tracks the number of successful responses and no-responses.

    • By Category graph: Identifies areas of strength and weakness, grouped by category.

    You can customize the appearance of data in the graphs:

    • To switch between percentages and absolute numbers, click Absolute Numbers or Percentage .

    • To view the graph in full screen, click Maximize .

    • To toggle the display of different answer statuses, click on the Response or No Response legends in the graphs.

  6. Scroll down beneath the graphs to see the data grouped by Intent. The Intent view organizes the summaries into different categories, offering a structured approach to analysis. Each intent presents:

    • The intent of the summary

    • Number and average of summaries

    • Average adherence and the score assigned

  7. To see the specific queries and their details, click on a category, such as:

    • Suggested Summary

    • Actual Summary

    • Percentage and status of Adherence

    • Date and time of the summary

Autopilot

The Observability Dashboard for Autopilot shows how well your knowledge base is able to handle customer questions. Use this dashboard to look for areas where you can add more articles to your knowledge base or customize the articles to better answer the customers' questions.

  1. Click the app selector and select Actions.

  2. On Actions, click Observability Dashboard.

  3. Click the Autopilot tab.

  4. Set the desired Date Range for the dashboard, and click Run Query.

  5. Click the GenAI Performance label to view details about the questions asked and the articles provided for the time period specified.

    Two graphs appear:

    • Over Time graph: Shows the number of successful responses and no-responses over the period of time. A response indicates the user was shown an article. A no-response indicates that no matching article was found.

    • By Category graph: Shows the number of successful responses and no-responses by categories.

    You can customize the appearance of data in the graphs:

    • To switch between percentages and absolute numbers, click Absolute Numbers or Percentage .

    • To view the graph in full screen, click Maximize .

    • To toggle the display of different answer statuses, click on the Response or No Response legends in the graphs.

  6. Scroll down beneath the graphs to see a table that provides details on the queries. The queries are grouped by categories.

  7. Clicking on a category reveals the specific queries and the provided responses. Use this to identify areas where you might be missing articles in your knowledge base.