Quality monitoring and tuning

At rapidflare.ai, we are committed to maintaining the highest standards of performance and accuracy in our enterprise agent. Our comprehensive quality reports provide valuable insights, enabling both our internal teams and customers to track performance and ensure continuous improvement.

Types of quality reports

We offer two primary types of quality reports to assess and enhance the agent's performance:

• Initial QA Report: Conducted during the onboarding phase by our customer success team, this report helps assess and fine-tune the agent's responses before full deployment, ensuring it aligns with customer-specific needs.
• Ongoing QA Report: After deployment, this report captures, analyzes, and resolves customer feedback. It is essential for maintaining the agent's accuracy and relevance over time. The ongoing QA report features a pie chart that highlights the percentage of issues across various "Issue RCA" (Root Cause Analysis) categories, including missing information, glossary gaps, analytical errors, and poor references.

Key performance metrics

Our quality reports measure several key performance indicators (KPIs) to ensure the agent's effectiveness:

• Correctness of answers: Evaluating the accuracy of responses across various query categories.
• Correctness of references: Ensuring the right sources and references are used for each query.
• Number of conversations: Tracking the volume of interactions to monitor overall engagement.
• Customer feedback: Collecting both positive and negative feedback to drive improvements.
• Issue RCA: Categorizing issues into root causes such as missing information, glossary mismatches, analytical errors, and poor references. This analysis allows for targeted improvements to the agent's responses.

Each instance of negative feedback is recorded in the report, with a status field indicating the progress of issue resolution, ensuring transparency and accountability in addressing customer concerns.

Report frequency

We provide quality reports on a weekly and monthly basis, allowing customers to regularly monitor the agent's performance. Additional reports can be delivered on demand, ensuring flexibility and real-time updates.

Report format

Our reports are presented in Excel format, with organized data and visual elements. Pie charts are used to display key performance metrics, including the percentage of correct answers and a breakdown of issues based on `Issue RCA` categories. This visual representation offers customers a clear and quick overview of the agent's performance trends.

Actionable insights

Both the initial and ongoing QA reports include actionable insights, with comments and queries from our customer success team. Customers are encouraged to engage with these insights, helping us resolve issues such as missing documentation or incorrect references. This collaborative feedback loop ensures that the agent continues to evolve and meet the dynamic needs of the business.