🗜️Backtesting

Purpose: Backtesting evaluates the effectiveness of the current project documentation by generating bulk responses to historical queries. This workflow identifies how well the documentation covers key topics and where improvements are needed.

Prerequisites:

  • Discord, Telegram, or Slack Loader Agents configured to pull historical data.

  • Bulk Response and Q&A Agents for processing queries.

How it Works:

  1. Historical queries are pulled from the community platforms.

  2. The Bulk Response Agent generates automated responses based on the current documentation.

  3. The system evaluates the accuracy and relevance of these responses to determine documentation effectiveness.

Inputs:

  • Historical queries from community platforms (Discord, Telegram, Slack).

Processing:

  • Queries are processed in bulk, and automated responses are generated using the Q&A Agent.

Outputs:

  • A report detailing the effectiveness of the current documentation, highlighting any gaps or inaccuracies.

Usage: Admins can run backtesting via the admin dashboard, selecting historical data for processing.

Example: The system generates responses to 100 past queries, revealing that 80% of them were answered correctly, while 20% indicated gaps in the documentation.

Advanced Features:

  • Integration with the Taxonomy Agent for deeper analysis of content gaps.

Troubleshooting:

  • Inaccurate Responses: Review and update documentation based on the backtesting results.

Logs:

  • Logs can be accessed through the Backtesting dashboard, detailing the results of the bulk response generation.

Related Workflows:

  • Taxonomy Agent for improving documentation coverage.

  • Alert Agent for notifying teams of significant documentation gaps.

Last updated