Most dashboards show you a lot of numbers. Not all of them are useful. If you find yourself staring at charts wondering what to act on, this list is for you.
Metrics That Reflect Real Behavior
- Containment rate — the share of conversations the bot resolves without a human agent. A rate below 60% often signals that intent coverage is too narrow, not that users are difficult.
- Fallback rate — how often the bot fails to match any intent. Anything above 15% consistently means your training data has gaps worth addressing.
- Session length distribution — not the average, but the full spread. A spike at very short sessions usually means users gave up early. A spike at very long ones may mean the bot is looping.
- Goal completion rate — did the user accomplish what they came to do? This is harder to measure but far more meaningful than session count.
- Re-engagement rate — how many users who left mid-conversation returned within 24 hours. A high number here can indicate the bot left something unresolved that users still needed.
A Note on Averages
Averages hide the edges, and the edges are where problems live. When reviewing containment or fallback rates, always segment by topic category or entry point. A bot performing well overall can be failing badly on a specific intent cluster — and an average will not show you that.
Spending 20 minutes weekly reviewing fallback logs by category tends to surface more actionable issues than any automated report. It is not glamorous work, but it is where the real patterns are.