analytics-que-importam
Every AI platform dashboard is a carnival: 30 charts, 12 colors, a handful of "insights" generated by some AI trying to look helpful. The problem is that 80% of those numbers don't change a single decision.
After looking at data from hundreds of hubs on Member AI, we can say with confidence: four metrics matter for an agent creator. The rest is noise. If you haven't set up your first agent yet, start with this 15-minute tutorial.
Why almost every metric is noise
A metric only matters if it answers "yes" to two questions:
- If it gets worse or better, will you do something different tomorrow?
- Does it correlate with revenue, retention, or member experience?
"Total messages sent", for example, fails both. 10 thousand messages can mean 300 productive conversations or 5 stuck in loops. Doesn't change a decision.
Now, in-conversation resolution rate — that one changes decisions. If it drops from 72% to 58%, you know something's broken in the agent and you have to act today.
1. In-conversation resolution rate
Percentage of conversations the member ends with the problem/request resolved, without escalating to a human and without abandoning halfway.
How Member AI measures it: the agent marks the conversation as "resolved" when (a) the member signals positively at the end, (b) the action was completed (payment, scheduling, content delivery), or (c) the agent sent the last message and 48h passed with no reply and no negative signal.
< 60%
something is broken
If it drops a lot: first thing to do is open the 20 most recent conversations marked "unresolved" and read them. In 80% of cases the problem is missing content in the knowledge base, not a dumb agent.
2. Cost per resolved conversation
Tokens spent on the model + Stripe fee (when there was a payment) divided by the number of resolved conversations.
This is the most important metric for anyone who wants to actually understand unit economics. A conversation that costs $0.04 and closes a $99 sale has different economics than one that costs $0.18 and closes a free booking. (Token cost lands directly on your account with BYOK — see why BYOK matters.)
Reference points we typically see:
- Educational hub with simple content: $0.02 to $0.05 per resolved conversation;
- Mentorship/consulting hub (longer conversations): $0.06 to $0.18;
- B2B hub with complex reasoning: $0.25 to $0.60.
If your cost is well above that, the culprit is almost always: a giant system prompt (shrink it), an overkill model (swap 4o for 4o-mini), or badly configured memory (it's re-injecting old context every turn).
3. Conversion to revenue action
Percentage of conversations ending in a monetizable action: payment, sales call booking, signup to a paid list, plan upgrade.
This is the most "business-y" metric in the quartet. It doesn't tell you whether the agent is fun to talk to — it tells you whether it's generating revenue.
Benchmarks:
- Content/education agent: 3% to 8% conversion to revenue action;
- Sales/SDR agent: 12% to 25%;
- Existing-customer support agent (upsell): 6% to 15%.
If conversion is low, it's rarely the agent — it's the offer. Product without a clear contextual trigger, price misaligned with the promise of the conversation, or weak CTA at the moment of yes.
4. Member retention (D30)
Out of 100 new members who arrived at the agent on a given day, how many come back and chat again within 30 days?
This is the metric that separates an agent that works from one that addicts. A member who comes back is one who saw value. A member who doesn't was never going to be a success story.
< 25%
rethink positioning
What to ignore
Metrics that show up in every dashboard and won't tell you anything you need to know:
- Total messages sent. High isn't good, low isn't bad.
- Average agent response time. Always < 3s on a decent LLM. Looking at this is wasted time.
- Average conversation sentiment. Very weak proxy, easy to bias.
- Model "assertiveness" or "confidence". Made-up number that no AI provider guarantees.
- Tokens per conversation (isolated). Only matters inside metric #2.
How to turn this into decisions
The ritual we recommend for hubs:
- Weekly: look at resolution rate. Dropped? Open 20 unresolved conversations and read them.
- Bi-weekly: cost per resolved conversation. Up? Review system prompt and model.
- Monthly: conversion to revenue action. Below benchmark? Review offer/trigger, not the agent.
- Quarterly: D30 retention. Dropping? Re-positioning, not fine-tuning.
See these metrics in your hub
14 days free. Dashboard with the 4 right metrics — no 30 useless charts.
Create account
A good dashboard doesn't have more charts. It has the right charts, with the right questions in front. The rest is pretty noise.
Read more at memberai.pro/en/blog/analytics-that-matter.
Learn more: plans and pricing · about Member AI · real customer cases · full blog.