Chat Ratings
Description
Chat Ratings gives administrators a quick, rolling snapshot of how learners are experiencing a specific agentâby connecting the History (recent chats) and Memory (saved user context) features.
The rating aggregates the past 24 hours of learner interactions and refreshes daily, helping you see whatâs working, whatâs not, and where to intervene.
Target Audience
Administrator
Features
24-Hour Rolling Rating
Calculates an agentâs learner-experience rating from the most recent 24 hours of chat activity; updates automatically every day.
History Ă Memory Integration
Links recent conversation data (History) with user context (Memory) to ground ratings in real usage, not one-off anecdotes.
Per-Agent View
Ratings are scoped to the specific agent (e.g., âAgentic OSâ), allowing accurate comparisons between agents.
Actionable Insight
Use the rating trend to spot when learners are thrivingâor strugglingâand prioritize follow-ups or prompt refinements.
How to Use (step by step)
Open the Agent
- Select the agent you want to review (e.g., Agentic OS).
Verify Memory Is Enabled
- Go to Memory to confirm itâs On and (optionally) that Reference Saved Memories is enabled.
- You can browse which learners have saved memories such as:
- Personal Information
- Knowledge Gaps
- Help Requests
- Lessons Learned
Check the Chat Rating
- Open History (or view the rating indicator in the agentâs overview, if available).
- View the 24-hour rating that reflects recent learner experiences with this agent.
Drill Into Evidence
- In History, review recent transcripts from the same time window to understand why the rating changed.
- Cross-reference with Memory entries for those users (e.g., known gaps or help requests) to see if the agent addressed them effectively.
Take Action
- If the rating dips, adjust one or more factors:
- Prompts â refine tone, structure, or guidance.
- Datasets â fill content gaps.
- Tools â enable relevant features (e.g., Web Search, Code Interpreter).
- Recheck the rating the next day to assess the impact of your changes.
Pedagogical Use Cases
Early Warning for Struggle
A downward trend signals confusionâreview transcripts, add resources, or tweak prompts to clarify key concepts.
Quality & Tone Assurance
Ensure the agentâs responses align with course expectations; refine the System Prompt or tone as needed.
Measure Improvements
After changing prompts, datasets, or tools, use the next dayâs rating to validate that your intervention improved learner experience.
Targeted Support
Combine rating trends with Memory insights (knowledge gaps, help requests) to identify and reach out to specific learners or cohorts needing support.
With Chat Ratings, you get a simple, always-current gauge of learner experienceâgrounded in the last day of real conversationsâso you can keep each agent effective, supportive, and on track.