Better Summaries: Building Domain-Specific Clustering¶

Series Overview: This is the second notebook in our three-part series on systematically analyzing and improving RAG systems. In the first notebook, we discovered query patterns but found limitations with generic summaries. Now we'll fix that.

Prerequisites: Complete "1. Cluster Conversations" notebook first. You'll need the same dependencies and OPENAI_API_KEY from the previous notebook.

Reproducing Results: To reproduce the exact results from this notebook, the first cell downloads pre-computed checkpoints from our server. These checkpoints contain the intermediate results from each step of the clustering pipeline, allowing you to follow along without waiting for the computationally expensive embedding and clustering operations to complete.

To download our precomputed checkpoints, make sure that you curl the checkpoints_2 directory

Why This Matters¶

The generic summaries from our initial clustering missed crucial details that would enable effective query understanding. When working with specialized domains like machine learning experiment tracking, generic descriptions like "user seeks information about tracking" fail to capture the specific W&B features, user goals, and pain points that matter for system improvement.

Custom summarization transforms vague descriptions into precise, actionable insights. Instead of "user requests assistance with tool integration," we can generate "user is configuring W&B Artifacts for model versioning in PyTorch workflows." This precision is critical for building clusters that truly reflect how users interact with your platform.

Domain-specific summaries enable us to:

1. Capture exact features users are working with (Artifacts, Configs, Reports)
2. Identify specific goals and pain points rather than generic categories
3. Reveal usage patterns that generic summaries obscure
4. Create foundations for more targeted system improvements

What You'll Learn¶

In this notebook, you'll discover how to:

1. Extend Kura's summary functionality

   • Design specialized prompts that extract domain-specific information
   • Evaluate new domain focused approaches vs generic ones
   • Replace Kura's default summarization with your custom approach

2. Compare Summarization Approaches

   • Analyze the limitations of generic vs. domain-specific summaries
   • See how improved summaries change clustering outcomes
   • Understand the impact of summary quality on cluster interpretability

3. Generate Enhanced Clusters

   • Apply custom summaries to create more representative topic groups
   • Configure clustering parameters for optimal domain-specific results
   • Extract actionable insights about user behavior patterns

What You'll Discover¶

By the end of this notebook, you'll transform your seven generic clusters into three highly actionable categories: Artifacts, Visualisations, and Integrations. This dramatic improvement in cluster quality—from vague topics to specific, actionable user needs—will provide the foundation for building production classifiers in the next notebook.

The Power of Domain-Specific Clustering¶

While generic clustering tells you "what" users are asking about, domain-specific clustering reveals "why" and "how" they're struggling. This shift from surface-level topics to deep user intent understanding is what enables you to build targeted solutions rather than generic improvements.

By the end of this series, you'll have a complete framework for turning raw user queries into systematic, data-driven RAG improvements that address real user needs rather than perceived ones.

Creating Domain Specific Summaries¶

To address the limitations that we identified in our default summaries, we'll now see how it easy it is to replace our generic summarisation approach with a domain-specific prompt which will allow us to generate summaries that allow us to precisely capture the tools, features and goals relevant to W&B users.

With this new prompt, our goal is to

1. Identify specific W&B features mentioned in the query (e.g., Artifacts, Configs, Reports)
2. Clearly state the problem the user is trying to solve
3. Format responses concisely (25 words or less) to ensure summaries remain focused

This approach generates summaries that are not only more informative but also more consistent, making them ideal building blocks for meaningful clustering. Let's implement our custom model and see how it transforms our understanding of user query patterns.

Loading in Conversation¶

Let's first start by loading in our conversations and parsing it into a list of Conversation objects that Kura can work with

We'll then use the JSONL checkpoint manager to help store our conversations to our checkpoint

Let's now try to see how our default summaries look like

Looking at these default summaries, we can identify several key limitations that prevent them from being truly useful for clustering W&B-specific queries:

Problems with Default Summaries

1. Lack of Specificity: The first summary refers to "a specific tool" rather than explicitly naming Weights & Biases, missing the opportunity to highlight the domain context.

2. Missing Feature Details: Neither summary identifies which specific W&B features the users are interested in (experiment tracking, Bayesian optimization for hyperparameter tuning), which would be crucial for meaningful clustering.

These generic summaries would lead to clusters based primarily on query structure ("users asking for information") rather than meaningful W&B feature categories or user goals.

By defining our own summarisation model, we can address these limitations and cluster our user queries based off the specific problems and features they are trying to use.

Defining Our New Summary Model¶

Notice how simple this is: We just need to modify the prompt and pass it to Kura's existing summarise_conversations function. This approach makes the code easier to work with since we can configure the behavior through parameters rather than creating complex class hierarchies

By customizing the prompt in the summarise_conversations function, our summaries become more precise and feature-focused. This allows us to better reflect how users interact with Weights and Biases and in turn translate to more representative clusters.

We can now see the generated summaries below by passing in our new prompt into the summarise_conversations function. We'll be using the same conversations above which we generated summaries for.

from kura import ( generate_base_clusters_from_conversation_summaries, reduce_clusters_from_base_clusters, reduce_dimensionality_from_clusters, ) from kura.checkpoints import JSONLCheckpointManager from kura.cluster import ClusterDescriptionModel from kura.meta_cluster import MetaClusterModel from kura.dimensionality import HDBUMAP

async def analyze_conversations(conversations, checkpoint_manager): # Set up models summary_model = SummaryModel() cluster_model = ClusterDescriptionModel() meta_cluster_model = MetaClusterModel(max_clusters=4) dimensionality_model = HDBUMAP()

# Run pipeline steps
summaries = await summarise_conversations(
    conversations, model=summary_model, checkpoint_manager=checkpoint_manager, prompt=WNB_SUMMARY_PROMPT
)

clusters = await generate_base_clusters_from_conversation_summaries(
    summaries, model=cluster_model, checkpoint_manager=checkpoint_manager
)

reduced_clusters = await reduce_clusters_from_base_clusters(
    clusters, model=meta_cluster_model, checkpoint_manager=checkpoint_manager
)

projected = await reduce_dimensionality_from_clusters(
    reduced_clusters,
    model=dimensionality_model,
    checkpoint_manager=checkpoint_manager,
)

return projected

checkpoint_manager = JSONLCheckpointManager(CHECKPOINT_DIRECTORY, enabled=True) checkpoint_manager.save_checkpoint("conversations", conversations) clusters = await analyze_conversations( conversations, checkpoint_manager=checkpoint_manager )

Conclusion¶

What You Learned¶

In this notebook, you learned how to create domain-specific summarization models that dramatically improve clustering quality. You discovered how to:

• Create custom summary models using specialized prompts tailored to your domain
• Replace generic descriptions with precise, feature-specific summaries
• Configure clustering parameters to achieve optimal grouping results
• Compare clustering outcomes between default and custom approaches

What We Accomplished¶

We were able to address the key limitation of our initial clustering using a new custom prompt that was domain specific, focusing on W&B features and user intentions. This allowed us to transform our clsutering results from generic topic groups into highly actionable categories.

1. Optimize machine learning experiments and resources (449 conversations) - The largest cluster covering users exploring experiment tracking, hyperparameter optimization, multi-GPU training, and maximizing W&B utilization for improved ML performance
2. Enhance machine learning project collaboration with W&B (74 conversations) - Users integrating W&B with PyTorch/TensorFlow workflows and optimizing team collaboration features including roles, permissions, and enterprise capabilities
3. Manage audio files and dataset versioning (37 conversations) - Users organizing audio data for classification analysis and managing dataset versioning workflows in W&B

This represents a significant upgrade from our previous clusters, providing much more specific and actionable information about user needs. The improved summaries eliminated the vagueness of descriptions like "user seeks information about tracking" and replaced them with precise insights about specific W&B workflows, optimization goals, and collaboration requirements.

Next: Building Production Classifiers¶

While our improved clustering gives us deep insights into historical query patterns, we need a way to act on these insights in real-time production environments. In the next notebook, "Classifiers", we'll bridge the gap between discovery and action by:

• Building production-ready classifiers using the instructor library to achieve ~90% accuracy through systematic prompt engineering
• Creating automated labeling workflows with weak supervision to efficiently generate labeled datasets for training
• Focusing on three high-impact categories - artifacts (20% of queries), integrations (15%), and visualizations (14%) - that account for almost 50% of all user conversations
• Applying classifiers at scale to understand true query distributions and identify exactly where to focus improvement efforts

This classifier will enable you to automatically categorize incoming queries in real-time, detect production drift when certain query types surge, and intelligently route questions to specialized retrieval systems. More importantly, you'll move from "we think users struggle with X" to "we know 20% of users need help with artifacts, 15% with integrations, and 14% with visualizations—and we can automatically detect and route these queries for specialized handling."