Watch and track your favorite playlist.
Curated by: CampusX (12 videos)
In this session of the LLM Evaluation Masterclass, we transition from theory to practical application by exploring the specific mechanisms used to judge whether an LLM's output is good. This lecture was taken live for insiders, join here: https://youtu.be/j_G30FLmCcw 📱 Grow with us: CampusX' LinkedIn: https://www.linkedin.com/company/campusx-official CampusX on Instagram for daily tips: https://www.instagram.com/campusx.official My LinkedIn: https://www.linkedin.com/in/nitish-singh-03412789 Discord: https://discord.gg/PsWu8R87Z8 E-mail us at support@campusx.in Chapters: 00:00 - Introduction to LLM Evaluation Methods 00:36 - Defining the Core Eval Method Mechanism 01:18 - Overview of the 3 Core Execution Styles 02:14 - Practical Review: The Zomato Email Classifier App 03:57 - Method 1: Programmatic / Deterministic Evaluation 05:55 - Understanding Retriever Evaluation and the Recall@K Metric 06:35 - Math Walkthrough: Calculating Recall on a Sample Dataset 13:01 - Optimizing the Retriever Pipeline: Embeddings & K-Values 13:58 - Method 2: Human-Based Evaluation Systems 16:47 - Setting Up a Subjective Rubric (The 1 to 5 Helpful Scale) 19:55 - Managing Multi-Grader Workflows & Disagreement Metrics 24:50 - The 5 Styles of Human Evals: Red Teaming, AB Testing, & Grading 28:09 - Evaluating Human Trade-offs: Quality vs. Cost Overhead 30:05 - Method 3: Model-Graded Evaluation (LLM as a Judge) 31:12 - Setting Up the UPSC Mains Exam Evaluation Case Study 37:37 - Designing Complex Question Rubrics for the Model Judge 42:15 - Prompting the Judge: Structuring Marks, Constraints, & Reasoning 45:01 - Comparing Performance: Human vs. LLM Side-by-Side Scoring 45:41 - Quantifying Judge Variance with Mean Absolute Error (MAE) 48:11 - Shifting Your Mindset to Production-Grade AI Engineering 48:33 - Conceptual Distinctions: Reference-Based vs. Reference-Free Evals 52:41 - Looking Forward: Shifting from Offline to Online App Evals