Uploads from CampusX

Currently Playing: RAG Operational Evals: Building Faster & Cheaper RAG Systems | CampusX

This session focuses on the final operational pillar of our RAG Evaluation Suite, moving beyond quality and safety to evaluate whether your application can run reliably, fast, and economically at scale. You will explore why telemetry-driven operational evaluations are essential even in offline environments before production deployment to detect performance and cost regressions. We break down the core mechanics of latency (including P95/P99 percentiles, Time to First Token, and component-level tracing), token economics (input/output cost breakdowns, caching dynamics, and budget SLOs), and system reliability (error categorization, retry tracking, and timeout thresholds), followed by running custom Python evaluation scripts. This lecture was taken live for insiders, join here: https://youtu.be/j_G30FLmCcw 📱 Grow with us: CampusX' LinkedIn: https://www.linkedin.com/company/campusx-official CampusX on Instagram for daily tips: https://www.instagram.com/campusx.official My LinkedIn: https://www.linkedin.com/in/nitish-singh-03412789 Discord: https://discord.gg/PsWu8R87Z8 E-mail us at support@campusx.in Chapters: 00:00 - Introduction & Recap: Completing the 3-Tier RAG Eval Suite 02:00 - What Are Operational Evaluations? (Quality vs. Ops Evals) 06:00 - Why You Must Run Operational Evals Offline (Differential Benchmarking) 14:15 - Core Metric 1: Latency Deep Dive & Why Averages Lie 17:15 - Tail Latency Distributions: Understanding P50, P95, and P99 21:45 - Component-Level Latency: Retriever vs. Generator Breakdown 23:00 - Time to First Token (TTFT) & Streaming Output Architecture 24:45 - Managing Cold Starts, Warm-Up Runs & Output Length Scaling 28:00 - Latency vs. Throughput & Handling API Noise 34:00 - Defining Latency Budgets & Service Level Objectives (SLOs) 36:50 - Writing & Executing the Latency Evaluation Script (`eval_latency.py`) 41:20 - Analyzing Latency Results & Practical Optimization Levers 51:55 - Core Metric 2: Cost & Token Economics in RAG Applications 55:10 - Calculating Cost Per Query & Input vs. Output Pricing Differences 58:00 - Cost Distributions, Budget Caps & Prompt Caching Dynamics 01:00:30 - Running the Cost Evaluation Script (`eval_cost.py`) 01:04:20 - Strategies for Reducing Token Costs (Model Routing & Compression) 01:08:05 - Core Metric 3: System Reliability, Error Rates & Retry Limits 01:10:10 - Categorizing Failures: API Errors, Rate Limits & Timeouts 01:13:40 - Running the Reliability Evaluation Script (`eval_reliability.py`) 01:18:10 - Summary & Next Steps