LLM Evaluation

Currently Playing: What are LLM Benchmarks | The Evolution of AI Knowledge Benchmarks | CampusX

In this lesson of our LLM Evaluation Masterclass, we take a deep dive into the Knowledge & Reasoning Capability of Large Language Models. Instead of looking at random leaderboards, we trace the full evolutionary story of knowledge benchmarks—from the early days of broad factual testing to frontier-level research exams. This lecture was taken live for insiders, join here: https://youtu.be/j_G30FLmCcw website: https://benchwiki.vercel.app/ 📱 Grow with us: CampusX' LinkedIn: https://www.linkedin.com/company/campusx-official CampusX on Instagram for daily tips: https://www.instagram.com/campusx.official My LinkedIn: https://www.linkedin.com/in/nitish-singh-03412789 Discord: https://discord.gg/PsWu8R87Z8 E-mail us at support@campusx.in Chapters: 00:00 - Introduction & Recap of Model Evaluation 01:03 - The 8 Core LLM Capabilities & Structuring Benchmark Study 04:22 - What is Knowledge Capability? (Parametric Memory) 07:32 - 2020: MMLU Arrives (Testing Breadth Across 57 Subjects) 10:11 - The First Branch: Testing Reliability with TruthfulQA 14:12 - The Second Branch: Human Exam Baselines with AGI Eval 16:47 - The Third Branch: Depth of Knowledge with GPQA 18:28 - Fixing Flaws: Upgrading to MMLU Pro (10 Options & Reasoning) 20:01 - The Modern Frontier: Humanity's Last Exam (HLE) & SimpleQA 24:00 - Summary Roadmap of the 7 Key Knowledge Benchmarks 25:35 - Introducing BenchWiki: Our Open Benchmark Database 28:47 - Deep Dive 1: MMLU Architecture, Log-Likelihoods & Saturated Limits 43:23 - Deep Dive 2: TruthfulQA, Misconceptions, & Alignment-Stage Contamination 57:46 - Deep Dive 3: AGI Eval, Bilingual Exams, & Human vs. LLM Baselines 01:05:38 - Deep Dive 4: GPQA (Google-Proof Q&A) & The Diamond Dataset 01:16:46 - Deep Dive 5: MMLU Pro (10 Options, 14 Disciplines, & Reasoning Bias) 01:27:04 - Deep Dive 6: SimpleQA (Open-Ended Short Answers & Calibration Metrics) 01:39:29 - Deep Dive 7: Humanity's Last Exam (Breadth x Depth, Multimodality, & Private Sets) 01:49:21 - Conclusion & Next Steps in the Series