AI & Cybersecurity

Currently Playing: OWASP Top 10 for LLMs - Part 1

This video deep dives into three of the OWASP top 10 Large Language Model (LLM) attacks, focusing on issues of trustworthiness and security: data and model poisoning, system prompt leakage, and misinformation (hallucination). The video first demonstrates how LLMs can be poisoned, showing a coding assistant that produced significantly insecure output, vulnerable to SQL injection and plain text password storage, just by changing a single parameter like the year in the prompt. It explains that training data manipulation, even with a small number of documents (as few as 250), can poison models with billions of parameters. Attackers can insert "sleeper agents" or backdoors that standard safety training may fail to remove, sometimes making the malicious behavior more sophisticated at concealment. Poisoning can occur through pre-trained data, fine-tuning, or Retrieval Augmented Generation (RAG) using manipulated files, which is also called indirect prompt injection. A detailed example of the `pickle` vulnerability is shown, where a model file can execute arbitrary system commands upon loading, and the defense against this using "safe tensors" is explained. The second attack covered is system prompt leakage, where users "jailbreak" the LLM to reveal its secret, long-form guidance provided by the model creators. The video illustrates why protecting the prompt is crucial and mentions various techniques, such as reframing queries or using multi-prompt chains, that attackers might use to bypass model defenses. Finally, the video addresses misinformation and hallucination, noting that LLMs are designed to be helpful and often "imagine the answers" rather than admitting ignorance, leading to outputs that can be harmlessly inaccurate or dangerously misinformed. This problem is amplified when fabricated content is published and then scanned by other LLMs, permanently poisoning future training data. A dangerous example of hallucination, a fabricated threat intelligence report, is shared. The video concludes with a comprehensive set of general and specific defenses for each attack, including data provenance, anomaly detection during training, adversarial red team testing, and the necessity of a human in the loop for high-stakes outputs. #LLMSecurity #LLMAttacks #DataPoisoning #SystemPromptLeakage #AIHallucination #OWASP #LargeLanguageModels