Uploads from Programming Throwdown

Currently Playing: 185: Workflow Orchestrators

Intro topic: Asymmetric Returns News/Links: • NanoChat by Andrej Karpathy • https://github.com/karpathy/nanochat • Pydantic AI • https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/ • 1000th Starlink this year • https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/ • ChatGPT Apps SDK • https://openai.com/index/introducing-apps-in-chatgpt/ Book of the Show • Patrick • The Will of the Many by James Islington • https://amzn.to/43IfU8Q • Jason • Interview with DHH (Founder of Ruby on Rails) • https://www.youtube.com/watch?v=vagyIcmIGOQ Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h Tool of the Show • Patrick • Factorio • https://www.factorio.com/  • Jason • nip.io (http://nip.io)   Topic: Workflow Orchestrators • Why • Batch jobs (embarrassingly parallel) • Long-running tasks (e.g. transcoding video) • Checkpointing/resuming • How • Message Queues • Containerization • Worker Pools & Autoscaling • History & Backfill • Steps to run workflows: • Containerize the workflow definition and send to the cloud • Containerize all the individual tasks • Submit job(s) • Examples • Airflow • Legacy but dominant • Dagster • Great UX for python developers • Temporal: https://temporal.io/ • The new hotness • Ray • Low-level but very powerful • Kubeflow • Designed for ML workflows, integrated dashboard ★ Support this podcast on Patreon ★ (https://www.patreon.com/programmingthrowdown)