AWS released a hot patch on 3.2.1 fixing the recycle workers. We tested it thoroughly on dev for a week. In the meantime I took the astronomer course which was genuinely fun. The curriculum leaned hard into best practices—how to structure DAGs, think about observability, and gradually adopt the new Task SDK without turning every pipeline into a science project. There were some really neat bits around the new trigger API, including a pattern for merging cron schedules so you don’t end up with fifteen nearly identical DAGs firing at slightly different minutes. Good stuff. Read more

I just signed up for a new course from Astronomer in a few days, and honestly, I think I’ll be trying to convince my company to migrate over to them. We are currently relying on AWS MWAA (Managed Workflows for Apache Airflow), but boy, this has been quite bad. Lately, I’ve been putting a lot of effort into optimizing our setup. I adapted and improved all our existing Airflow code, made a heavy bet on deferred operators, moved our workloads—specifically the non-intense but long-running processes like dbt calls—into ECS, and meticulously prepared our migration to version 3.0.6. Read more