AWS released a hot patch on 3.2.1 fixing the recycle workers. We tested it thoroughly on dev for a week. In the meantime I took the astronomer course which was genuinely fun. The curriculum leaned hard into best practices—how to structure DAGs, think about observability, and gradually adopt the new Task SDK without turning every pipeline into a science project. There were some really neat bits around the new trigger API, including a pattern for merging cron schedules so you don’t end up with fifteen nearly identical DAGs firing at slightly different minutes. Good stuff. Read more

I just signed up for a new course from Astronomer in a few days, and honestly, I think I’ll be trying to convince my company to migrate over to them. We are currently relying on AWS MWAA (Managed Workflows for Apache Airflow), but boy, this has been quite bad. Lately, I’ve been putting a lot of effort into optimizing our setup. I adapted and improved all our existing Airflow code, made a heavy bet on deferred operators, moved our workloads—specifically the non-intense but long-running processes like dbt calls—into ECS, and meticulously prepared our migration to version 3.0.6. Read more

With the new Databricks jobs API 2.1 you have different parameters depending on the kind of tasks you have in your workflow. Like: jar_params, sql_params, python_params, notebook_params… And not always the airflow operator is ready to handle all of the. If we check the current release of the DatabricksRunNowOperator, we can see that there is only support for: notebook_params python_params python_named_parameters jar_params spark_submit_params And not the query_params mentioned earlier. But there is a way of combining both, there is a param called jsob that allows you to write the payload of a databricksrunnow and it will also merge the content of the JSON with your named_params! Read more