I just signed up for a new course from Astronomer in a few days, and honestly, I think I’ll be trying to convince my company to migrate over to them. We are currently relying on AWS MWAA (Managed Workflows for Apache Airflow), but boy, this has been quite bad. Lately, I’ve been putting a lot of effort into optimizing our setup. I adapted and improved all our existing Airflow code, made a heavy bet on deferred operators, moved our workloads—specifically the non-intense but long-running processes like dbt calls—into ECS, and meticulously prepared our migration to version 3.0.6. Read more

With the new Databricks jobs API 2.1 you have different parameters depending on the kind of tasks you have in your workflow. Like: jar_params, sql_params, python_params, notebook_params… And not always the airflow operator is ready to handle all of the. If we check the current release of the DatabricksRunNowOperator, we can see that there is only support for: notebook_params python_params python_named_parameters jar_params spark_submit_params And not the query_params mentioned earlier. But there is a way of combining both, there is a param called jsob that allows you to write the payload of a databricksrunnow and it will also merge the content of the JSON with your named_params! Read more