AWS released a hot patch on 3.2.1 fixing the recycle workers. We tested it thoroughly on dev for a week. In the meantime I took the astronomer course which was genuinely fun. The curriculum leaned hard into best practices—how to structure DAGs, think about observability, and gradually adopt the new Task SDK without turning every pipeline into a science project. There were some really neat bits around the new trigger API, including a pattern for merging cron schedules so you don’t end up with fifteen nearly identical DAGs firing at slightly different minutes. Good stuff. Read more

I just signed up for a new course from Astronomer in a few days, and honestly, I think I’ll be trying to convince my company to migrate over to them. We are currently relying on AWS MWAA (Managed Workflows for Apache Airflow), but boy, this has been quite bad. Lately, I’ve been putting a lot of effort into optimizing our setup. I adapted and improved all our existing Airflow code, made a heavy bet on deferred operators, moved our workloads—specifically the non-intense but long-running processes like dbt calls—into ECS, and meticulously prepared our migration to version 3.0.6. Read more

I’ve been doing a lot of blogging lately. You might be tired of hearing me complain about my work, but I feel like this is helping me—like a mantra—to find balance and move on. It’s been a good ride: In 2018, I started in data by maintaining a data platform based on Hive (Azure HDInsight at the time, coordinated with Azure Batch and webjobs), and then I helped maintain the migration to Databricks. At that time, I was mostly an ETL developer. I learned Scala (and fell in love with it) and wrote a lot of complex business logic. Read more

I quit my job after the trial period because I didn’t find what I expected from a startup, nor did I get the benefits of an established company. It was a dealbreaker. However, I will remain for an extra 3 months since there is a project I want to be involved in. I’ve been granted more flexibility, and I want to use this time to prepare myself for what’s next. Read more

Lately I’m not exactly enjoying my job. Snowflake step curve from a data engineer perspective is really hard. Yesterday my boss asked me to test the Snowflake integrated MCP from Cursor. He’d been looking into Snowflake OAuth access, so I went to check the resources. Once again—as of January 27th, 2026—the Terraform resource for this still doesn’t exist (though the data source does, for some reason). For connecting to snowflake you need to define a security integration. Read more

I switched jobs all over again. I’m working in a data company called Shalion, where we provide insights about retail products online. If you’re brave enough to handle Snowflake’s quirks, we’re hiring… I’m working on revamping our permission grant system from permifrost to a terraform managed approach. We also decided to embrace a 2 layer permission grant, using Access Roles (AR) and Functional Roles (FR). The former has access to the resources with the typical permissions (SELECT, USAGE, CREATE)…, while the latter is the one that we assign to people and just groups access roles. Read more

2025 ha sido un año bastante loco. Pasé la mayor parte del año sentando las bases de una startup en el mundo de la seguridad, haciendo frontend, backend, mucho devops y… poco data. Me pilló de lleno la era de la AI y como siempre, el campo ajeno es más verde. Viví la burbuja de v0 y mi trabajo respecto a este, fue bastante miserable. El proyecto a medida que crecía dejaba de ser funcional. Un cambio manual que arreglase el mismo era incompatible. Read more

For the past seven years, I worked in data, and I have mixed feelings about it. I still believe data is the most important part of any app, but it’s meaningless without the app itself. Now that I’m working at a startup, I’ve decided to focus on building things. To start, I revisited one of my older projects: a PDF parser about professor designations in the Canary Islands, where one of my best friends works as a teacher. Read more