Esto es una pequeña maratón, cada día, un tema :) Hoy hablaré de dos cosas interesantes, de data modeling y cost & optimization. Cambio un poco el formato porque esto más bien, será un pequeño dump de mis memorias. Section 6: Cost & Performance Optimization Understand delta optimization techniques, such as deletion vectors and liquid clustering. Delta fue una gran ventaja sobre parquet, antes eliminar datos de una tabla, había que o reescribir la tabla entera, o al menos la partición de esta. Cuando llegó delta, esto se hacía automáticamente, muchísimo más cómodo, ¿pero te imaginas reescribir un fichero de mil millones de filas solo porque quieres eliminar 4 datos? Pues esto pasaba, al menos hasta que llegaron los deletion vectors Read more

Continuamos aprendiendo, hoy con github caído :) Hoy me toca centrarme en Data Sharing and Federation y Data Governance. De nuevo vamos a la guía del PDF: Section 4: Data Sharing and Federation Demonstrate delta sharing securely between Databricks deployments using Databricks to Databricks Sharing (D2D) or to external platforms using the open sharing protocol (D2O). Configure Lakehouse Federation with proper governance across the supported source Systems. Use Delta Share to share live data from Lakehouse to any computing platform. <- Eso es ahora Opensharing, así que no sé. Ya pido perdón porque esta parte es un poco de peñazo teórico, pero es lo que hay. De paso dejo en negrita la tipica pregunta trampa: Read more

Continuando en mi cambio de trabajo por año. Intento volver a databricks. Es la tecnología que mejor se adapta a mí y que me permite mantener casi sin esfuerzo esa mentalidad constante de kaizen. Hace unos meses fue el Databricks Learning Festival donde por completar unos cursos, obtuve un 50% de descuento en certificaciones. Ahora tengo el examen en aproximadamente 11 días y el problema, es que hace dos años que no toco databricks. Hace dos años mi stack se basaba en: Read more

AWS released a hot patch on 3.2.1 fixing the recycle workers. We tested it thoroughly on dev for a week. In the meantime I took the astronomer course which was genuinely fun. The curriculum leaned hard into best practices—how to structure DAGs, think about observability, and gradually adopt the new Task SDK without turning every pipeline into a science project. There were some really neat bits around the new trigger API, including a pattern for merging cron schedules so you don’t end up with fifteen nearly identical DAGs firing at slightly different minutes. Good stuff. Read more

I just signed up for a new course from Astronomer in a few days, and honestly, I think I’ll be trying to convince my company to migrate over to them. We are currently relying on AWS MWAA (Managed Workflows for Apache Airflow), but boy, this has been quite bad. Lately, I’ve been putting a lot of effort into optimizing our setup. I adapted and improved all our existing Airflow code, made a heavy bet on deferred operators, moved our workloads—specifically the non-intense but long-running processes like dbt calls—into ECS, and meticulously prepared our migration to version 3.0.6. Read more

I’ve been doing a lot of blogging lately. You might be tired of hearing me complain about my work, but I feel like this is helping me—like a mantra—to find balance and move on. It’s been a good ride: In 2018, I started in data by maintaining a data platform based on Hive (Azure HDInsight at the time, coordinated with Azure Batch and webjobs), and then I helped maintain the migration to Databricks. At that time, I was mostly an ETL developer. I learned Scala (and fell in love with it) and wrote a lot of complex business logic. Read more

I quit my job after the trial period because I didn’t find what I expected from a startup, nor did I get the benefits of an established company. It was a dealbreaker. However, I will remain for an extra 3 months since there is a project I want to be involved in. I’ve been granted more flexibility, and I want to use this time to prepare myself for what’s next. Read more

Lately I’m not exactly enjoying my job. Snowflake step curve from a data engineer perspective is really hard. Yesterday my boss asked me to test the Snowflake integrated MCP from Cursor. He’d been looking into Snowflake OAuth access, so I went to check the resources. Once again—as of January 27th, 2026—the Terraform resource for this still doesn’t exist (though the data source does, for some reason). For connecting to snowflake you need to define a security integration. Read more