I’ve been doing a lot of blogging lately. You might be tired of hearing me complain about my work, but I feel like this is helping me—like a mantra—to find balance and move on.
It’s been a good ride: In 2018, I started in data by maintaining a data platform based on Hive (Azure HDInsight at the time, coordinated with Azure Batch and webjobs), and then I helped maintain the migration to Databricks. At that time, I was mostly an ETL developer. I learned Scala (and fell in love with it) and wrote a lot of complex business logic.
In 2021, I moved to Wallbox as a platform engineer. It was fun. I did complex, near-real-time processing to feed our Druid platform and also tested all the new Databricks features. Most importantly, I implemented the Unity Catalog migration entirely on my own. It took me a while—almost a month—to migrate it and make it work.
I’ve never been much of a platform guy, but I find a lot of value in providing tools. I like to build stuff, but I hate when there is friction. Why do our dbt tests take 50 minutes? Why do we have so many databases? Why does nobody know how much we spent last week and on what?
Those are fun problems to solve, and lately, I haven’t dug enough into them.
I wanted to set some goals for this year—nothing too crazy, just pet projects to stay sharp, to learn to love the problem, to… yeah, just enjoy it.
I’m thinking about making my own small TMB platform, orchestrated with Docker Compose, and relying on DuckDB + dbt (and maybe dlt!).
I will try to share the process in the next few posts.
Also, some fun news: I’m ditching Windows for Linux. I miss my terminal!