I’ve been deep in YugabyteDB trenches for the last 6 months. At Klarrio, we are building a Database as a Service solution for the Data Services Hub. This is a self-managed, multi-tenant solution designed to run on top of Apache Mesos. We have a small team doing all kinds of integration work. I am focusing on the core database rollout.
Working with new technology, for me, often means going all in, deep into the darkness of the source code. I like to know how things work. It’s no different with YugbyteDB. I’m not the kind of guy who takes an upstream component and just runs it in production without understanding how it breaks. The best way to figure out how things work is to take them apart and put back together.
In the last 6 months, I’ve been doing all sorts of crazy things with YugabyteDB. To some, it might look like procrastinating but I truly believe that running a complex system in production without understanding how that system works is a very bad idea. And so, I’ve been doing things like:
- Building a number of Docker[1], Mesos and Kubernetes based clusters,
- Running them at different sizes and in different configurations,
- Building my own Docker images with all sorts of bells and whistles included,
- Digging deep in the source code of YugabyteDB and PostgreSQL,
- First time ever writing PostgreSQL extensions and figuring out how to build YugabyteDB targeted extensions or soft PostgreSQL multi-tenancy,
- Contributed a couple of features:
- Building my own bench for working with YugabyetDB source code[4] and built it from sources so many times, in my dreams, I sometimes hear all 44 cores of my lab server blasting out at full speed at 3am,
- Writing a bunch of tools[5] for YugabyteDB, looked into monitoring, observability, logging, and operation of a cluster,
- Speaking at the Distributed SQL Summit 2021[6].
I cannot thank the Yugabyte team enough for the help they have provided me along the way. They’ve been patient, supportive, professional. The depth of technical answers to my questions is outstanding.
Credit also has to go to the client and my employer, who supported me on this journey.
It’s fair to say that I’ve learned a lot. But I am in no way an expert. You see, the nature of my work is to deliver up to spec projects to clients. I do R&D, architectural work, implementation, full delivery: functional tests, operations and en user documentation, sometimes day two operations for a bit.
But once the project is finished, I move to another one. What I learn never goes to waste. To a certain extent what I learn will always be relevant in future projects. I often find myself building all sorts of crazy stuff with technology from previous projects. A false premise, building stuff I dream about becoming products but it always lacks focus and I am not really in a position to build a product today. I very much enjoy doing all that crazy lab work but let’s face it: that’s actually procrastinating.
I learn a lot, I love it, it keeps me honest. But it is procrastinating. All that lab stuff ends up behind closed doors and rots away.
While working on this huge YugabyteDB rollout, I could observe those same emotions again. What can I do differently this time? Around August an idea started growing in my head that maybe instead of building imaginary products, I should write a book.
Have you ever seen the book from Jacek Laskowski on Spark internals[7]? This book describes each individual Spark internal component so one can really get to know how Spark works.
Sidetrack: back in 2015 I’ve spent good six months bending Spark in all sorts of ways. I made it work in Mesos with Docker bridge networking with the patched Akka build[8] before it was all hip.
Suddenly I was toying with the idea of writing something similar about YugabyteDB. And then, about a month ago, Jonathan from Apress reached out on LinkedIn asking if I would be interested in writing a book on YugabyteDB.
That gave it focus. The answer is: yes. Someone I used to work with, who also wrote a book, said:
Writing a book takes you out of your comfort zone and forces you to look at every corner of the product, things you’d never touch in your daily work.
Sounds like a fantastic way to learn YugabyteDB inside out. And yes, I know there’s no money in writing technical books but money isn’t the driver for me here. For once to have continuity and contribute in a meaningful way—those are the drivers.
As I learned later, Jonathan also reached out to Franck Pachot—Developer Advocate at Yugabyte.
So I’ve spoken to Franck and we agreed to co-author a book. It’s all very early stages, the proposal is being written, priorities have to be taken care of, everybody has be aligned, everyone has to feel comfortable with choices being made. It can still go in every direction.
Stay tuned.