The YugabyteDB RPC API isn’t an official API, and that’s what makes it interesting. The whole distributed aspect of YugabyteDB is based on that API. Digging through it is a perfect way to understand the internals of the database. The RPC API can also be used to automate various aspects of the database. I’ll come back to this subject in near future.
Another reason for investigating the RPC API was to have a lightweight version of yb-admin tools. I run everything in containers. When I want to connect to a cluster, I start another container with a client and connect with it to the cluster. The YugabyteDB Docker image isn’t tiny, in fact, 18.104.22.168-b89 is 1.77GB.
One point seventy seven gigabyte!
I’m kinda not a fan of pulling in an image of such size just to execute
list_all_masters. Neither I am a fan of doing
docker exec in a production setting. I wanted to have a tool comparable to yb-admin but of a smaller footprint. And I wanted a library, an embeddable library to help automate the hell out of it.
And that’s why some months ago I’ve written the very first version of the YugabyteDB client library for Go. Today, I’ve published the 0.0.2-beta.1 release. Here’s a brief overview of what it can do:
- given a list of master addresses, find the leader master and use that for further communication with the cluster
- handle leader change, auto-discovery of a new leader, handles redelivery
- auto-reconnect on connection loss, handles redelivery
- wraps API error types and exposes them as Go errors
- utilities to deal with various ID types and hybrid time
- automatically discovers available messages and services, generic execution API is completely type-driven
- client discovers the service and operation based on the request type
- configurable logging sink, via
- configurable metrics sink
The only feature unsupported feature are side cars but they will come.
What’s the best way to sharpen your appetite? Code, obviously! So here’s how to start with this client:
Maybe you need to adjust the master addresses list in line
The client comes with
github.com/radekg/yugabyte-db-go-client/testutils/tserver packages. Those contain a bunch of
dockertest based utilities for writing tests using containers. Simply write your integration test and have a cluster right there, when running
go test (use some higher timeout, these need some time for the cluster to spin up).
testutils/master/master_test.go: shows how to spin up a single master and run some tests against it
testutils/tserver/tserver_test.go: starts a mini-cluster and goes as far as executing an SQL command against it.
That’ll get you going.
§there’s … more
Being able to execute calls is one thing, knowing what to execute is a totally different beast.
There’s this little repository here: radekg/yugabyte-db-go-client-api. It contains a naive implementation of some of the yb-admin commands. That code is currently written against v0.0.1-beta.X version of the client (so 22.214.171.124 and 126.96.36.199 YugabyteDB) but it might be a good source of inspiration.
There’s a rewrite planned for that repository so best is not to invest too much time integrating that as a dependency in your own code.
The client depends on the radekg/yugabyte-db-go-proto library. That’s where the extracted *.proto files and generated Go code live. Have a look at the readme in that repository to find out more on how to generate your own code from YugabyteDB sources.
The radekg/yugabyte-db-go-proto library versioning follows the YugabyteDB versioning.
§a couple of closing notes
- 0.0.1-beta.4 is the last version for YugabyteDB 188.8.131.52 and 184.108.40.206
- 0.0.2-beta.1 is targeted for YugabyteDB 2.11.2+, there are some protobuf changes which might not be exactly compatible with previous database versions, I don’t really know yet, haven’t tried it myself, but I doubt it’ll work
Would I recommend using this in production? Yeah, why not. It’s used in production. But please keep in mind, there’s no warranty!
If you find a bug, feel free to submit a pull request.
If your company is looking for someone who can help you migrate your database to the cloud and/or your company needs a solid partner who knows a thing or two about YugabyteDB, feel free to reach out to me at email@example.com.