yq - the yaml power tool | gruchalski.com

The jq tool was a game-changer for JSON on the command line. Before jq manipulating JSON data meant invoking programs in third-party languages. jq changed that for the better. But the world has moved on, more and more people have adopted YAML. There was a need for a YAML processing jq-like tool—the yq.

There were multiple attempts to solve this problem, often sharing the same name.

The yq from Mike Farah is the best one.

The MIT-licensed yq source code is available on GitHub^[1].

§what is yq

After GitHub repository:

yq: a lightweight and portable command-line YAML, JSON and XML processor. yq uses jq like syntax but works with yaml files as well as json, xml, properties, csv and tsv. It doesn’t yet support everything jq does - but it does support the most common operations and functions, and more is being added continuously.

§how do I install yq

yq is written in go. If you have go installed, the easiest method is:

1

go install github.com/mikefarah/yq/v4@v4.34.1

If you don’t have go installed, no worries. There are plenty of other ways to install the tool, simply consult the GitHub repository^[2].

§how do I use it

The documentation is thorough^[3]. The best way to start is to check examples and simply play with it.

There are many easy to find yq tutorials so I’ll spare that here.

However, I’ll share some pretty cool teasers.

§yq teasers

§extracting all helm chart dependencies with versions

1
2
3


curl --silent -L \
    https://github.com/kubeshop/helm-charts/raw/develop/charts/testkube/Chart.yaml \
    | yq '.dependencies[] | .repository + "@" + .version'

file://../testkube-operator@1.13.0
https://charts.bitnami.com/bitnami@12.1.31
https://nats-io.github.io/k8s/helm/charts/@0.19.1
file://../testkube-api@1.13.0
file://../testkube-dashboard@1.13.0
file://../global@0.1.1

§extracting only non-local helm chart dependencies with versions

1
2
3


curl --silent -L \
    https://github.com/kubeshop/helm-charts/raw/develop/charts/testkube/Chart.yaml \
    | yq '.dependencies | filter(.repository | match("^https?")) | .[] | .repository + "@" + .version'

https://charts.bitnami.com/bitnami@12.1.31
https://nats-io.github.io/k8s/helm/charts/@0.19.1

§extracting default images from helm chart values

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


cat > /tmp/runme.sh << 'EOF'
#!/bin/bash

set -eu

DEP_NAME=mongodb
DEP_VERSION=$(curl --silent -L \
    https://github.com/kubeshop/helm-charts/raw/develop/charts/testkube/Chart.yaml \
    | yq '.dependencies[] | select(.name == "'${DEP_NAME}'") | .version' -r)
DEP_REPO=$(curl --silent -L \
    https://github.com/kubeshop/helm-charts/raw/develop/charts/testkube/Chart.yaml \
    | yq '.dependencies[] | select(.name == "'${DEP_NAME}'") | .repository' -r)

HELM_REPO_NAME=${DEP_NAME}-${DEP_VERSION}
helm repo add ${HELM_REPO_NAME} ${DEP_REPO}
helm repo update ${HELM_REPO_NAME}
mkdir -p /tmp/${HELM_REPO_NAME}
cd /tmp/${HELM_REPO_NAME}
helm pull ${DEP_NAME}-${DEP_VERSION}/${DEP_NAME} --version ${DEP_VERSION}
tar xvf ${DEP_NAME}-${DEP_VERSION}.tgz

MONGODB_IMAGE=$(cat /tmp/${DEP_NAME}-${DEP_VERSION}/${DEP_NAME}/values.yaml \
    | yq '.image.registry + "/" + .image.repository + ":" + .image.tag')

cd -
rm -rf /tmp/${DEP_NAME}-${DEP_VERSION}
helm repo remove ${DEP_NAME}-${DEP_VERSION}

echo MongoDB image is: ${MONGODB_IMAGE}
EOF
chmod +x /tmp/runme.sh
/tmp/runme.sh
rm -f /tmp/runme.sh

"mongodb-12.1.31" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "mongodb-12.1.31" chart repository
Update Complete. ⎈Happy Helming!⎈
x mongodb/Chart.yaml
...
x mongodb/charts/common/README.md
/Users/radek/dev
"mongodb-12.1.31" has been removed from your repositories
MongoDB image is: docker.io/bitnami/mongodb:5.0.10-debian-11-r3

§count number of CRDs

1
2
3
4


curl --silent -L \
    https://github.com/istio/istio/raw/1.18.0/manifests/charts/base/crds/crd-all.gen.yaml \
    | yq '. | select(.kind == "CustomResourceDefinition") | .metadata.name' -r -o json \
    | wc -l

§find latest non-literal-latest ory hydra image tag in docker hub

1
2
3


curl --silent \
    https://registry.hub.docker.com/v2/repositories/oryd/hydra/tags?page_size=50 \
    | yq '.results | filter(.name | match("^v")) | .[0].name' -r

v2.2.0

§find latest numerical keycloak image tag available in quay.io

1
2
3


curl --silent -L \
    https://quay.io/api/v1/repository/keycloak/keycloak/tag?limit=50 \
    | yq '.tags | filter(.name | match("^\d")) | .[0].name'

21.1

§find unique statefulset images used by the yugabytedb helm chart

1
2
3
4
5
6
7
8


helm repo add yugabytedb https://charts.yugabyte.com
helm repo update yugabytedb
helm template yb-demo yugabytedb/yugabyte \
    --version 2.19.0 \
    --set resource.master.requests.cpu=0.5,resource.master.requests.memory=0.5Gi,resource.tserver.requests.cpu=0.5,resource.tserver.requests.memory=0.5Gi \
    --namespace yb-demo \
    | yq '. | select(.kind == "StatefulSet") | .spec.template.spec.containers[] | .image' -r -o json \
    | uniq

yugabytedb/yugabyte:2.19.0.0-b190

1

helm repo remove yugabytedb

§apply selected resources only

1
2
3


cat ...yaml \
    | yq '. | select(.kind == "Deployment" or .kind == "StatefulSet") | . |  select(.metadata.name == "yb-tserver")' \
    | kubectl apply -f -

§closing words

yq brings new superpowers to the terminal. Processing YAML, JSON, XML, TSV, and CSV with one tool is absolutely outstanding.

A real must-have tool in modern shell data processing stack.

yq - the yaml power tool

table of contents