Working with json using jq

jq is a tool for working with json objects from the command line.

Installing jq

You can install jq using your usual package managers, e.g:

# Debian/Ubuntu
$ apt-get install jq
# Fedora
$ dnf install jq
# macOS
$ brew install jq

Parsing a json object into jq

jq can work with json objects that are in the form of a variable, e.g.:

$ json_data='{"name":"Tony","age":"35","gender":"male"}'
$ echo ${json_data}
{"name":"Tony","age":"35","gender":"male"}

…or a json file, e.g.:

$ cat sample.json
{"name":"Tony","age":"35","gender":"male"}

You can feed your json object to jq using pipes:

$ echo $json_data | jq '.'
{
"name": "Tony",
"age": "35",
"gender": "male"
}

jq pretty-prints all output by default. 🙂

The single-quotes '.' is how you set jq filters. In my example, I’ve set my filter to ., This is the simplest filter and it instructs jq to output all data that it has received.

Here’s the same example, but this time piping a json file into jq:

$ cat sample.json | jq '.'
{
"name": "Tony",
"age": "35",
"gender": "male"
}

Alternatively you can pass in json files as a jq argument:

$ jq '.' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}

To get the value of the “age” key, you can do:

$ jq '.age' sample.json
"35"
$ jq -r '.age' sample.json
35

The -r flag is used for stripping out the double quotes.

The .age filter is used to drill down to the part of the json object that we’re interested in.

json object are often made up of key-values, where the values can be another json object. You can use filters to drill down these nested json objects too, for example to output Tony’s favourite fruit, we can do:

$ echo '{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis"
}
}' | jq '.favourite.fruit'
"apple"

Notice that jq has its own builtin pipe system. This is how you can edit some json data. The {} brackets are used by jq for creating/editing json data. I’ll cover more about this later.

Builtin Operators

For the next few demos I’m going to use the following json file:

$ echo '{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis"
}
}' > tony.json

jq comes with a bunch of handy operators. Let’s start with the pipe operator, |. This is similar to how bash pipes work, but in the context of jq. For example let’s use pipes to print out Tony’s favourite fruit again:

$ jq '.favourite | .fruit' tony.json
"apple"

Earlier we printed out the fruit by drilling down to it using the .favourite.fruit expression. This time we’re using the pipe operator to apply 2 filters one after the next, to end up with the same result.

Next we have the assignment operator, =. You can use this to edit a key’s value:

$ jq '.favourite.color = "blue"' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}

Here’s an example of how you can use the pipe operator to string together multiple edits:

$ jq '.favourite.color = "blue" | .age = 36' tony.json
{
"name": "Tony",
"age": 36,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}

So far we’ve only seen how to edit a key’s value. However what if you want to add your own key-value? In that case you can use the update-assignment operator, |=. You can use this to add a new list-item to the favourites array:

$ jq '.favourite |= . + {"movie": "Die Hard"}' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis",
"movie": "Die Hard"
}
}

Here we used a combination of the update-assignment operator,|= and the addition operator, +. This has the effect of

If the list item you want to add already exists, then it will just end up updating the existing item instead, e.g. here’s how to change our favourite color to blue:

$ jq '.favourite |= . + {"color": "blue"}' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}

Builtin Functions

jq has a lot of builtin functions. For example the keys function is used to get all the keys in a json object and list them out:

$ jq '. | keys' sample.json
[
“age”,
“gender”,
“name”
]

jq has it’s own internal piping system as show here. Here we’re piping all the

Here we’re using the pipe operator to pass in the age value to the builtin type function:

$ jq '.age | type' tony.json
"number"

This function prints out the value’s datatype. Next, here’s an example of the builtin length function:

$ jq ‘.favourite.sport | length’ tony.json
6

This says that ‘tennis’ is 6 character long.

Working with arrays (aka lists)

In json, and array is indicated by [], here’s an example of an array at top level of this json object

$ cat sample.json
[{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]
$ jq '.' sample.json
[
{
"name": "Tony",
"age": "35",
"gender": "male"
},
{
"name": "Jane",
"age": "30",
"gender": "female"
}
]

You can use the keys filter to print out each list item’s number:

$ jq 'keys' sample.json
[
0,
1
]

This time top level of our json data is an array containing 2 items. We can drill inside this array and view it’s content:

$ jq '.[]' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}
{
"name": "Jane",
"age": "30",
"gender": "female"
}

Let’s now try getting the value for the ‘age’ key:

$ jq '.[].age' sample.json
"35"
"30"

This ends up getting the age out from each item. To just get Tony’s, we have to specify the corresponding item number:

$ jq '.[0].age' sample.json
"35"

Alternatively you can first drill down to Tony’s list item:

$ jq '.[] | select(.name=="Tony")' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}

Here we piped jq’s content from the .[] to a jq’s select query.

After that, you can drill down (or pipe) further to get Tony’s age:

$ jq '.[] | select(.name=="Tony").age' sample.json
"35"
$ jq '.[] | select(.name=="Tony") | .age' sample.json
"35"

Looping through an Array

Let’s say we have this sample.json:

$ cat sample.json
{ "employees": [{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]}
$ jq '.' sample.json
{
"employees": [
{
"name": "Tony",
"age": "35",
"gender": "male"
},
{
"name": "Jane",
"age": "30",
"gender": "female"
}
]
}

Here we have the employees key which is storing a 2 item array. To only print the second item, you can do:

$ jq '.employees[1]' sample.json
{
"name": "Jane",
"age": "30",
"gender": "female"
}

Alternatively you can loop through each item using bash. To do that you need to first compress each item into it’s own line using the -c flag:

$ jq -c '.employees[]' sample.json
{"name":"Tony","age":"35","gender":"male"}
{"name":"Jane","age":"30","gender":"female"}

Then you can use bash loop to process each line in turn, for example

#!/bin/bashfor employee in $(jq -c '.employees[]' sample.json)
do
employee_name=$(echo ${employee} | jq '.name')
employee_age=$(echo ${employee} | jq '.age')
echo "employee name is: ${employee_name}"
echo "employee age is: ${employee_age}"
echo ""
done

This outputs:

employee name is: "Tony"
employee age is: "35"
employee name is: "Jane"
employee age is: "30"

Creating and editing json objects

You can create and modify json data using {} brackets. Here’s how to create json data from scratch:

$ jq -n --arg my_username "john" --arg my_id "25" '{
"username": $my_username,
"id": $my_id
}'

Here we use -n to tell jq to start with a null json object. Then we used the content inside the {} as a template to build out a custom json object. We also passed in some variables to the template using the --arg flags:

{
"username": "john",
"id": "25"
}

Instead of using the --arg flag, you can use the content inside an existing json file to generate a new json file. Lets use our sample.json as our source data:

$ cat sample.json
[{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]

Now let’s say we only want us Tony’s data to create our new json object. So we need to run:

$ cat sample.json | jq '.[] | select(.name=="Tony")' | jq '{
"username": .name,
"current_age": .age
}'

Notice this time we call variables using . so instead of $name we used .name.

When we run the above, we get:

{
"username": "Tony",
"current_age": "35"
}

Miscellaneous

jq can also minify the output back again too using the compact flag:

$ jq '.' sample.json | jq -c '.'
{"name":"Tony","age":"35","gender":"male"}

To just output the keys, you do:

$ jq keys sample1.json
[
"age",
"gender",
"name"
]

Advanced examples

Here’s how to drill down a complex json, edit it, and then reapply it.

kubectl get deployments {deployment-name} -o json | \ 
jq '(.spec.template.spec.containers[] | select(.name | contains("{container-name}")) | .env[] | select(.name | contains("{environment-variable-name}")) | .value) |= "{new-value}"' | \
kubectl -f -

In this example we are editing one of the container’s environment variable.

Here’s another example of taking a backup of secrets. Here we delete unwanted bits:

kubectl get secrets -o json | jq 'del(.items[].metadata ["creationTimestamp", "namespace", "resourceVersion", "generation", "selfLink", "uid"] )'

Further Reading

--

--

--

Blogger at codingbee.net

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Build A NodeJS App Server With NanoPipe

Hypercalc: Simplifying JavaScript spreadsheet analytics with $summary

What’s new in JavaScript — ES2020

Spread Operator Basics and React

p5.js sketch with physical interface

Send Emails with EJS Template Using NodeMailer

Objects and Its Internal Representation

Basic concept of JavaScript and some problem solving

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sher Chowdhury

Sher Chowdhury

Blogger at codingbee.net

More from Medium

Docker is my {I.D.E} — a developer’s perspective.

Bitbucket Cloud recently stopped supporting account passwords for Git authentication.

Running Gatsby inside a Docker® Container on macOS

Scan SonarQube branches and pull request branches without paid licenses