Working with json using jq
jq is a tool for working with json objects from the command line.
Installing jq
You can install jq using your usual package managers, e.g:
# Debian/Ubuntu
$ apt-get install jq# Fedora
$ dnf install jq# macOS
$ brew install jq
Parsing a json object into jq
jq can work with json objects that are in the form of a variable, e.g.:
$ json_data='{"name":"Tony","age":"35","gender":"male"}'
$ echo ${json_data}
{"name":"Tony","age":"35","gender":"male"}…or a json file, e.g.:
$ cat sample.json
{"name":"Tony","age":"35","gender":"male"}You can feed your json object to jq using pipes:
$ echo $json_data | jq '.'
{
"name": "Tony",
"age": "35",
"gender": "male"
}jq pretty-prints all output by default. 🙂
The single-quotes '.' is how you set jq filters. In my example, I’ve set my filter to ., This is the simplest filter and it instructs jq to output all data that it has received.
Here’s the same example, but this time piping a json file into jq:
$ cat sample.json | jq '.'
{
"name": "Tony",
"age": "35",
"gender": "male"
}Alternatively you can pass in json files as a jq argument:
$ jq '.' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}To get the value of the “age” key, you can do:
$ jq '.age' sample.json
"35"$ jq -r '.age' sample.json
35
The -r flag is used for stripping out the double quotes.
The .age filter is used to drill down to the part of the json object that we’re interested in.
json object are often made up of key-values, where the values can be another json object. You can use filters to drill down these nested json objects too, for example to output Tony’s favourite fruit, we can do:
$ echo '{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis"
}
}' | jq '.favourite.fruit'
"apple"Notice that jq has its own builtin pipe system. This is how you can edit some json data. The {} brackets are used by jq for creating/editing json data. I’ll cover more about this later.
Builtin Operators
For the next few demos I’m going to use the following json file:
$ echo '{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis"
}
}' > tony.jsonjq comes with a bunch of handy operators. Let’s start with the pipe operator, |. This is similar to how bash pipes work, but in the context of jq. For example let’s use pipes to print out Tony’s favourite fruit again:
$ jq '.favourite | .fruit' tony.json
"apple"Earlier we printed out the fruit by drilling down to it using the .favourite.fruit expression. This time we’re using the pipe operator to apply 2 filters one after the next, to end up with the same result.
Next we have the assignment operator, =. You can use this to edit a key’s value:
$ jq '.favourite.color = "blue"' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}If the new value is stored in a variable, then you set it using the --arg flag:
$ tonyNewColor=green
jq --arg newColor $tonyNewColor '.favourite.color = $newColor' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "green",
"fruit": "apple",
"sport": "tennis"
}
}Here’s an example of how you can use the pipe operator to string together multiple edits:
$ jq '.favourite.color = "blue" | .age = 36' tony.json
{
"name": "Tony",
"age": 36,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}So far we’ve only seen how to edit a key’s value. However what if you want to add your own key-value? In that case you can use the update-assignment operator, |=. You can use this to add a new list-item to the favourites array:
$ jq '.favourite |= . + {"movie": "Die Hard"}' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "red",
"fruit": "apple",
"sport": "tennis",
"movie": "Die Hard"
}
}Here we used a combination of the update-assignment operator,|= and the addition operator, +. This has the effect of
If the list item you want to add already exists, then it will just end up updating the existing item instead, e.g. here’s how to change our favourite color to blue:
$ jq '.favourite |= . + {"color": "blue"}' tony.json
{
"name": "Tony",
"age": 35,
"gender": "male",
"favourite": {
"color": "blue",
"fruit": "apple",
"sport": "tennis"
}
}Builtin Functions
jq has a lot of builtin functions. For example the keys function is used to get all the keys in a json object and list them out:
$ jq '. | keys' sample.json
[
“age”,
“gender”,
“name”
]jq has it’s own internal piping system as show here. Here we’re piping all the
Here we’re using the pipe operator to pass in the age value to the builtin type function:
$ jq '.age | type' tony.json
"number"This function prints out the value’s datatype. Next, here’s an example of the builtin length function:
$ jq ‘.favourite.sport | length’ tony.json
6This says that ‘tennis’ is 6 character long.
Working with arrays (aka lists)
In json, and array is indicated by [], here’s an example of an array at top level of this json object
$ cat sample.json
[{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]$ jq '.' sample.json
[
{
"name": "Tony",
"age": "35",
"gender": "male"
},
{
"name": "Jane",
"age": "30",
"gender": "female"
}
]
You can use the keys filter to print out each list item’s number:
$ jq 'keys' sample.json
[
0,
1
]This time top level of our json data is an array containing 2 items. We can drill inside this array and view it’s content:
$ jq '.[]' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}
{
"name": "Jane",
"age": "30",
"gender": "female"
}Let’s now try getting the value for the ‘age’ key:
$ jq '.[].age' sample.json
"35"
"30"This ends up getting the age out from each item. To just get Tony’s, we have to specify the corresponding item number:
$ jq '.[0].age' sample.json
"35"Alternatively you can first drill down to Tony’s list item:
$ jq '.[] | select(.name=="Tony")' sample.json
{
"name": "Tony",
"age": "35",
"gender": "male"
}Here we piped jq’s content from the .[] to a jq’s select query.
After that, you can drill down (or pipe) further to get Tony’s age:
$ jq '.[] | select(.name=="Tony").age' sample.json
"35"$ jq '.[] | select(.name=="Tony") | .age' sample.json
"35"
Looping through an Array
Let’s say we have this sample.json:
$ cat sample.json
{ "employees": [{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]}$ jq '.' sample.json
{
"employees": [
{
"name": "Tony",
"age": "35",
"gender": "male"
},
{
"name": "Jane",
"age": "30",
"gender": "female"
}
]
}
Here we have the employees key which is storing a 2 item array. To only print the second item, you can do:
$ jq '.employees[1]' sample.json
{
"name": "Jane",
"age": "30",
"gender": "female"
}Alternatively you can loop through each item using bash. To do that you need to first compress each item into it’s own line using the -c flag:
$ jq -c '.employees[]' sample.json
{"name":"Tony","age":"35","gender":"male"}
{"name":"Jane","age":"30","gender":"female"}Then you can use bash loop to process each line in turn, for example
#!/bin/bashfor employee in $(jq -c '.employees[]' sample.json)
do
employee_name=$(echo ${employee} | jq '.name')
employee_age=$(echo ${employee} | jq '.age') echo "employee name is: ${employee_name}"
echo "employee age is: ${employee_age}"
echo ""done
This outputs:
employee name is: "Tony"
employee age is: "35"employee name is: "Jane"
employee age is: "30"
Creating and editing json objects
You can create and modify json data using {} brackets. Here’s how to create json data from scratch:
$ jq -n --arg my_username "john" --arg my_id "25" '{
"username": $my_username,
"id": $my_id
}'Here we use -n to tell jq to start with a null json object. Then we used the content inside the {} as a template to build out a custom json object. We also passed in some variables to the template using the --arg flags:
{
"username": "john",
"id": "25"
}Instead of using the --arg flag, you can use the content inside an existing json file to generate a new json file. Lets use our sample.json as our source data:
$ cat sample.json
[{"name":"Tony","age":"35","gender":"male"},{"name":"Jane","age":"30","gender":"female"}]Now let’s say we only want us Tony’s data to create our new json object. So we need to run:
$ cat sample.json | jq '.[] | select(.name=="Tony")' | jq '{
"username": .name,
"current_age": .age
}'Notice this time we call variables using . so instead of $name we used .name.
When we run the above, we get:
{
"username": "Tony",
"current_age": "35"
}Miscellaneous
jq can also minify the output back again too using the compact flag:
$ jq '.' sample.json | jq -c '.'
{"name":"Tony","age":"35","gender":"male"}To just output the keys, you do:
$ jq keys sample1.json
[
"age",
"gender",
"name"
]Advanced examples
Here’s how to drill down a complex json, edit it, and then reapply it.
kubectl get deployments {deployment-name} -o json | \
jq '(.spec.template.spec.containers[] | select(.name | contains("{container-name}")) | .env[] | select(.name | contains("{environment-variable-name}")) | .value) |= "{new-value}"' | \
kubectl -f - In this example we are editing one of the container’s environment variable.
Here’s another example of taking a backup of secrets. Here we delete unwanted bits:
kubectl get secrets -o json | jq 'del(.items[].metadata ["creationTimestamp", "namespace", "resourceVersion", "generation", "selfLink", "uid"] )'