jq: Working With JSON on the Command Line
jq filters, transforms and formats JSON in a shell pipeline. Learn the filters, the operators, and the patterns you will use daily.
By Velloreth
What jq does
jq reads JSON, applies a filter, and writes JSON. The filter is a small program, and the same filter is applied to every value in the input.
echo '{"name":"alice","age":30}' | jq '.name'
# "alice"
The dot is the identity filter. It returns the input unchanged.
echo '{"a":1}' | jq '.'
# {
# "a": 1
# }
A dot followed by a name selects a key.
echo '{"name":"alice"}' | jq '.name'
# "alice"
A dot followed by a number selects an array index.
echo '[10,20,30]' | jq '.[1]'
# 20
The flags that matter
| Flag | Meaning |
|---|---|
-r | raw output, no quotes on a string |
-c | compact output, one line |
-e | set the exit code from the result |
-n | use null as the input |
-s | slurp the whole input into an array |
-S | sort the keys |
-j | join the output without a newline |
-r is the one you will use most. A value that goes into a shell variable or a URL should not have quotes.
TOKEN=$(curl -s https://api.example.com/token | jq -r '.access_token')
-c is for a pipeline. One line per value is easier to pass to another tool.
curl -s https://api.example.com/users | jq -c '.[]'
-e sets the exit code. It is 0 if the last output was neither false nor null, and 1 otherwise, which lets a script branch.
if jq -e '.error' response.json > /dev/null; then
echo "the response has an error"
fi
Selecting and composing
The pipe composes filters, passing the output of one to the next.
echo '{"user":{"name":"alice"}}' | jq '.user.name'
# "alice"
The comma produces multiple outputs.
echo '{"a":1,"b":2}' | jq '.a, .b'
# 1
# 2
An object constructor builds a new object.
echo '{"first":"alice","last":"smith"}' | jq '{name: (.first + " " + .last)}'
# {
# "name": "alice smith"
# }
An array constructor builds a new array.
echo '{"a":1,"b":2}' | jq '[.a, .b]'
# [1, 2]
Working with arrays
.[] iterates over the elements.
echo '[1,2,3]' | jq '.[]'
# 1
# 2
# 3
map applies a filter to every element.
echo '[1,2,3]' | jq 'map(. * 2)'
# [2, 4, 6]
select keeps the elements that match.
echo '[{"age":25},{"age":40}]' | jq 'map(select(.age > 30))'
# [{"age": 40}]
length counts, and sort_by orders.
echo '[3,1,2]' | jq 'length'
# 3
echo '[{"n":3},{"n":1}]' | jq 'sort_by(.n)'
group_by groups, and unique removes duplicates.
echo '[{"t":"a"},{"t":"b"},{"t":"a"}]' | jq 'group_by(.t) | map({t: .[0].t, n: length})'
Handling a missing key
A missing key returns null.
echo '{}' | jq '.name'
# null
// provides a default.
echo '{}' | jq '.name // "unknown"'
# "unknown"
? suppresses an error.
echo '{"a":1}' | jq '.a.b?'
# null
has tests for a key.
echo '{"a":1}' | jq 'has("a")'
# true
type reports the type, which is useful when the shape is not what you expected.
echo '[]' | jq 'type'
# "array"
Common patterns
Extract a field from every object in an array.
curl -s https://api.example.com/users | jq -r '.[].email'
Filter and reshape.
curl -s https://api.example.com/users \
| jq '[.[] | select(.active) | {id, email}]'
Count the elements.
curl -s https://api.example.com/users | jq 'length'
Sum a field.
echo '[{"n":1},{"n":2}]' | jq 'map(.n) | add'
# 3
Build a CSV.
curl -s https://api.example.com/users \
| jq -r '.[] | [.id, .email] | @csv'
Read a value into a shell variable.
VERSION=$(jq -r '.version' package.json)
Update a value and write it back.
jq '.version = "2.0.0"' package.json > tmp.json && mv tmp.json package.json
Merge two objects.
jq -s '.[0] * .[1]' a.json b.json
Slurp a stream of objects into an array.
jq -s '.' <(echo '{"a":1}'; echo '{"b":2}')
Why jq is worth learning
jq is a small language, and the investment is repaid quickly because the alternative is a script in a general-purpose language that does the same thing with more code. A filter that extracts a field from an API response is one line, and the same filter composes with curl, with a loop, and with another jq invocation. The second reason is that jq is present in most environments where JSON is processed: a container image, a CI runner, and a developer’s machine. The third is that a jq filter is a precise description of a transformation, which makes it reviewable in a way that a bespoke script is not. The cost is the syntax, which is terse and unfamiliar at first, and the payoff is a tool that turns a multi-line script into a one-line command.
A jq workflow
A jq filter is easiest to build incrementally, because each step is a small transformation that can be checked on its own. The first step is to look at the shape of the data, which is a jq with the identity filter and a compact flag, or a jq that reports the type of the top-level value. The second is to select the part that matters, which is a path such as a key or an index. The third is to iterate, which is a map over an array or a stream over the elements. The fourth is to reshape, which is an object or array constructor. The fifth is to format, which is a raw flag for a value that goes into a variable or a CSV conversion for a table. Building the filter in those steps means a mistake is visible at the step where it was introduced, rather than as a wrong result at the end of a long pipeline.
Common mistakes
Forgetting -r. The value has quotes, which breaks a URL or a comparison.
Not quoting the filter. The shell interprets the special characters. Use single quotes.
Assuming a key exists. A missing key is null, and a null in a string concatenation produces an error or an unexpected value. Use // or ?.
Using .[] on an object and expecting an array. It iterates the values, which is often what you want, but the result is a stream rather than an array. Wrap it in [...] to collect.
A filter that produces a stream where an array is expected. map returns an array; .[] returns a stream.
Editing a file in place with a redirect to the same file. The shell truncates the file before jq reads it. Write to a temporary file and move it.
Ignoring the exit code. A jq filter that produces null exits 0 by default. Use -e when the result is a condition.
A quotable summary
jq applies a filter to JSON and writes JSON, where the dot is the identity, a dot and a name selects a key, and a pipe composes filters. Use -r for a raw value, -c for compact output, and -e to set the exit code from the result. map applies a filter to every element, select keeps the matching ones, and // provides a default for a missing key. Quote the filter in single quotes, and write to a temporary file when editing a file in place.
Related reading
For the shell around this, see linux-command-line-basics and for the search tools, grep-and-find-guide. For the broader context, json-vs-yaml-vs-toml and api-error-responses-guide.