jq notes

Guy Lapalme , RALI-DIRO, Université de Montréal, June 2024.

[Version française de ce document]

jq is a command line tool that allows you to manipulate.json or .jsonl files for validation, formatting, information extraction or transformation. It can be considered the counterpart of sed or awk for text files containing tab-separated line information. The system is fast and efficient for processing large files, il can also be used in streaming mode .

The official documentation presents a tutorial with a few examples and a manual which lists all the functions illustrated with a short example, but it does not explain their use on interesting tasks. The syntax of jq expressions may seem daunting, as it relies on composing filters rather than using a more usual functional notation. jq is a full-fledged programming language with control structures, variables, function definitions and even modules. Given that all my applications have been limited to a single expression, this document focuses on this aspect of the language.

There are several tutorials on the web, including this one which is excellent, but now full of advertisements. However, they don't emphasize the parts that I think are most useful for the use cases I've encountered, i.e., extracting or transforming JSON values in files.

This document will therefore serve as a memory aid even if, now that I have written it, I am less likely to need it, but I hope that it will be useful to others.

Basic concepts

jq expression is a filter that transforms its input into an output. All elements of the jq language are filters, even a number or string which can be considered as filters which ignore their input and always produce the same value.

Once installed, jq is called from the command line: jq [options] 'filter' file , the filter being surrounded by single apostrophes to prevent its interpretation by the shell before being passed to jq.

A filter is applied to a JSON data stream. jq parses this stream as a sequence of space-separated JSON values that are validated and given as input to the filter one by one. The filter's output(s) are sent to stdout as a sequence of JSON values separated by a newline . By default, the JSON value is displayed with an indentation to highlight the JSON structure (pretty-print). This indentation is not applied if the -c option is specified.

Remember that in addition to primitive values (number, string, boolean, null), JSON provides two data structures:

• object : pairs key :value separated by commas inside curly braces.

• array : values separated by commas surrounded by square brackets.

The following is a selection of functions from the hundreds described in the manual .

Basic filters

• .: identity : returns the input as is: this filter might seem useless, but when applied it to a JSON file, it validates it and outputs an indented version.

• f1 | f2: filter composition : redirects the output of filter f1 to the input of f2 .

• .["key"] : object index : applied to an object, returns the value associated with key if it appears in the object, otherwise null . If key is a simple identifier, we can simplify the notation to .key.

• .[number] : array index : applied to an array, returns the value at the position indicated by the number, number can be negative to index from the end.

• .[]: value generation : applied to an object, returns all values of the object as separate results; .key | .[] can also be written ..key[].

• f1 , f2 : filter concatenation : the output of f2 is added at the end of the output of f1 .

Groupings

• {…}: construction of an object : collects the content which must be of the form key : value in a single object. key is often a character string, but can also be an expression (e.g. a filter) if put in parentheses.

• […]: array construction : combines the content into an array

• (…): grouping : groups a subexpression according to the usual conventions

Filters

• select(f): selection : keep only the elements for which f returns true

• keys: keys of an object : returns an array with the keys of the input object. An array of the values of an object is obtained with .[] (see above).

• to_entries: key-value pairs of an object : transforms an object {keyi:valuei} into an array of objects of the form {"key":keyi:"value":valuei}

• del(key): deletion : remove the key and associated value in an object

• f1 + f2 : addition of filters : depends on the filter types: numbers: arithmetic addition; string or array: concatenation; objects: fusion.

• f1 - f2 : subtraction of filters : depends on the types of filters: numbers: arithmetic subtraction; array: removes from f1 the elements of f2 .

Tests

• == != > >= < <=: comparison according to the usual conventions

• and or not: combination of booleans

• test(regex): check with a regular expression

String processing

• startswith(str) endswith(str): check : does the start or end of string match str .

• split(str) join(str): conversion string <=>array : array of substrings between str, create a string from an array by adding str between each element.

Practical examples

**Tip: Try these examples with [jqTerm] (https://jqterm.com/) or [jqplay] (https://jqplay.org), which allow you to run them directly in a web browser and see the results immediately.

.json file

The following JSON structure represents an excerpt from an English dictionary whose keys are words. Each word is associated with an object which describes information according to its category ( A: adjective, N: noun, V: verb, etc.). Each category is associated with a table of inflections ( tab) and, for some, if the word is common (basic). A noun can be countable (yes), non-countable (no) or both .

json {"comfortable":{"A":{"tab":"a1"}, "basic":true}, "comic":{"A":{"tab":"a1"}, "N":{"tab":"n1", "cnt":"yes"}}, "comical":{"A":{"tab":"a1"}}, "comically":{"Adv":{"tab":"b1"}}, "coming":{"A":{"tab":"a1"}, "N":{"tab":"n1", "cnt":"yes"}, "basic":true}, "comity":{"N":{"tab":"n5", "cnt":"no"}}, "comma":{"N":{"tab":"n1", "cnt":"yes"}}, "command":{"N":{"tab":"n1", "cnt":"both"}, "V":{"tab":"v1"}, "basic":true}, "commandant":{"N":{"tab":"n1", "cnt":"yes"}}, "commandment":{"N":{"tab":"n1", "cnt":"yes"}}, "commando":{"N":{"tab":"n2", "cnt":"yes"}}, "commemorate":{"V":{"tab":"v3"}}, "commemoration":{"N":{"tab":"n1", "cnt":"both"}}, "commemorative":{"A":{"tab":"a1"}}} <code></code>

Operations

keys: array of all words ⇒ ["comfortable", "comic","comical", ...]

to_entries[]|select(.value.basic)|.key: list of words with the basic indicator: as select must be applied to an array, you the entry must be transformed into an array and then only the key is selected. Result on several lines ⇒

json "comfortable" "coming" "command" <code></code>

• to_entries|.[]|{(.key):.value|del(.basic)} : list of JSON structures whose key is the word, but where the basic field has been removed .⇒

19
1
{
2
  "comfortable": {
3
    "A": {
4
      "tab": "a1"
5
    }
6
  }
7
}
8
{
9
  "comic": {
10
    "A": {
11
      "tab": "a1"
12
    },
13
    "N": {
14
      "tab": "n1",
15
      "cnt": "yes"
16
    }
17
  }
18
}
19
...

.jsonl file

Each line of a JSON Lines file is a JSON object describing a word. This file could have been created from the previous one with the expression to_entries[] | {word:.key}+.value

14
1
{"word":"comfortable","A":{"tab":"a1"},"basic":true}
2
{"word":"comic","A":{"tab":"a1"},"N":{"tab":"n1","cnt":"yes"}}
3
{"word":"comical","A":{"tab":"a1"}}
4
{"word":"comically","Adv":{"tab":"b1"}}
5
{"word":"coming","A":{"tab":"a1"},"N":{"tab":"n1","cnt":"yes"},"basic":true}
6
{"word":"comity","N":{"tab":"n5","cnt":"no"}}
7
{"word":"comma","N":{"tab":"n1","cnt":"yes"}}
8
{"word":"command","N":{"tab":"n1","cnt":"both"},"V":{"tab":"v1"},"basic":true}
9
{"word":"commandant","N":{"tab":"n1","cnt":"yes"}}
10
{"word":"commandment","N":{"tab":"n1","cnt":"yes"}}
11
{"word":"commando","N":{"tab":"n2","cnt":"yes"}}
12
{"word":"commemorate","V":{"tab":"v3"}}
13
{"word":"commemoration","N":{"tab":"n1","cnt":"both"}}
14
{"word":"commemorative","A":{"tab":"a1"}}

Operations

• .word : list of all words ⇒ ["comfortable", "comic","comical", ...]

• select(.basic)|.word : list of words with the basic indicator ⇒ 

swift "comfortable" "coming" "command" <code></code>

• {(.word):.|del(.basic)} : list of JSON structures whose key is the word, but where the basic field has been removed. ⇒

swift { "comfortable": { "word": "comfortable", "A": { "tab": "a1" } } } { "comic": { "word": "comic", "A": { "tab": "a1" }, "N": { "tab": "n1", "cnt": "yes" } } } ... <code></code>

It is possible to process the lines of a .jsonl file as an array of structures by specifying the --slurp ( -s ) option.

For the inquisitive reader : with the -s option , the expression

1
1
    reduce . [] as $elem ( {} ; .+ { ($elem|.word) : $elem|del(.word) } )

recreates the JSON structure from the previous section. It starts from an empty object and then for each element of the array, we merge an object having for key the value of the key word and, for value, all the properties of the element except the word field.

Interesting links

• Exploring jq : High-level tips for using jq in conjunction with other systems.

• jq Language Description: more formal definition of the jq language with many interesting insights

• Examples of more complex jq programs:

  • jq Recipes

  • jq Cookbook