If you’ve ever had to parse JSON from your terminal you probably know about jq. It’s basically sed for JSON and it works wonderfully well. If you’ve had to parse YAML from your terminal however, the problem becomes a bit harder. You can either go for some super obscure 15 lines sed and awk combination that has the advantage of being pure bash, or go with a higher level language (ruby or python comes to mind) to actually do the parsing and outputting the result to stdout. In this post I’ll show jyparser, a simple tool (packaged as a nice docker image) that allows you to use a jq-like syntax to parse and also update JSON and YAML files from your terminal using exactly the same commands.

The problem

So imagine you have your app and different JSON files for the different environments your app will be deployed to, with each file containing things like the environment name, the build version currently deployed, etc. Maybe something like:

~ cat my_app.json

{
  "app_name" : "awesome app",
  "build_version" : 1,
  "tags" : ["myTeam", "myCompany"]
}

Now as part of your deployment process you want to read the build_version variable from the JSON file, increase it by 1 and then update the original JSON with the new value.

This would not be super hard to do with plain jq:

~ version=$(cat my_app.json | jq '.build_version')
~ echo $version
1

~ new_version=$((version+1))
~ echo $new_version
2

~ cat my_app.json | jq --arg value $new_version '.build_version |= $value'
{
  "app_name": "awesome app",
  "build_version": "2",
  "tags": [
    "myTeam",
    "myCompany"
  ]
}

It’s not too hard but it’s not straightforward either, specially the update part. You have to know about jq update operator (|=) and how you can pass env variables using --arg.

Now imagine you decide to switch to YAML instead of JSON because either you started using a different tool that only accepts YAML or the same tool accepts both and you prefer it over JSON.

~ cat my_app.yml

app_name: awesome app
build_version: 1
tags:
- myTeam
- myCompany

You still want to accomplish the same thing, bump the build_version of your YAML. But your previous deployment bash script with your fancy jq query obviously doesn’t work anymore. Now you need to figure out how you’re going to parse and update that YAML, which like I mentioned in the beginning is not trivial (or at least I didn’t find a nice and easy way to do it).

Wouldn’t it be nice if you could somehow say: cat my_app.{yml, json} | get .build_version to read the value you are interested in and cat my_app.{yml, json} | set .build_version <new_value> to update it? That is, use exactly the same command regardless of where the input is coming from (JSON or YAML). Enter jyparser

jyparser

jyparser stands for JSON/YAML Parser (I know, not very original but I always sucked at names) and it was created specifically for the use case I described above. Getting a single value from a JSON or YAML, doing something with it (if needed) and then setting a new value for it on the original input. Of course reading/updating entire objects/arrays in JSON or entire hashes/lists in YAML is also supported.

At its hearth jyparser is a simple wrapper around jq and 2 python 1 liners to convert from JSON to YAML and vice versa. It will detect the input’s type and, in the case of YAML, convert to JSON before applying jq and then convert the result back to YAML. Note that since YAML is actually a superset of JSON this will only work for those YAML files that can be correctly converted to JSON.

You can see the code here and the docker image here.

Usage

Let’s look at some examples of how you would usually use the tool.

The image’s entry point accepts 2 operations: get and set. It can take its inputs from stdin or read from a file if this is passed as the first parameter.

Read

The get command takes an arbitrary jq filter. If the result is a simple value (number, string or boolean) then that value is returned. Otherwise, the resulting JSON or YAML is returned (depending on what the input was).

Given the following JSON file:

~ cat test.json

{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}

If you wanted to get the value of the id property you could use:

~ cat test.json | docker run -i --rm jlordiales/jyparser get .menu.id

"file"

The JSON is passed via stdin, which is useful if you get that from something like curl. If you have an actual file that you want to use as input then you can pass it directly as the first parameter to the script:

~ docker run -i --rm -v `pwd`:/jyparser:ro jlordiales/jyparser test.json get ".menu.id"

"file"

The example above mounts the current dir with the file into /jyparser (which is the default WORKDIR for the docker image) and then uses that file as input.

Exactly the same command works for YAML as well. Given the equivalent YAML file:

~ cat test.yml
menu:
  id: file
  value: File
  popup:
    menuitem:
    - onclick: CreateNewDoc()
      value: New
    - onclick: OpenDoc()
      value: Open
    - onclick: CloseDoc()
      value: Close

We can get the id property with:

~ cat test.yml | docker run -i --rm jlordiales/jyparser get .menu.id

"file"

If the result from running the jq filter is not a simple value, then the corresponding JSON or YAML is returned:

~ cat test.json | docker run -i --rm jlordiales/jyparser get ".menu.popup.menuitem[1]"

{
  "value": "Open",
  "onclick": "OpenDoc()"
}

~ cat test.yml | docker run -i --rm jlordiales/jyparser get ".menu.popup.menuitem[1]"

onclick: OpenDoc()
value: Open

The jq filter that is passed as parameter is sent as is to the tool, so you are not limited so simple filters. Anything that is valid for jq is valid for jyparser as well.

Update

Similarly to the get operation, there’s a set one. This operation takes 2 parameters: a jq filter to select a specific element of the input and a new value to update that element to. The result is the original input with the value updated.

~ cat test.json | docker run -i --rm jlordiales/jyparser set ".menu.id" \"new_id\"
{
  "menu": {
    "id": "new_id",
    "value": "File",
    "popup": {
      "menuitem": [
        {
          "value": "New",
          "onclick": "CreateNewDoc()"
        },
        {
          "value": "Open",
          "onclick": "OpenDoc()"
        },
        {
          "value": "Close",
          "onclick": "CloseDoc()"
        }
      ]
    }
  }
}

Important: given the way bash scripts handle quotes on parameters passed to them, if the new value you want to set for the property is a string you need to explicitly escape the quotes as in the example. Otherwise, jq will complain that the value is not valid (rightfully so). This is not needed for numbers or booleans. So the following works as expected:

~ cat test.json | docker run -i --rm jlordiales/jyparser set ".menu.id" 15
{
  "menu": {
    "id": 15,
    "value": "File",
    "popup": {
      "menuitem": [
        {
          "value": "New",
          "onclick": "CreateNewDoc()"
        },
        {
          "value": "Open",
          "onclick": "OpenDoc()"
        },
        {
          "value": "Close",
          "onclick": "CloseDoc()"
        }
      ]
    }
  }
}

This way of updating the JSON is arguably a lot easier to read and use than the jq version we saw at the beginning. It’s just set <key> <value> and, best of all, the same works for YAML:

~ cat test.yml | docker run -i --rm jlordiales/jyparser set ".menu.id" \"new_id\"

menu:
  id: new_id
  popup:
    menuitem:
    - onclick: CreateNewDoc()
      value: New
    - onclick: OpenDoc()
      value: Open
    - onclick: CloseDoc()
      value: Close
  value: File

As with the get operation, set can take the input both from stdin and a file if passed as first argument.

Conclusion

If you are doing regular parsing/updating of JSON and/or YAML and you don’t want to have hugely complex combinations of jq with sed and awk but instead have a simple interface to work with both types then give jyparser a try. It was created for a very specific use case but it might be able to adapt to yours as well.

jyparser was heavily inspired by y2j, so make sure to check it out as well.