Parsing Command Output in Saltstack with JC

In my last blog post I demonstrated how we can easily parse remote command output in Ansible. Since then it was requested that I demonstrate something similar using Saltstack.

Saltstack (or Salt, as it is known) is a little different than Ansible in that it primarily uses a pub/sub architecture vs. SSH and requires an agent, or a Minion, to be installed on the remote hosts you are managing.

It turns out it is fairly straightforward to add jc functionality to Saltstack via a custom Output Module and/or Serializer Module. We’ll go over both methods, plus a bonus method in this post.

For more information on the motivations for creating jc, see my blog post.

Output Module

With a Salt Output Module you can restructure the output of the command results that are written to STDOUT on the Master. The default output is typically YAML, but you can change it to JSON or other formats with builtin output modules.

Here is the default YAML output:

# salt '*' cmd.run 'uptime'
minion1:
     16:31:16 up 2 days,  3:04,  1 user,  load average: 0.03, 0.03, 0.00
minion2:
     16:31:16 up 2 days,  3:04,  1 user,  load average: 0.00, 0.00, 0.00

And here is the output using the builtin JSON ouputter:

# salt '*' cmd.run 'uptime' --out=json
{
    "minion2": " 16:33:02 up 2 days,  3:06,  1 user,  load average: 0.00, 0.00, 0.00"
}
{
    "minion1": " 16:33:02 up 2 days,  3:06,  1 user,  load average: 0.00, 0.02, 0.00"
}

But we can do better with jc by turning the uptime output into a JSON object:

# JC_PARSER=uptime salt '*' cmd.run 'uptime' --out=jc --out-indent=2
{
  "minion1": {
    "time": "16:36:04",
    "uptime": "2 days, 3:09",
    "users": 1,
    "load_1m": 0.07,
    "load_5m": 0.02,
    "load_15m": 0.0
  }
}
{
  "minion2": {
    "time": "16:36:04",
    "uptime": "2 days, 3:09",
    "users": 1,
    "load_1m": 0.0,
    "load_5m": 0.0,
    "load_15m": 0.0
  }
}

Now we can pipe this output to jq, jello, or any other JSON filter to more easily consume this data.

We’ll go over the Output Module installation and usage later in this post.

Serializer Module

With a Salt Serializer Module you can restructure the output of the command results during runtime on each Minion so they can be used as objects/variables within a Salt state. For example, If I only cared about the number of users currently logged into each minion and wanted to set that number as a variable for use elsewhere, we could do that with a jc Serializer Module.

Here is a simple, contrived example Salt state file to show how it works:

{% set uptime_out = salt.cmd.shell('uptime') %}
{% set uptime_jc = salt.slsutil.deserialize('jc', uptime_out, parser='uptime') %}

run_uptime:
  cmd.run:
    - name: >
        echo 'The number of users logged in is {{ uptime_jc.users }}'

And here is the output after applying this state file:

# salt '*' state.apply uptime-users
minion1:
----------
          ID: run_uptime
    Function: cmd.run
        Name: echo 'The number of users logged in is 1'

      Result: True
     Comment: Command "echo 'The number of users logged in is 1'
              " run
     Started: 17:01:43.992058
    Duration: 6.107 ms
     Changes:   
              ----------
              pid:
                  23208
              retcode:
                  0
              stderr:
              stdout:
                  The number of users logged in is 1

Summary for minion1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   6.107 ms
minion2:
----------
          ID: run_uptime
    Function: cmd.run
        Name: echo 'The number of users logged in is 2'

      Result: True
     Comment: Command "echo 'The number of users logged in is 2'
              " run
     Started: 17:01:44.005482
    Duration: 6.55 ms
     Changes:   
              ----------
              pid:
                  23371
              retcode:
                  0
              stderr:
              stdout:
                  The number of users logged in is 2

Summary for minion2
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   6.550 ms

Since jc deserialized the command output into an object, we can simply reference the object attributes in our Salt states. We’ll go over installation and usage of the jc Serializer Module later in this post.

Installation and Usage

To use the jc Output Module, you will need to install jc on the Master. To use the jc Serializer Module, you will need to install jc on the Minions. Depending on your use case you may decide to install one or the other or both modules.

Installing jc

You can install jc on the Master and Minions with the following command. Of course, this can also be automated via Salt!

$ pip3 install jc

Installing the Output Module

To install the Output Module on the Master, you need to place the Python module in a directory where the Master is configured to look for it.

First, edit the /etc/salt/master configuration file to configure a custom Module directory. In this example we will use /srv/modules by adding this line to the configuration file:

module_dirs: ["/srv/modules"]

Next we need to create the /srv/modules/output directory, if it doesn’t already exist:

# mkdir -p /srv/modules/output

Next, copy the python module into the directory. I have uploaded the code to Github as a Gist:

# curl https://gist.githubusercontent.com/kellyjonbrazil/24e10f0c3e438ea22fc1e2bfaee22efc/raw/263e4eaf8e51f974b34d44e0483540b163667bdf/jc.py -o /srv/modules/output/jc.py

Finally, restart the Salt Master:

# systemctl restart salt-master

Using the Output Module

To use the jc Output Module, you need to call it with the --out=jc option of the salt command.

Additionally, you need to tell the jc Output Module which parser to use. To do this, you can set the JC_PARSER environment variable inline with the command:

# JC_PARSER=date salt '*' cmd.run 'date' --out=jc
{"minion2": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}}
{"minion1": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}}

For a list of jc parsers, see the parser documentation.

Additionally, you can add the --out-indent option to pretty-print the output:

# JC_PARSER=date salt '*' cmd.run 'date' --out=jc --out-indent=2
{
  "minion2": {
    "year": 2020,
    "month_num": 9,
    "day": 15,
    "hour": 18,
    "minute": 29,
    "second": 8,
    "month": "Sep",
    "weekday": "Tue",
    "weekday_num": 3,
    "timezone": "UTC"
  }
}
{
  "minion1": {
    "year": 2020,
    "month_num": 9,
    "day": 15,
    "hour": 18,
    "minute": 29,
    "second": 8,
    "month": "Sep",
    "weekday": "Tue",
    "weekday_num": 3,
    "timezone": "UTC"
  }
}

Installing the Serializer Module

To install the Serializer Module on the Minions, you can copy the Python module to the _serializers folder within your Salt fileserver directory on the Master (typically /srv/salt) and sync to the Minions.

First, create the /srv/salt/_serializers directory if it doesn’t already exist:

# mkdir -p /srv/salt/_serializers

Next, copy the Python module into the _serializers directory on the Master. I have uploaded the code to Github as a Gist:

# curl https://gist.githubusercontent.com/kellyjonbrazil/7d67cfa003735bf80ef43fe5652950dd/raw/1541a7d327aed0366ccfea91bd0533032111d11c/jc.py -o /srv/salt/_serializers/jc.py

Finally, sync the jc Serializer Module to the Minions:

# salt '*' saltutil.sync_all

Using the Serializer Module

To use the jc Serializer Module, invoke it with the salt.slsutil.deserialize() function within a Salt state file. The function requires three arguments to deserialize with jc:

• Argument 1: 'jc'
  • This should always be the literal string 'jc' to call the jc Serializer Module
• Argument 2: String data to be parsed
  • This is the STDOUT string output of the command you want to deserialize
• Argument 3: parser='<parser>'
  • <parser> is the jc parser you want to use to parse the command output. For example, to use the ifconfig parser, Argument 3 would look like this: parser='ifconfig'. For a list of jc parsers, see the parser documentation.

For example, via Jinja2 template:

{% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %}

Then you can reference any attribute of the date object (Python dictionary) in any other part of the Salt state file. Here is a full example:

{% set date_stdout = salt.cmd.shell('date') %}
{% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %}

run_date:
  cmd.run:
    - name: >
        echo 'The timezone is {{ date.timezone }}'

One More Thing

It is also possible to deserialize command output into objects using jc without using the jc Serializer Module. If jc is installed on the Minion, then you can pipe the command output to jc as you would normally do on the command line, then use the buit-in JSON Serializer Module to deserialize the jc JSON output into Python objects:

{% set date = salt.slsutil.deserialize('json', salt.cmd.shell('date | jc --date')) %}

run_date:
  cmd.run:
    - name: >
        echo 'The timezone is {{ date.timezone }}'

Happy parsing!