In my last blog post I demonstrated how we can easily parse remote command output in Ansible. Since then it was requested that I demonstrate something similar using Saltstack.
Saltstack (or Salt, as it is known) is a little different than Ansible in that it primarily uses a pub/sub architecture vs. SSH and requires an agent, or a Minion, to be installed on the remote hosts you are managing.
It turns out it is fairly straightforward to add jc
functionality to Saltstack via a custom Output Module and/or Serializer Module. We’ll go over both methods, plus a bonus method in this post.
For more information on the motivations for creating
jc
, see my blog post.
Output Module
With a Salt Output Module you can restructure the output of the command results that are written to STDOUT
on the Master. The default output is typically YAML, but you can change it to JSON or other formats with builtin output modules.
Here is the default YAML output:
# salt '*' cmd.run 'uptime' minion1: 16:31:16 up 2 days, 3:04, 1 user, load average: 0.03, 0.03, 0.00 minion2: 16:31:16 up 2 days, 3:04, 1 user, load average: 0.00, 0.00, 0.00
And here is the output using the builtin JSON ouputter:
# salt '*' cmd.run 'uptime' --out=json { "minion2": " 16:33:02 up 2 days, 3:06, 1 user, load average: 0.00, 0.00, 0.00" } { "minion1": " 16:33:02 up 2 days, 3:06, 1 user, load average: 0.00, 0.02, 0.00" }
But we can do better with jc
by turning the uptime
output into a JSON object:
# JC_PARSER=uptime salt '*' cmd.run 'uptime' --out=jc --out-indent=2 { "minion1": { "time": "16:36:04", "uptime": "2 days, 3:09", "users": 1, "load_1m": 0.07, "load_5m": 0.02, "load_15m": 0.0 } } { "minion2": { "time": "16:36:04", "uptime": "2 days, 3:09", "users": 1, "load_1m": 0.0, "load_5m": 0.0, "load_15m": 0.0 } }
Now we can pipe this output to jq
, jello
, or any other JSON filter to more easily consume this data.
We’ll go over the Output Module installation and usage later in this post.
Serializer Module
With a Salt Serializer Module you can restructure the output of the command results during runtime on each Minion so they can be used as objects/variables within a Salt state. For example, If I only cared about the number of users currently logged into each minion and wanted to set that number as a variable for use elsewhere, we could do that with a jc
Serializer Module.
Here is a simple, contrived example Salt state file to show how it works:
{% set uptime_out = salt.cmd.shell('uptime') %} {% set uptime_jc = salt.slsutil.deserialize('jc', uptime_out, parser='uptime') %} run_uptime: cmd.run: - name: > echo 'The number of users logged in is {{ uptime_jc.users }}'
And here is the output after applying this state file:
# salt '*' state.apply uptime-users minion1: ---------- ID: run_uptime Function: cmd.run Name: echo 'The number of users logged in is 1' Result: True Comment: Command "echo 'The number of users logged in is 1' " run Started: 17:01:43.992058 Duration: 6.107 ms Changes: ---------- pid: 23208 retcode: 0 stderr: stdout: The number of users logged in is 1 Summary for minion1 ------------ Succeeded: 1 (changed=1) Failed: 0 ------------ Total states run: 1 Total run time: 6.107 ms minion2: ---------- ID: run_uptime Function: cmd.run Name: echo 'The number of users logged in is 2' Result: True Comment: Command "echo 'The number of users logged in is 2' " run Started: 17:01:44.005482 Duration: 6.55 ms Changes: ---------- pid: 23371 retcode: 0 stderr: stdout: The number of users logged in is 2 Summary for minion2 ------------ Succeeded: 1 (changed=1) Failed: 0 ------------ Total states run: 1 Total run time: 6.550 ms
Since jc
deserialized the command output into an object, we can simply reference the object attributes in our Salt states. We’ll go over installation and usage of the jc
Serializer Module later in this post.
Installation and Usage
To use the jc
Output Module, you will need to install jc
on the Master. To use the jc
Serializer Module, you will need to install jc
on the Minions. Depending on your use case you may decide to install one or the other or both modules.
Installing jc
You can install jc
on the Master and Minions with the following command. Of course, this can also be automated via Salt!
$ pip3 install jc
Installing the Output Module
To install the Output Module on the Master, you need to place the Python module in a directory where the Master is configured to look for it.
First, edit the /etc/salt/master
configuration file to configure a custom Module directory. In this example we will use /srv/modules
by adding this line to the configuration file:
module_dirs: ["/srv/modules"]
Next we need to create the /srv/modules/output
directory, if it doesn’t already exist:
# mkdir -p /srv/modules/output
Next, copy the python module into the directory. I have uploaded the code to Github as a Gist:
# curl https://gist.githubusercontent.com/kellyjonbrazil/24e10f0c3e438ea22fc1e2bfaee22efc/raw/263e4eaf8e51f974b34d44e0483540b163667bdf/jc.py -o /srv/modules/output/jc.py
Finally, restart the Salt Master:
# systemctl restart salt-master
Using the Output Module
To use the jc
Output Module, you need to call it with the --out=jc
option of the salt
command.
Additionally, you need to tell the jc
Output Module which parser to use. To do this, you can set the JC_PARSER
environment variable inline with the command:
# JC_PARSER=date salt '*' cmd.run 'date' --out=jc {"minion2": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}} {"minion1": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}}
For a list of
jc
parsers, see the parser documentation.
Additionally, you can add the --out-indent
option to pretty-print the output:
# JC_PARSER=date salt '*' cmd.run 'date' --out=jc --out-indent=2 { "minion2": { "year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 29, "second": 8, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC" } } { "minion1": { "year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 29, "second": 8, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC" } }
Installing the Serializer Module
To install the Serializer Module on the Minions, you can copy the Python module to the _serializers
folder within your Salt fileserver directory on the Master (typically /srv/salt
) and sync to the Minions.
First, create the /srv/salt/_serializers
directory if it doesn’t already exist:
# mkdir -p /srv/salt/_serializers
Next, copy the Python module into the _serializers
directory on the Master. I have uploaded the code to Github as a Gist:
# curl https://gist.githubusercontent.com/kellyjonbrazil/7d67cfa003735bf80ef43fe5652950dd/raw/1541a7d327aed0366ccfea91bd0533032111d11c/jc.py -o /srv/salt/_serializers/jc.py
Finally, sync the jc
Serializer Module to the Minions:
# salt '*' saltutil.sync_all
Using the Serializer Module
To use the jc
Serializer Module, invoke it with the salt.slsutil.deserialize()
function within a Salt state file. The function requires three arguments to deserialize with jc
:
- Argument 1:
'jc'
- This should always be the literal string
'jc'
to call thejc
Serializer Module
- This should always be the literal string
- Argument 2: String data to be parsed
- This is the
STDOUT
string output of the command you want to deserialize
- This is the
- Argument 3:
parser='<parser>'
<parser>
is thejc
parser you want to use to parse the command output. For example, to use theifconfig
parser, Argument 3 would look like this:parser='ifconfig'
. For a list ofjc
parsers, see the parser documentation.
For example, via Jinja2 template:
{% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %}
Then you can reference any attribute of the date
object (Python dictionary) in any other part of the Salt state file. Here is a full example:
{% set date_stdout = salt.cmd.shell('date') %} {% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %} run_date: cmd.run: - name: > echo 'The timezone is {{ date.timezone }}'
One More Thing
It is also possible to deserialize command output into objects using jc
without using the jc
Serializer Module. If jc
is installed on the Minion, then you can pipe the command output to jc
as you would normally do on the command line, then use the buit-in JSON Serializer Module to deserialize the jc
JSON output into Python objects:
{% set date = salt.slsutil.deserialize('json', salt.cmd.shell('date | jc --date')) %} run_date: cmd.run: - name: > echo 'The timezone is {{ date.timezone }}'
Happy parsing!
Excellent write-up! Would you have any interest in submitting the output and serializer modules upstream?
Hi Christian,
Absolutely – that was my original intent and I have an issue open at the Saltstack Github repo: https://github.com/saltstack/salt/issues/58355