Tutorial: Rapid Script Development with Bash, JC, and JQ

I thought it would be fun to show the power of using JSON for rapid development of Bash scripts. When you don’t need to parse unstructured command output and common string types manually you are freed to focus on the code logic which can really reduce development and troubleshooting cycles.

To that end, I created a toy CLI application that scans the local subnet to see which hosts respond to ICMP requests. This application took less than a half-hour to write (plus some fine-tuning) and I think it demonstrates some cool concepts utilizing the JSON command output of several commands, including ifconfig, ping, arp, date, and wc. We use jc to convert the command output to JSON and jq to grab the values we want from that output.

Yes, this is not as efficient as pinging the broadcast address and checking the local ARP table, but this is arguably more fun!

Here is the Bash script of around 80 lines. Because there is no low-level command output parsing via grep, cut, sed, awk, etc. it is pretty simple to understand, but we’ll go through some of the interesting parts in more detail.

Here is some sample output when using the script:

% ./scansubnet.sh    
Please enter the interface name as command argument.

Options:
lo0
en0

% ./scansubnet.sh wrong-interface                     
ifconfig: interface wrong-interface does not exist

% ./scansubnet.sh lo0
Subnet is too large (16777214 IPs). Exiting.

% ./scansubnet.sh en0
My IP: 192.168.1.221/24
Sending ICMP requests to 254 IPs: 192.168.1.1 - 192.168.1.254
Start Time: 2022-08-29T07:05:40
   13.623 ms   192.168.1.249   f0:ef:86:f6:21:84   camera1.local
  350.634 ms   192.168.1.72    f0:18:98:3:f8:39    laptop1.local
   10.645 ms   192.168.1.243   fc:ae:34:a1:35:82   
  561.997 ms   192.168.1.188   18:3e:ef:d3:3f:82   laptop2.local
   19.775 ms   192.168.1.254   fc:ae:34:a1:3a:80   router.local
                          <snip>
   27.917 ms   192.168.1.197   cc:a7:c1:5e:c3:f1   camera2.local
   28.582 ms   192.168.1.235   56:c8:36:64:2a:8d   camera3.local
   38.199 ms   192.168.1.246   d8:30:62:2e:a6:cf   extender.local
   44.617 ms   192.168.1.242   50:14:79:1e:42:3e   vacuum.local
    5.350 ms   192.168.1.88    c8:d0:83:cd:f4:2d   tv.local
    0.087 ms   192.168.1.221   a4:83:e7:2d:62:4e   laptop3.local
Scanned 192.168.1.0/24 subnet in 27 seconds.
30 alive hosts found.
End Time: 2022-08-29T07:06:07

Note: For best results, use jc version 1.21.1 or higher in this script.

Notice there are some basic checks and features:

  • If you don’t specify an interface it will suggest any interfaces on the system with a configured IPv4 address
  • If you select an interface attached to too large of a subnet (say a 127.0.0.1/8 address of a loopback) it will exit
  • It will show the subnet information, including the number of hosts it will scan and the range of IP addresses to be scanned
  • It will log the start and end time and calculate how long it took to complete the scan
  • It will run the pings in the background for parallel processing
  • It will report the round-trip time, IP, MAC address, and name (if known) of the hosts that respond

Let’s take a closer look!

Displaying valid interface options

We could make the user figure out the valid interface names that can be used as the only command argument. Instead, we can use the ifconfig output passed through jc and a simple jq query to print valid options (that is, interfaces with an IPv4 address configured):

if [[ $1 == "" ]]; then
    echo "Please enter the interface name as command argument."
    echo
    echo "Options:"
    # Only show interfaces with an assigned IP address
    jc ifconfig | jq -r '.[] | select(.ipv4_addr != null) | .name'
    exit 1
fi

Once jc converts the ifconfig output to JSON we pipe it to jq so it will filter only items with an ipv4_addr field that is not set to null. Pretty straightforward and easy to read.

Note: We could have gotten the same result by using the ip command with the JSON output option instead of parsing the ifconfig output to JSON via jc.

Grab the selected interface IP and subnet mask

Once the user selects a valid interface, let’s grab the IP and Subnet:

interfaceInfo=$(jc ifconfig "$1") || exit 1
ip=$(jq -r '.[0].ipv4_addr' <<<"$interfaceInfo")
mask=$(jq -r '.[0].ipv4_mask' <<<"$interfaceInfo")

Here you can see we parse the ifconfig output to JSON with jc in the first line. If the interface name from the user is invalid, ifconfig will give us a non-zero exit code and quit the program with a helpful message. Then we do two variable assignments to pull the IP address and subnet mask via jq queries.

Grab detailed subnet information for the IP/Mask

Next we take the Interface IP and Mask values and use the IP Address string parser in jc to gather more detailed subnet information we will use later in the script.

ipInfo=$(jc --ip-address <<<"$ip/$mask")
network=$(jq -r '.network' <<<"$ipInfo")
numHosts=$(jq -r '.hosts' <<<"$ipInfo")
cidrMask=$(jq -r '.cidr_netmask' <<<"$ipInfo")
firstHostIp=$(jq -r '.first_host' <<<"$ipInfo")
lastHostIp=$(jq -r '.last_host' <<<"$ipInfo")
firstHost=$(jq -r '.int.first_host' <<<"$ipInfo")
lastHost=$(jq -r '.int.last_host' <<<"$ipInfo")

The IP Address parser in jc is nice because it acts like a subnet calculator and gives us a lot of data, including the subnet, number of hosts in the subnet, first host, and last host in different formats (decimal, hex, binary, and standard format). This will help us build a simple for loop that will do most of the work. All we need to do is pick the fields we want with jq. Here is an example of all of the information available with the jc IP Address string parser (IPv6 is also supported):

% echo 192.168.1.10/255.255.255.0 | jc --ip-address -p
{
  "version": 4,
  "max_prefix_length": 32,
  "ip": "192.168.1.10",
  "ip_compressed": "192.168.1.10",
  "ip_exploded": "192.168.1.10",
  "scope_id": null,
  "ipv4_mapped": null,
  "six_to_four": null,
  "teredo_client": null,
  "teredo_server": null,
  "dns_ptr": "10.1.168.192.in-addr.arpa",
  "network": "192.168.1.0",
  "broadcast": "192.168.1.255",
  "hostmask": "0.0.0.255",
  "netmask": "255.255.255.0",
  "cidr_netmask": 24,
  "hosts": 254,
  "first_host": "192.168.1.1",
  "last_host": "192.168.1.254",
  "is_multicast": false,
  "is_private": true,
  "is_global": false,
  "is_link_local": false,
  "is_loopback": false,
  "is_reserved": false,
  "is_unspecified": false,
  "int": {
    "ip": 3232235786,
    "network": 3232235776,
    "broadcast": 3232236031,
    "first_host": 3232235777,
    "last_host": 3232236030
  },
  "hex": {
    "ip": "c0:a8:01:0a",
    "network": "c0:a8:01:00",
    "broadcast": "c0:a8:01:ff",
    "hostmask": "00:00:00:ff",
    "netmask": "ff:ff:ff:00",
    "first_host": "c0:a8:01:01",
    "last_host": "c0:a8:01:fe"
  },
  "bin": {
    "ip": "11000000101010000000000100001010",
    "network": "11000000101010000000000100000000",
    "broadcast": "11000000101010000000000111111111",
    "hostmask": "00000000000000000000000011111111",
    "netmask": "11111111111111111111111100000000",
    "first_host": "11000000101010000000000100000001",
    "last_host": "11000000101010000000000111111110"
  }
}

In addition to accepting a CIDR subnet mask, the jc IP Address string parser also accepts a dotted-quad subnet mask (that’s how ifconfig gives it to us) and provides us the CIDR notation for it in the cidr_netmask field. The IP Address string parser is fairly liberal in the IP formats it will accept.

We’ll see how having the first_host and last_host values in decimal makes for easy looping later.

Sanity check the subnet size

We can use the hosts value from the jc IP Address string parser to see if the subnet is a suitable size to scan. If the subnet supports any more than 1022 hosts (/22) then we don’t want to bother spinning up that many ping processes in the background for the scan. The following code does that sanity check for us:

if [[ $numHosts -gt 1022 ]]; then
    echo "Subnet is too large ($numHosts IPs). Exiting."
    exit 1
fi

Grab the start time in ISO and Unix Epoch format

Next we want to grab the start time. The date command parser in jc gives us the current time in ISO and Epoch formats that we can easily pull with jq. This allows us to display the time in a nice, standard human readable format and also have the date-time information in an easy-to-use format for calculating the duration later.

startTime=$(jc date)
startTimeIso=$(jq -r '.iso' <<<"$startTime")
startTimeEpoch=$(jq -r '.epoch' <<<"$startTime")

Here are all of the fields available when running the date command through jc:

% jc -p date
{
  "year": 2022,
  "month": "Aug",
  "month_num": 8,
  "day": 28,
  "weekday": "Sun",
  "weekday_num": 7,
  "hour": 1,
  "hour_24": 13,
  "minute": 39,
  "second": 20,
  "period": "PM",
  "timezone": "PDT",
  "utc_offset": null,
  "day_of_year": 240,
  "week_of_year": 34,
  "iso": "2022-08-28T13:39:20",
  "epoch": 1661719160,
  "epoch_utc": null,
  "timezone_aware": false
}

Show the user what is going to happen

Next we use a series of simple echo commands to provide the subnet and time information back to the user before the scan:

echo "My IP: $ip/$cidrMask"
echo "Sending ICMP requests to $numHosts IPs: $firstHostIp - $lastHostIp"
echo "Start Time: $startTimeIso"

The main loop

Now comes the fun part – here is the main loop where we ping every host in the subnet and record the round-trip time, IP, MAC address, and name of each host that responds:

for (( ipDecimal=$firstHost; ipDecimal<=$lastHost; ipDecimal++ )); do
    # Do each ping in the background for parallel processing
    {
        # grab the packets received and rtt values from the ping output
        thisIp=$(jc --ip-address <<<"$ipDecimal" | jq -r '.ip')
        pingResult=$(ping -c1 "$thisIp" | jc --ping)
        packetsReceived=$(jq -r '.packets_received' <<<"$pingResult")
        rtTime=$(jq -r '.round_trip_ms_max' <<<"$pingResult")

        if [[ $packetsReceived -gt 0 ]]; then
            # Grab the MAC address and name for each alive host from the arp command
            thisIpArpInfo=$(arp -a | jc --arp | jq --arg myip "$thisIp" '.[] | select(.address == $myip)')
            thisIpMac=$(jq -r '.hwaddress // empty' <<<"$thisIpArpInfo")
            thisIpName=$(jq -r '.name // empty' <<<"$thisIpArpInfo")

            printf "%9.3f ms   %-16s%-20s%s\n" "$rtTime" "$thisIp" "$thisIpMac" "$thisIpName" | tee -a "$tempFile"
        fi
    } &
done
wait

Let’s break this down a little bit:

for (( ipDecimal=$firstHost; ipDecimal<=$lastHost; ipDecimal++ )); do

We use a C-style for loop which allows us to use those decimal versions of the first and last host IP addresses. I told you those decimal values would come in handy!

        thisIp=$(jc --ip-address <<<"$ipDecimal" | jq -r '.ip')
        pingResult=$(ping -c1 "$thisIp" | jc --ping)
        packetsReceived=$(jq -r '.packets_received' <<<"$pingResult")
        rtTime=$(jq -r '.round_trip_ms_max' <<<"$pingResult")

The decimal IP format is nice to loop over, but unfortunately the ping and arp commands do not seem to accept IP addresses in decimal format (at least not on all platforms). Not to worry – we simply send the decimal IP address to the jc IP Address string parser and it will tell us what the IP address is in standard dotted-quad notation.

Then we give ping that IP address and parse its output with jc. We only care about the packets_received and round_trip_ms_max fields, so we assign them to Bash variables.

Next, let’s take a look at the if block:

        if [[ $packetsReceived -gt 0 ]]; then
            # Grab the MAC address and name for each alive host from the arp command
            thisIpArpInfo=$(arp -a | jc --arp | jq --arg myip "$thisIp" '.[] | select(.address == $myip)')
            thisIpMac=$(jq -r '.hwaddress // empty' <<<"$thisIpArpInfo")
            thisIpName=$(jq -r '.name // empty' <<<"$thisIpArpInfo")

            printf "%9.3f ms   %-16s%-20s%s\n" "$rtTime" "$thisIp" "$thisIpMac" "$thisIpName" | tee -a "$tempFile"
        fi

There’s a bit going on in this if block:

  • We only run the below commands if there was an ICMP reply from the ping output
  • Since we got an ICMP reply, we check the ARP table via the arp -a command and filter for the current IP address’ MAC address and name. Having jc parse the arp-a output into JSON allows us to use a simple jq query to accomplish this.
  • Notice the use of the --arg option in jq that allows us to use the $thisIp value in the query.
  • Notice the jq -r '.name // empty' section. This tells jq to output an empty string if it sees a null value for name.
  • We use the printf command with string format specifications to print our output in nice, even columns.
  • The tee command copies what is printed to the screen and appends it to a temporary file that we will use later.
    } &
done
wait

The & at the end of the Bash command grouping tells Bash to run all of the commands enclosed in {} brackets in the background, so we get parallel processing. The wait command tells bash to pause until all of the background processes complete.

Grab the end time

After all of the background ping and arp processes return, we can grab the end time by parsing the date command with jc and returning the iso and epoch values:

endTime=$(jc date)
endTimeIso=$(jq -r '.iso' <<<"$endTime")
endTimeEpoch=$(jq -r '.epoch' <<<"$endTime")
totalTime=$(($endTimeEpoch-$startTimeEpoch))

Then we subtract the epoch values to get the total run time.

Grab the number of alive hosts

We can run the temporary file through wc to get the number of lines. The wc parser in jc makes it easy to pull the number of lines with a quick jq query. Then we delete the temporary file.

totalAlive=$(jc wc "$tempFile" | jq '.[0].lines')
rm "$tempFile"

Print the summary message

Finally, we print a summary message with the total run-time, subnet information, number of alive hosts, and the human-readable end time:

echo "Scanned $network/$cidrMask subnet in $totalTime seconds."
echo "$totalAlive alive hosts found."
echo "End Time: $endTimeIso"

Conclusion

There you go – that was a pretty fun exercise demonstrating how you can rapidly develop a prototype in Bash using the output of existing commands on the system without needing to manually parse them. By using jq to query the JSON output from the commands and jc the script becomes very easy to understand. It’s nearly self documenting!

Let me know if you have built any cool scripts or programs with jc and jq!

Published by kellyjonbrazil

I'm a cybersecurity and cloud computing nerd.

Leave a Reply

%d bloggers like this: