JC Version 1.22.0 Released

I’m excited to announce the release of jc version 1.22.0 available on github and pypi. jc now supports over 170 standard and streaming parsers. Thank you to the Open Source community for making this possible!

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, see the project README.

To upgrade with pip:

$ pip3 install --upgrade jc

Try the jc web demo!

What’s New

• Add /proc file parsers for linux. Support for the following files:
  /proc/buddyinfo
/proc/consoles
/proc/cpuinfo
/proc/crypto
/proc/devices
/proc/diskstats
/proc/filesystems
/proc/interrupts
/proc/iomem
/proc/ioports
/proc/loadavg
/proc/locks
/proc/meminfo
/proc/modules
/proc/mtrr
/proc/pagetypeinfo
/proc/partitions
/proc/slabinfo
/proc/softirqs
/proc/stat
/proc/swaps
/proc/uptime
/proc/version
/proc/vmallocinfo
/proc/vmstat
/proc/zoneinfo
/proc/driver/rtc
/proc/net/arp
/proc/net/dev
/proc/net/dev_mcast
/proc/net/if_inet6
/proc/net/igmp
/proc/net/igmp6
/proc/net/ipv6_route
/proc/net/netlink
/proc/net/netstat
/proc/net/packet
/proc/net/protocols
/proc/net/route
/proc/net/unix
/proc/<pid>/fdinfo/<fd>
/proc/<pid>/io
/proc/<pid>/maps
/proc/<pid>/mountinfo
/proc/<pid>/numa_maps
/proc/<pid>/smaps
/proc/<pid>/stat
/proc/<pid>/statm
/proc/<pid>/status
• Magic syntax support for /proc files
• Enhance bash and zsh autocompletions for /proc files, including the Magic syntax
• Enhance metadata output to output metadata even when results are empty
• Enhance free parser to support -w option integer conversions
• Fix ini and kv parsers so they don’t change key-names to lower case
  NOTE: This can be a breaking change in your scripts
• Fix id command parser to allow usernames and group names with spaces

New Features

Metadata Enhancement

The --meta-out option was introduced in jc version 1.21.0. This feature has been enhanced to ensure metadata is always output – even if there are no results. for example, when running arp on macOS without the required -a flag:

% jc --meta-out --pretty arp
usage: arp [-n] [-i interface] hostname
       arp [-n] [-i interface] [-l] -a
       arp -d hostname [pub] [ifscope interface]
       arp -d [-i interface] -a
       arp -s hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface]
       arp -S hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface]
       arp -f filename
[
  {
    "_jc_meta": {
      "parser": "arp",
      "timestamp": 1664302590.903263,
      "magic_command": [
        "arp"
      ],
      "magic_command_exit": 1
    }
  }
]

New Parsers

/proc File Parsers

Many new /proc file parsers have been added and more will be released in future versions of jc. The easiest way to use these parsers is by invoking the --proc parser. The --proc parser (Documentation) will analyze the input data and select the correct /proc parser automatically:

% cat /proc/uptime | jc --proc -p
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

It is possible to manually select the /proc file parser by designating the specific parser. For example:

% cat /proc/uptime | jc --proc-uptime -p
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

Finally, you can also use the Magic syntax to convert /proc files:

% jc -p /proc/uptime
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

The bash and zsh autocompletion scripts have been updated to allow autocompletion of all of the new /proc file parsers and the /proc file magic syntax.

Individual /proc file parsers are hidden from jc --help. To find the low-level parser names, you can use jc -hh or jc --about:

% jc -hh
jc converts the output of many commands, file-types, and strings to JSON or YAML

Usage:

    Standard syntax:

        COMMAND | jc [OPTIONS] PARSER

        cat FILE | jc [OPTIONS] PARSER

        echo STRING | jc [OPTIONS] PARSER

    Magic syntax:

        jc [OPTIONS] COMMAND

        jc [OPTIONS] /proc/<path-to-procfile>

Parsers:
<snip>
    --ping              `ping` and `ping6` command parser
    --ping-s            `ping` and `ping6` command streaming parser
    --pip-list          `pip list` command parser
    --pip-show          `pip show` command parser
    --plist             PLIST file parser
    --postconf          `postconf -M` command parser
    --proc              `/proc/` file parser
    --proc-buddyinfo    `/proc/buddyinfo` file parser
    --proc-consoles     `/proc/consoles` file parser
    --proc-cpuinfo      `/proc/cpuinfo` file parser
    --proc-crypto       `/proc/crypto` file parser
    --proc-devices      `/proc/devices` file parser
    --proc-diskstats    `/proc/diskstats` file parser
    --proc-filesystems  `/proc/filesystems` file parser
    --proc-interrupts   `/proc/interrupts` file parser
    --proc-iomem        `/proc/iomem` file parser
<snip>

Options:
    -a,  --about        about jc
    -C,  --force-color  force color output even when using pipes (overrides -m)
    -d,  --debug        debug (double for verbose debug)
    -h,  --help         help (--help --parser_name for parser documentation)
    -m,  --monochrome   monochrome output
    -M,  --meta-out     add metadata to output including timestamp, etc.
    -p,  --pretty       pretty print output
    -q,  --quiet        suppress warnings (double to ignore streaming errors)
    -r,  --raw          raw output
    -u,  --unbuffer     unbuffer output
    -v,  --version      version info
    -y,  --yaml-out     YAML output
    -B,  --bash-comp    gen Bash completion: jc -B > /etc/bash_completion.d/jc
    -Z,  --zsh-comp     gen Zsh completion: jc -Z > "${fpath[1]}/_jc"

Examples:
    Standard Syntax:
        $ dig www.google.com | jc --pretty --dig
        $ cat /proc/meminfo | jc --pretty --proc

    Magic Syntax:
        $ jc --pretty dig www.google.com
        $ jc --pretty /proc/meminfo

    Parser Documentation:
        $ jc --help --dig

    Show Hidden Parsers:
        $ jc -hh

You can find the schema for each /proc parser with the same help syntax:

% jc --help --proc-uptime

v1.22.1 Updates

• Add udevadm command parser
• Add lspci command parser
• Add pci.ids file parser
• Fix proc-pid-stat parser for command names with spaces and newlines
• Enhance ip-address parser to add ip_split field
• Rename iso-datetime parser to datetime-iso. A deprecation warning will display until iso-datetime is removed in a future version.
• Optimize performance of calculated timestamps
• Add support for deprecating parsers
• Move jc-web demo site from heroku to render.com

In addition to three new parsers, this release includes several enhancements and fixes. The proc-pid-stat parser now supports command names that contain spaces and newlines. The ip-address string parser now includes a new ip-split field which splits each IPv4 octet or IPv6 segment into its own string within a list:

$ echo 192.168.2.10/24 | jc --ip-address -p
{
  "version": 4,
  "max_prefix_length": 32,
  "ip": "192.168.2.10",
  "ip_compressed": "192.168.2.10",
  "ip_exploded": "192.168.2.10",
  "ip_split": [
    "192",
    "168",
    "2",
    "10"
  ],
  ...
}

The iso-datetime parser has been renamed to datetime-iso. Don’t worry, the old name still works, but will print a deprecation warning to STDERR. This change introduces the new parser deprecation functionality. Deprecated parsers are identified with a deprecated field in the --about JSON output.

Some performance optimizations have been included for calculated timestamps. And, finally, we have moved the jc-web demo from Heroku to Render due to pricing changes.

udevadm command parser

A new udevadm linux command parser has been released. (Documentation)

$ udevadm info --query=all /dev/sda | jc --udevadm -p
{
  "P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
  "N": "sda",
  "L": 0,
  "S": [
    "disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
  ],
  "E": {
    "DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
    "DEVNAME": "/dev/sda",
    "DEVTYPE": "disk",
    "MAJOR": "8",
    "MINOR": "0",
    "SUBSYSTEM": "block",
    "USEC_INITIALIZED": "6100111",
    "SCSI_TPGS": "0",
    "SCSI_TYPE": "disk",
    "SCSI_VENDOR": "VMware,",
    "SCSI_VENDOR_ENC": "VMware,\\x20",
    "SCSI_MODEL": "VMware_Virtual_S",
    "SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
    "SCSI_REVISION": "1.0",
    "ID_SCSI": "1",
    "ID_VENDOR": "VMware_",
    "ID_VENDOR_ENC": "VMware\\x2c\\x20",
    "ID_MODEL": "VMware_Virtual_S",
    "ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
    "ID_REVISION": "1.0",
    "ID_TYPE": "disk",
    "MPATH_SBIN_PATH": "/sbin",
    "ID_BUS": "scsi",
    "ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
    "ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
    "ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
    "ID_PART_TABLE_TYPE": "gpt",
    "DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
    "TAGS": ":systemd:"
  }
}

lspci command parser

A new lspci linux command parser has been added. (Documentation)

$ lspci -nnmmv | jc --lspci -p
[
  {
    "slot": "ff:02:05.0",
    "domain": "ff",
    "domain_int": 255,
    "bus": "02",
    "bus_int": 2,
    "dev": "05",
    "dev_int": 5,
    "function": "0",
    "function_int": 0,
    "class": "SATA controller",
    "class_id": "0106",
    "class_id_int": 262,
    "vendor": "VMware",
    "vendor_id": "15ad",
    "vendor_id_int": 5549,
    "device": "SATA AHCI controller",
    "device_id": "07e0",
    "device_id_int": 2016,
    "svendor": "VMware",
    "svendor_id": "15ad",
    "svendor_id_int": 5549,
    "sdevice": "SATA AHCI controller",
    "sdevice_id": "07e0",
    "sdevice_id_int": 2016,
    "physlot": "37",
    "physlot_int": 55,
    "progif": "01",
    "progif_int": 1
  },
  ...
]

pci.ids file parser

A new file parser for the pci.ids database file has been added. (Documentation) This parser allows you to use tools like jq to query the database. The pci.ids database file can be found here: https://raw.githubusercontent.com/pciutils/pciids/master/pci.ids

$ cat pci.ids | jc --pci-ids | jq '.vendors._001c._0001._001c._0005.subsystem_name'
"2 Channel CAN Bus SJC1000 (Optically Isolated)"

$ cat pci.ids | jc --pci-ids | jq '.classes._0c._03._40'
"USB4 Host Interface"

v1.22.2 Updates

• Add sshd-conf parser for sshd configuration files and sshd -T output
• Add findmnt command parser
• Add git ls-remote command parser
• Add os-prober command parser
• Add SemVer string parser
• Enhance the ifconfig parser so it can output multiple IPv4 and IPv6 addresses
• Enhance the ifconfig parser so it can output additional fields common on BSD
• Enhance XML file parser with optional _ prefix for attributes instead of @ by using the --raw option. This can make it easier to filter the JSON output in some tools.
• Fix the XML file parser to output a normal Dictionary instead of OrderdDict. This cleans up YAML output. (No !!omap comments)
• Fix the standard and streaming CSV parsers for UTF-8 encoded CSV files with leading BOM bytes
• Fix exit code to be non-zero on keyboard interrupt
• Allow parser module objects to be used as arguments to jc.get_help() and jc.parser_info()
• Catch unexpected exceptions in the CLI
• Add error message on keyboard interrupt to STDERR
• Add python 3.11 tests to GitHub actions

sshd-conf file and command parser

A new sshd-conf file and sshd -T parser has been added. (Documentation)

$ sshd -T | jc --sshd-conf -p
{
  "acceptenv": [
    "LANG",
    "LC_*"
  ],
  "addressfamily": "any",
  "allowagentforwarding": "yes",
  "allowstreamlocalforwarding": "yes",
  "allowtcpforwarding": "yes",
  "authenticationmethods": "any",
  "authorizedkeyscommand": "none",
  "authorizedkeyscommanduser": "none",
  "authorizedkeysfile": [
    ".ssh/authorized_keys",
    ".ssh/authorized_keys2"
  ],
  "authorizedprincipalscommand": "none",
  ...
}

findmnt command parser

A new findmnt command parser has been added. (Documentation)

$ findmnt | jc --findmnt -p
[
  {
    "target": "/",
    "source": "/dev/mapper/centos-root",
    "fstype": "xfs",
    "options": [
      "rw",
      "relatime",
      "seclabel",
      "attr2",
      "inode64",
      "noquota"
    ]
  },
  {
    "target": "/sys/fs/cgroup",
    "source": "tmpfs",
    "fstype": "tmpfs",
    "options": [
      "ro",
      "nosuid",
      "nodev",
      "noexec",
      "seclabel"
    ],
    "kv_options": {
      "mode": "755"
    }
  },
  ...
]

git ls-remote command parser

A new git ls-remote command parser has been added. (Documentation)

$ git ls-remote | jc --git-ls-remote -p
{
  "HEAD": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
  "refs/heads/dev": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389",
  "refs/heads/master": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
  "refs/pull/1/head": "e416c77bed1267254da972b0f95b7ff1d43fccef",
  ...
}

$ git ls-remote | jc --git-ls-remote -p -r
[
  {
    "reference": "HEAD",
    "commit": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338"
  },
  {
    "reference": "refs/heads/dev",
    "commit": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389"
  },
  ...
]

os-prober command parser

A new os-prober command parser has been added. (Documentation)

$ os-prober | jc --os-prober -p
{
  "partition": "/dev/sda1",
  "name": "Windows 10",
  "short_name": "Windows",
  "type": "chain"
}

Semantic Version string parser

A new Semantic Version string parser has been added. (Documentation)

$ echo 1.2.3-rc.1+44837 | jc --semver -p
{
  "major": 1,
  "minor": 2,
  "patch": 3,
  "prerelease": "rc.1",
  "build": "44837"
}

ifconfig command parser enhancements

The ifconfig command parser has been enhanced to support multiple IPv4 and IPv6 addresses. Also many more BSD/macOS fields are parsed.

$ ifconfig | jc --ifconfig -p
[
  { 
    "name": "en0",
    "flags": 8863,
    "state": [
      "UP",
      "BROADCAST",
      "SMART",
      "RUNNING",
      "SIMPLEX",
      "MULTICAST"
    ],
    "mtu": 1500,
    "type": null,
    "mac_addr": "f0:18:98:03:d9:30",
    "ipv4_addr": "192.168.1.72",
    "ipv4_mask": "255.255.255.0",
    "ipv4_bcast": "192.168.1.255",
    "ipv6_addr": "fe80::8b7:1281:7499:b504",
    "ipv6_mask": 64,
    "ipv6_scope": "0x8",
    "ipv6_type": null,
    "metric": null,
    "rx_packets": null,
    "rx_errors": null,
    "rx_dropped": null,
    "rx_overruns": null,
    "rx_frame": null,
    "tx_packets": null,
    "tx_errors": null,
    "tx_dropped": null,
    "tx_overruns": null,
    "tx_carrier": null,
    "tx_collisions": null,
    "rx_bytes": null,
    "tx_bytes": null,
    "nd6_options": 201,
    "nd6_flags": [
      "PERFORMNUD",
      "DAD"
    ],
    "status": "active",
    "ipv4": [
      {
        "address": "192.168.1.72",
        "mask": "255.255.255.0",
        "broadcast": "192.168.1.255"
      }
    ],
    "ipv6": [
      {
        "address": "fe80::8b7:1281:7499:b504",
        "mask": 64,
        "scope": "0x8"
      }
    ]
  },
  ...
]

v1.22.3 Updates

• Add Common Log Format and Combined Log Format file parser (standard and streaming)
• Add PostgreSQL password file parser
• Add openvpn-status.log file parser
• Add cbt command parser (Google Big Table)
• Enhance ifconfig parser with more information on BSD
• Fix ifconfig parser to capture some IPv6 addresses missed on BSD
• Fix git-log and git-log-s parsers for failure on empty author name
• Update os-prober parser with split EFI partition fields

v1.22.4 Updates

• Add iwconfig command parser
• Add NeXTSTEP format support to the PLIST file parser
• Fix proc parser magic signature detection for /proc/pid/stat hacks
• Fix x509-cert parser for string serial numbers
• Add category tags to parser metadata: generic, standard, file, string, binary, command
• Add “list parsers by category” view to help
• Fix python 3.6-related issues
• Add python 3.6 to automated tests

v1.22.5 Updates

• Add TOML file parser
• Add INI with duplicate key support file parser
• Add AIX support for the arp command parser
• Add AIX support for the mount command parser
• Fix lsusb command parser when extra hub port status information is output
• Fix INI file parser to include top-level values with no section header
• Fix INI file parser to not specially handle the [DEFAULT] section
• Fix INI file and Key/Value parsers to only remove one quotation mark from the
  beginning and end of values.

Happy parsing!