I’m excited to announce the release of jc
version 1.22.0 available on github and pypi. jc
now supports over 170 standard and streaming parsers. Thank you to the Open Source community for making this possible!
jc
can be installed via pip
or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc
, see the project README.
To upgrade with pip
:
$ pip3 install --upgrade jc
Try the
jc
web demo!
What’s New
- Add
/proc
file parsers for linux. Support for the following files:/proc/buddyinfo
/proc/consoles
/proc/cpuinfo
/proc/crypto
/proc/devices
/proc/diskstats
/proc/filesystems
/proc/interrupts
/proc/iomem
/proc/ioports
/proc/loadavg
/proc/locks
/proc/meminfo
/proc/modules
/proc/mtrr
/proc/pagetypeinfo
/proc/partitions
/proc/slabinfo
/proc/softirqs
/proc/stat
/proc/swaps
/proc/uptime
/proc/version
/proc/vmallocinfo
/proc/vmstat
/proc/zoneinfo
/proc/driver/rtc
/proc/net/arp
/proc/net/dev
/proc/net/dev_mcast
/proc/net/if_inet6
/proc/net/igmp
/proc/net/igmp6
/proc/net/ipv6_route
/proc/net/netlink
/proc/net/netstat
/proc/net/packet
/proc/net/protocols
/proc/net/route
/proc/net/unix
/proc/<pid>/fdinfo/<fd>
/proc/<pid>/io
/proc/<pid>/maps
/proc/<pid>/mountinfo
/proc/<pid>/numa_maps
/proc/<pid>/smaps
/proc/<pid>/stat
/proc/<pid>/statm
/proc/<pid>/status
- Magic syntax support for
/proc
files - Enhance bash and zsh autocompletions for
/proc
files, including the Magic syntax - Enhance metadata output to output metadata even when results are empty
- Enhance
free
parser to support-w
option integer conversions - Fix
ini
andkv
parsers so they don’t change key-names to lower case
NOTE: This can be a breaking change in your scripts - Fix
id
command parser to allow usernames and group names with spaces
New Features
Metadata Enhancement
The --meta-out
option was introduced in jc
version 1.21.0. This feature has been enhanced to ensure metadata is always output – even if there are no results. for example, when running arp
on macOS without the required -a
flag:
% jc --meta-out --pretty arp usage: arp [-n] [-i interface] hostname arp [-n] [-i interface] [-l] -a arp -d hostname [pub] [ifscope interface] arp -d [-i interface] -a arp -s hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface] arp -S hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface] arp -f filename [ { "_jc_meta": { "parser": "arp", "timestamp": 1664302590.903263, "magic_command": [ "arp" ], "magic_command_exit": 1 } } ]
New Parsers
/proc
File Parsers
Many new /proc
file parsers have been added and more will be released in future versions of jc
. The easiest way to use these parsers is by invoking the --proc
parser. The --proc
parser (Documentation) will analyze the input data and select the correct /proc
parser automatically:
% cat /proc/uptime | jc --proc -p { "up_time": 46901.13, "idle_time": 46856.66 }
It is possible to manually select the /proc
file parser by designating the specific parser. For example:
% cat /proc/uptime | jc --proc-uptime -p { "up_time": 46901.13, "idle_time": 46856.66 }
Finally, you can also use the Magic syntax to convert /proc
files:
% jc -p /proc/uptime { "up_time": 46901.13, "idle_time": 46856.66 }
The bash and zsh autocompletion scripts have been updated to allow autocompletion of all of the new /proc
file parsers and the /proc
file magic syntax.
Individual /proc
file parsers are hidden from jc --help
. To find the low-level parser names, you can use jc -hh
or jc --about
:
% jc -hh jc converts the output of many commands, file-types, and strings to JSON or YAML Usage: Standard syntax: COMMAND | jc [OPTIONS] PARSER cat FILE | jc [OPTIONS] PARSER echo STRING | jc [OPTIONS] PARSER Magic syntax: jc [OPTIONS] COMMAND jc [OPTIONS] /proc/<path-to-procfile> Parsers: <snip> --ping `ping` and `ping6` command parser --ping-s `ping` and `ping6` command streaming parser --pip-list `pip list` command parser --pip-show `pip show` command parser --plist PLIST file parser --postconf `postconf -M` command parser --proc `/proc/` file parser --proc-buddyinfo `/proc/buddyinfo` file parser --proc-consoles `/proc/consoles` file parser --proc-cpuinfo `/proc/cpuinfo` file parser --proc-crypto `/proc/crypto` file parser --proc-devices `/proc/devices` file parser --proc-diskstats `/proc/diskstats` file parser --proc-filesystems `/proc/filesystems` file parser --proc-interrupts `/proc/interrupts` file parser --proc-iomem `/proc/iomem` file parser <snip> Options: -a, --about about jc -C, --force-color force color output even when using pipes (overrides -m) -d, --debug debug (double for verbose debug) -h, --help help (--help --parser_name for parser documentation) -m, --monochrome monochrome output -M, --meta-out add metadata to output including timestamp, etc. -p, --pretty pretty print output -q, --quiet suppress warnings (double to ignore streaming errors) -r, --raw raw output -u, --unbuffer unbuffer output -v, --version version info -y, --yaml-out YAML output -B, --bash-comp gen Bash completion: jc -B > /etc/bash_completion.d/jc -Z, --zsh-comp gen Zsh completion: jc -Z > "${fpath[1]}/_jc" Examples: Standard Syntax: $ dig www.google.com | jc --pretty --dig $ cat /proc/meminfo | jc --pretty --proc Magic Syntax: $ jc --pretty dig www.google.com $ jc --pretty /proc/meminfo Parser Documentation: $ jc --help --dig Show Hidden Parsers: $ jc -hh
You can find the schema for each /proc
parser with the same help syntax:
% jc --help --proc-uptime
v1.22.1 Updates
- Add
udevadm
command parser - Add
lspci
command parser - Add
pci.ids
file parser - Fix
proc-pid-stat
parser for command names with spaces and newlines - Enhance
ip-address
parser to addip_split
field - Rename
iso-datetime
parser todatetime-iso
. A deprecation warning will display untiliso-datetime
is removed in a future version. - Optimize performance of calculated timestamps
- Add support for deprecating parsers
- Move jc-web demo site from heroku to render.com
In addition to three new parsers, this release includes several enhancements and fixes. The proc-pid-stat
parser now supports command names that contain spaces and newlines. The ip-address
string parser now includes a new ip-split
field which splits each IPv4 octet or IPv6 segment into its own string within a list:
$ echo 192.168.2.10/24 | jc --ip-address -p { "version": 4, "max_prefix_length": 32, "ip": "192.168.2.10", "ip_compressed": "192.168.2.10", "ip_exploded": "192.168.2.10", "ip_split": [ "192", "168", "2", "10" ], ... }
The iso-datetime
parser has been renamed to datetime-iso
. Don’t worry, the old name still works, but will print a deprecation warning to STDERR
. This change introduces the new parser deprecation functionality. Deprecated parsers are identified with a deprecated
field in the --about
JSON output.
Some performance optimizations have been included for calculated timestamps. And, finally, we have moved the jc-web demo from Heroku to Render due to pricing changes.
udevadm
command parser
A new udevadm
linux command parser has been released. (Documentation)
$ udevadm info --query=all /dev/sda | jc --udevadm -p { "P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda", "N": "sda", "L": 0, "S": [ "disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0" ], "E": { "DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda", "DEVNAME": "/dev/sda", "DEVTYPE": "disk", "MAJOR": "8", "MINOR": "0", "SUBSYSTEM": "block", "USEC_INITIALIZED": "6100111", "SCSI_TPGS": "0", "SCSI_TYPE": "disk", "SCSI_VENDOR": "VMware,", "SCSI_VENDOR_ENC": "VMware,\\x20", "SCSI_MODEL": "VMware_Virtual_S", "SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S", "SCSI_REVISION": "1.0", "ID_SCSI": "1", "ID_VENDOR": "VMware_", "ID_VENDOR_ENC": "VMware\\x2c\\x20", "ID_MODEL": "VMware_Virtual_S", "ID_MODEL_ENC": "VMware\\x20Virtual\\x20S", "ID_REVISION": "1.0", "ID_TYPE": "disk", "MPATH_SBIN_PATH": "/sbin", "ID_BUS": "scsi", "ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0", "ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0", "ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7", "ID_PART_TABLE_TYPE": "gpt", "DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0", "TAGS": ":systemd:" } }
lspci
command parser
A new lspci
linux command parser has been added. (Documentation)
$ lspci -nnmmv | jc --lspci -p [ { "slot": "ff:02:05.0", "domain": "ff", "domain_int": 255, "bus": "02", "bus_int": 2, "dev": "05", "dev_int": 5, "function": "0", "function_int": 0, "class": "SATA controller", "class_id": "0106", "class_id_int": 262, "vendor": "VMware", "vendor_id": "15ad", "vendor_id_int": 5549, "device": "SATA AHCI controller", "device_id": "07e0", "device_id_int": 2016, "svendor": "VMware", "svendor_id": "15ad", "svendor_id_int": 5549, "sdevice": "SATA AHCI controller", "sdevice_id": "07e0", "sdevice_id_int": 2016, "physlot": "37", "physlot_int": 55, "progif": "01", "progif_int": 1 }, ... ]
pci.ids
file parser
A new file parser for the pci.ids
database file has been added. (Documentation) This parser allows you to use tools like jq
to query the database. The pci.ids
database file can be found here: https://raw.githubusercontent.com/pciutils/pciids/master/pci.ids
$ cat pci.ids | jc --pci-ids | jq '.vendors._001c._0001._001c._0005.subsystem_name' "2 Channel CAN Bus SJC1000 (Optically Isolated)" $ cat pci.ids | jc --pci-ids | jq '.classes._0c._03._40' "USB4 Host Interface"
v1.22.2 Updates
- Add
sshd-conf
parser forsshd
configuration files andsshd -T
output - Add
findmnt
command parser - Add
git ls-remote
command parser - Add
os-prober
command parser - Add SemVer string parser
- Enhance the
ifconfig
parser so it can output multiple IPv4 and IPv6 addresses - Enhance the
ifconfig
parser so it can output additional fields common on BSD - Enhance XML file parser with optional
_
prefix for attributes instead of@
by using the--raw
option. This can make it easier to filter the JSON output in some tools. - Fix the XML file parser to output a normal Dictionary instead of OrderdDict. This cleans up YAML output. (No
!!omap
comments) - Fix the standard and streaming CSV parsers for UTF-8 encoded CSV files with leading BOM bytes
- Fix exit code to be non-zero on keyboard interrupt
- Allow parser module objects to be used as arguments to
jc.get_help()
andjc.parser_info()
- Catch unexpected exceptions in the CLI
- Add error message on keyboard interrupt to STDERR
- Add python 3.11 tests to GitHub actions
sshd-conf
file and command parser
A new sshd-conf
file and sshd -T
parser has been added. (Documentation)
$ sshd -T | jc --sshd-conf -p { "acceptenv": [ "LANG", "LC_*" ], "addressfamily": "any", "allowagentforwarding": "yes", "allowstreamlocalforwarding": "yes", "allowtcpforwarding": "yes", "authenticationmethods": "any", "authorizedkeyscommand": "none", "authorizedkeyscommanduser": "none", "authorizedkeysfile": [ ".ssh/authorized_keys", ".ssh/authorized_keys2" ], "authorizedprincipalscommand": "none", ... }
findmnt
command parser
A new findmnt
command parser has been added. (Documentation)
$ findmnt | jc --findmnt -p [ { "target": "/", "source": "/dev/mapper/centos-root", "fstype": "xfs", "options": [ "rw", "relatime", "seclabel", "attr2", "inode64", "noquota" ] }, { "target": "/sys/fs/cgroup", "source": "tmpfs", "fstype": "tmpfs", "options": [ "ro", "nosuid", "nodev", "noexec", "seclabel" ], "kv_options": { "mode": "755" } }, ... ]
git ls-remote
command parser
A new git ls-remote
command parser has been added. (Documentation)
$ git ls-remote | jc --git-ls-remote -p { "HEAD": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338", "refs/heads/dev": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389", "refs/heads/master": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338", "refs/pull/1/head": "e416c77bed1267254da972b0f95b7ff1d43fccef", ... } $ git ls-remote | jc --git-ls-remote -p -r [ { "reference": "HEAD", "commit": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338" }, { "reference": "refs/heads/dev", "commit": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389" }, ... ]
os-prober
command parser
A new os-prober
command parser has been added. (Documentation)
$ os-prober | jc --os-prober -p { "partition": "/dev/sda1", "name": "Windows 10", "short_name": "Windows", "type": "chain" }
Semantic Version string parser
A new Semantic Version string parser has been added. (Documentation)
$ echo 1.2.3-rc.1+44837 | jc --semver -p { "major": 1, "minor": 2, "patch": 3, "prerelease": "rc.1", "build": "44837" }
ifconfig
command parser enhancements
The ifconfig
command parser has been enhanced to support multiple IPv4 and IPv6 addresses. Also many more BSD/macOS fields are parsed.
$ ifconfig | jc --ifconfig -p [ { "name": "en0", "flags": 8863, "state": [ "UP", "BROADCAST", "SMART", "RUNNING", "SIMPLEX", "MULTICAST" ], "mtu": 1500, "type": null, "mac_addr": "f0:18:98:03:d9:30", "ipv4_addr": "192.168.1.72", "ipv4_mask": "255.255.255.0", "ipv4_bcast": "192.168.1.255", "ipv6_addr": "fe80::8b7:1281:7499:b504", "ipv6_mask": 64, "ipv6_scope": "0x8", "ipv6_type": null, "metric": null, "rx_packets": null, "rx_errors": null, "rx_dropped": null, "rx_overruns": null, "rx_frame": null, "tx_packets": null, "tx_errors": null, "tx_dropped": null, "tx_overruns": null, "tx_carrier": null, "tx_collisions": null, "rx_bytes": null, "tx_bytes": null, "nd6_options": 201, "nd6_flags": [ "PERFORMNUD", "DAD" ], "status": "active", "ipv4": [ { "address": "192.168.1.72", "mask": "255.255.255.0", "broadcast": "192.168.1.255" } ], "ipv6": [ { "address": "fe80::8b7:1281:7499:b504", "mask": 64, "scope": "0x8" } ] }, ... ]
v1.22.3 Updates
- Add Common Log Format and Combined Log Format file parser (standard and streaming)
- Add PostgreSQL password file parser
- Add
openvpn-status.log
file parser - Add
cbt
command parser (Google Big Table) - Enhance
ifconfig
parser with more information on BSD - Fix
ifconfig
parser to capture some IPv6 addresses missed on BSD - Fix
git-log
andgit-log-s
parsers for failure on empty author name - Update
os-prober
parser with split EFI partition fields
v1.22.4 Updates
- Add
iwconfig
command parser - Add NeXTSTEP format support to the PLIST file parser
- Fix
proc
parser magic signature detection for/proc/pid/stat
hacks - Fix
x509-cert
parser for string serial numbers - Add category tags to parser metadata: generic, standard, file, string, binary, command
- Add “list parsers by category” view to help
- Fix python 3.6-related issues
- Add python 3.6 to automated tests
v1.22.5 Updates
- Add TOML file parser
- Add INI with duplicate key support file parser
- Add AIX support for the
arp
command parser - Add AIX support for the
mount
command parser - Fix
lsusb
command parser when extra hub port status information is output - Fix INI file parser to include top-level values with no section header
- Fix INI file parser to not specially handle the [DEFAULT] section
- Fix INI file and Key/Value parsers to only remove one quotation mark from the
beginning and end of values.
Happy parsing!