JC Version 1.22.0 Released

I’m excited to announce the release of jc version 1.22.0 available on github and pypi. jc now supports over 170 standard and streaming parsers. Thank you to the Open Source community for making this possible!

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, see the project README.

To upgrade with pip:

$ pip3 install --upgrade jc

Try the jc web demo!

Sections

What’s New

Add /proc file parsers for linux. Support for the following files:
/proc/buddyinfo
/proc/consoles
/proc/cpuinfo
/proc/crypto
/proc/devices
/proc/diskstats
/proc/filesystems
/proc/interrupts
/proc/iomem
/proc/ioports
/proc/loadavg
/proc/locks
/proc/meminfo
/proc/modules
/proc/mtrr
/proc/pagetypeinfo
/proc/partitions
/proc/slabinfo
/proc/softirqs
/proc/stat
/proc/swaps
/proc/uptime
/proc/version
/proc/vmallocinfo
/proc/vmstat
/proc/zoneinfo
/proc/driver/rtc
/proc/net/arp
/proc/net/dev
/proc/net/dev_mcast
/proc/net/if_inet6
/proc/net/igmp
/proc/net/igmp6
/proc/net/ipv6_route
/proc/net/netlink
/proc/net/netstat
/proc/net/packet
/proc/net/protocols
/proc/net/route
/proc/net/unix
/proc/<pid>/fdinfo/<fd>
/proc/<pid>/io
/proc/<pid>/maps
/proc/<pid>/mountinfo
/proc/<pid>/numa_maps
/proc/<pid>/smaps
/proc/<pid>/stat
/proc/<pid>/statm
/proc/<pid>/status
Magic syntax support for /proc files
Enhance bash and zsh autocompletions for /proc files, including the Magic syntax
Enhance metadata output to output metadata even when results are empty
Enhance free parser to support -w option integer conversions
Fix ini and kv parsers so they don’t change key-names to lower case
NOTE: This can be a breaking change in your scripts
Fix id command parser to allow usernames and group names with spaces

New Features

Metadata Enhancement

The --meta-out option was introduced in jc version 1.21.0. This feature has been enhanced to ensure metadata is always output – even if there are no results. for example, when running arp on macOS without the required -a flag:

% jc --meta-out --pretty arp
usage: arp [-n] [-i interface] hostname
       arp [-n] [-i interface] [-l] -a
       arp -d hostname [pub] [ifscope interface]
       arp -d [-i interface] -a
       arp -s hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface]
       arp -S hostname ether_addr [temp] [reject] [blackhole] [pub [only]] [ifscope interface]
       arp -f filename
[
  {
    "_jc_meta": {
      "parser": "arp",
      "timestamp": 1664302590.903263,
      "magic_command": [
        "arp"
      ],
      "magic_command_exit": 1
    }
  }
]

New Parsers

`/proc` File Parsers

Many new /proc file parsers have been added and more will be released in future versions of jc. The easiest way to use these parsers is by invoking the --proc parser. The --proc parser (Documentation) will analyze the input data and select the correct /proc parser automatically:

% cat /proc/uptime | jc --proc -p
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

It is possible to manually select the /proc file parser by designating the specific parser. For example:

% cat /proc/uptime | jc --proc-uptime -p
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

Finally, you can also use the Magic syntax to convert /proc files:

% jc -p /proc/uptime
{
  "up_time": 46901.13,
  "idle_time": 46856.66
}

The bash and zsh autocompletion scripts have been updated to allow autocompletion of all of the new /proc file parsers and the /proc file magic syntax.

Individual /proc file parsers are hidden from jc --help. To find the low-level parser names, you can use jc -hh or jc --about:

% jc -hh
jc converts the output of many commands, file-types, and strings to JSON or YAML

Usage:

    Standard syntax:

        COMMAND | jc [OPTIONS] PARSER

        cat FILE | jc [OPTIONS] PARSER

        echo STRING | jc [OPTIONS] PARSER

    Magic syntax:

        jc [OPTIONS] COMMAND

        jc [OPTIONS] /proc/<path-to-procfile>

Parsers:
<snip>
    --ping              `ping` and `ping6` command parser
    --ping-s            `ping` and `ping6` command streaming parser
    --pip-list          `pip list` command parser
    --pip-show          `pip show` command parser
    --plist             PLIST file parser
    --postconf          `postconf -M` command parser
    --proc              `/proc/` file parser
    --proc-buddyinfo    `/proc/buddyinfo` file parser
    --proc-consoles     `/proc/consoles` file parser
    --proc-cpuinfo      `/proc/cpuinfo` file parser
    --proc-crypto       `/proc/crypto` file parser
    --proc-devices      `/proc/devices` file parser
    --proc-diskstats    `/proc/diskstats` file parser
    --proc-filesystems  `/proc/filesystems` file parser
    --proc-interrupts   `/proc/interrupts` file parser
    --proc-iomem        `/proc/iomem` file parser
<snip>

Options:
    -a,  --about        about jc
    -C,  --force-color  force color output even when using pipes (overrides -m)
    -d,  --debug        debug (double for verbose debug)
    -h,  --help         help (--help --parser_name for parser documentation)
    -m,  --monochrome   monochrome output
    -M,  --meta-out     add metadata to output including timestamp, etc.
    -p,  --pretty       pretty print output
    -q,  --quiet        suppress warnings (double to ignore streaming errors)
    -r,  --raw          raw output
    -u,  --unbuffer     unbuffer output
    -v,  --version      version info
    -y,  --yaml-out     YAML output
    -B,  --bash-comp    gen Bash completion: jc -B > /etc/bash_completion.d/jc
    -Z,  --zsh-comp     gen Zsh completion: jc -Z > "${fpath[1]}/_jc"

Examples:
    Standard Syntax:
        $ dig www.google.com | jc --pretty --dig
        $ cat /proc/meminfo | jc --pretty --proc

    Magic Syntax:
        $ jc --pretty dig www.google.com
        $ jc --pretty /proc/meminfo

    Parser Documentation:
        $ jc --help --dig

    Show Hidden Parsers:
        $ jc -hh

You can find the schema for each /proc parser with the same help syntax:

% jc --help --proc-uptime

v1.22.1 Updates

Add udevadm command parser
Add lspci command parser
Add pci.ids file parser
Fix proc-pid-stat parser for command names with spaces and newlines
Enhance ip-address parser to add ip_split field
Rename iso-datetime parser to datetime-iso. A deprecation warning will display until iso-datetime is removed in a future version.
Optimize performance of calculated timestamps
Add support for deprecating parsers
Move jc-web demo site from heroku to render.com

In addition to three new parsers, this release includes several enhancements and fixes. The proc-pid-stat parser now supports command names that contain spaces and newlines. The ip-address string parser now includes a new ip-split field which splits each IPv4 octet or IPv6 segment into its own string within a list:

$ echo 192.168.2.10/24 | jc --ip-address -p
{
  "version": 4,
  "max_prefix_length": 32,
  "ip": "192.168.2.10",
  "ip_compressed": "192.168.2.10",
  "ip_exploded": "192.168.2.10",
  "ip_split": [
    "192",
    "168",
    "2",
    "10"
  ],
  ...
}

The iso-datetime parser has been renamed to datetime-iso. Don’t worry, the old name still works, but will print a deprecation warning to STDERR. This change introduces the new parser deprecation functionality. Deprecated parsers are identified with a deprecated field in the --about JSON output.

Some performance optimizations have been included for calculated timestamps. And, finally, we have moved the jc-web demo from Heroku to Render due to pricing changes.

`udevadm` command parser

A new udevadm linux command parser has been released. (Documentation)

$ udevadm info --query=all /dev/sda | jc --udevadm -p
{
  "P": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
  "N": "sda",
  "L": 0,
  "S": [
    "disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"
  ],
  "E": {
    "DEVPATH": "/devices/pci0000:00/0000:00:10.0/host32/target32:0:0/32:0:0:0/block/sda",
    "DEVNAME": "/dev/sda",
    "DEVTYPE": "disk",
    "MAJOR": "8",
    "MINOR": "0",
    "SUBSYSTEM": "block",
    "USEC_INITIALIZED": "6100111",
    "SCSI_TPGS": "0",
    "SCSI_TYPE": "disk",
    "SCSI_VENDOR": "VMware,",
    "SCSI_VENDOR_ENC": "VMware,\\x20",
    "SCSI_MODEL": "VMware_Virtual_S",
    "SCSI_MODEL_ENC": "VMware\\x20Virtual\\x20S",
    "SCSI_REVISION": "1.0",
    "ID_SCSI": "1",
    "ID_VENDOR": "VMware_",
    "ID_VENDOR_ENC": "VMware\\x2c\\x20",
    "ID_MODEL": "VMware_Virtual_S",
    "ID_MODEL_ENC": "VMware\\x20Virtual\\x20S",
    "ID_REVISION": "1.0",
    "ID_TYPE": "disk",
    "MPATH_SBIN_PATH": "/sbin",
    "ID_BUS": "scsi",
    "ID_PATH": "pci-0000:00:10.0-scsi-0:0:0:0",
    "ID_PATH_TAG": "pci-0000_00_10_0-scsi-0_0_0_0",
    "ID_PART_TABLE_UUID": "a5bd0c01-4210-46f2-b558-5c11c209a8f7",
    "ID_PART_TABLE_TYPE": "gpt",
    "DEVLINKS": "/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0",
    "TAGS": ":systemd:"
  }
}

`lspci` command parser

A new lspci linux command parser has been added. (Documentation)

$ lspci -nnmmv | jc --lspci -p
[
  {
    "slot": "ff:02:05.0",
    "domain": "ff",
    "domain_int": 255,
    "bus": "02",
    "bus_int": 2,
    "dev": "05",
    "dev_int": 5,
    "function": "0",
    "function_int": 0,
    "class": "SATA controller",
    "class_id": "0106",
    "class_id_int": 262,
    "vendor": "VMware",
    "vendor_id": "15ad",
    "vendor_id_int": 5549,
    "device": "SATA AHCI controller",
    "device_id": "07e0",
    "device_id_int": 2016,
    "svendor": "VMware",
    "svendor_id": "15ad",
    "svendor_id_int": 5549,
    "sdevice": "SATA AHCI controller",
    "sdevice_id": "07e0",
    "sdevice_id_int": 2016,
    "physlot": "37",
    "physlot_int": 55,
    "progif": "01",
    "progif_int": 1
  },
  ...
]

`pci.ids` file parser

A new file parser for the pci.ids database file has been added. (Documentation) This parser allows you to use tools like jq to query the database. The pci.ids database file can be found here: https://raw.githubusercontent.com/pciutils/pciids/master/pci.ids

$ cat pci.ids | jc --pci-ids | jq '.vendors._001c._0001._001c._0005.subsystem_name'
"2 Channel CAN Bus SJC1000 (Optically Isolated)"

$ cat pci.ids | jc --pci-ids | jq '.classes._0c._03._40'
"USB4 Host Interface"

v1.22.2 Updates

Add sshd-conf parser for sshd configuration files and sshd -T output
Add findmnt command parser
Add git ls-remote command parser
Add os-prober command parser
Add SemVer string parser
Enhance the ifconfig parser so it can output multiple IPv4 and IPv6 addresses
Enhance the ifconfig parser so it can output additional fields common on BSD
Enhance XML file parser with optional _ prefix for attributes instead of @ by using the --raw option. This can make it easier to filter the JSON output in some tools.
Fix the XML file parser to output a normal Dictionary instead of OrderdDict. This cleans up YAML output. (No !!omap comments)
Fix the standard and streaming CSV parsers for UTF-8 encoded CSV files with leading BOM bytes
Fix exit code to be non-zero on keyboard interrupt
Allow parser module objects to be used as arguments to jc.get_help() and jc.parser_info()
Catch unexpected exceptions in the CLI
Add error message on keyboard interrupt to STDERR
Add python 3.11 tests to GitHub actions

`sshd-conf` file and command parser

A new sshd-conf file and sshd -T parser has been added. (Documentation)

$ sshd -T | jc --sshd-conf -p
{
  "acceptenv": [
    "LANG",
    "LC_*"
  ],
  "addressfamily": "any",
  "allowagentforwarding": "yes",
  "allowstreamlocalforwarding": "yes",
  "allowtcpforwarding": "yes",
  "authenticationmethods": "any",
  "authorizedkeyscommand": "none",
  "authorizedkeyscommanduser": "none",
  "authorizedkeysfile": [
    ".ssh/authorized_keys",
    ".ssh/authorized_keys2"
  ],
  "authorizedprincipalscommand": "none",
  ...
}

`findmnt` command parser

A new findmnt command parser has been added. (Documentation)

$ findmnt | jc --findmnt -p
[
  {
    "target": "/",
    "source": "/dev/mapper/centos-root",
    "fstype": "xfs",
    "options": [
      "rw",
      "relatime",
      "seclabel",
      "attr2",
      "inode64",
      "noquota"
    ]
  },
  {
    "target": "/sys/fs/cgroup",
    "source": "tmpfs",
    "fstype": "tmpfs",
    "options": [
      "ro",
      "nosuid",
      "nodev",
      "noexec",
      "seclabel"
    ],
    "kv_options": {
      "mode": "755"
    }
  },
  ...
]

`git ls-remote` command parser

A new git ls-remote command parser has been added. (Documentation)

$ git ls-remote | jc --git-ls-remote -p
{
  "HEAD": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
  "refs/heads/dev": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389",
  "refs/heads/master": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338",
  "refs/pull/1/head": "e416c77bed1267254da972b0f95b7ff1d43fccef",
  ...
}

$ git ls-remote | jc --git-ls-remote -p -r
[
  {
    "reference": "HEAD",
    "commit": "214cd6b9e09603b3c4fa02203b24fb2bc3d4e338"
  },
  {
    "reference": "refs/heads/dev",
    "commit": "b884f6aacca39e05994596d8fdfa7e7c4f1e0389"
  },
  ...
]

`os-prober` command parser

A new os-prober command parser has been added. (Documentation)

$ os-prober | jc --os-prober -p
{
  "partition": "/dev/sda1",
  "name": "Windows 10",
  "short_name": "Windows",
  "type": "chain"
}

Semantic Version string parser

A new Semantic Version string parser has been added. (Documentation)

$ echo 1.2.3-rc.1+44837 | jc --semver -p
{
  "major": 1,
  "minor": 2,
  "patch": 3,
  "prerelease": "rc.1",
  "build": "44837"
}

`ifconfig` command parser enhancements

The ifconfig command parser has been enhanced to support multiple IPv4 and IPv6 addresses. Also many more BSD/macOS fields are parsed.

$ ifconfig | jc --ifconfig -p
[
  { 
    "name": "en0",
    "flags": 8863,
    "state": [
      "UP",
      "BROADCAST",
      "SMART",
      "RUNNING",
      "SIMPLEX",
      "MULTICAST"
    ],
    "mtu": 1500,
    "type": null,
    "mac_addr": "f0:18:98:03:d9:30",
    "ipv4_addr": "192.168.1.72",
    "ipv4_mask": "255.255.255.0",
    "ipv4_bcast": "192.168.1.255",
    "ipv6_addr": "fe80::8b7:1281:7499:b504",
    "ipv6_mask": 64,
    "ipv6_scope": "0x8",
    "ipv6_type": null,
    "metric": null,
    "rx_packets": null,
    "rx_errors": null,
    "rx_dropped": null,
    "rx_overruns": null,
    "rx_frame": null,
    "tx_packets": null,
    "tx_errors": null,
    "tx_dropped": null,
    "tx_overruns": null,
    "tx_carrier": null,
    "tx_collisions": null,
    "rx_bytes": null,
    "tx_bytes": null,
    "nd6_options": 201,
    "nd6_flags": [
      "PERFORMNUD",
      "DAD"
    ],
    "status": "active",
    "ipv4": [
      {
        "address": "192.168.1.72",
        "mask": "255.255.255.0",
        "broadcast": "192.168.1.255"
      }
    ],
    "ipv6": [
      {
        "address": "fe80::8b7:1281:7499:b504",
        "mask": 64,
        "scope": "0x8"
      }
    ]
  },
  ...
]

v1.22.3 Updates

Add Common Log Format and Combined Log Format file parser (standard and streaming)
Add PostgreSQL password file parser
Add openvpn-status.log file parser
Add cbt command parser (Google Big Table)
Enhance ifconfig parser with more information on BSD
Fix ifconfig parser to capture some IPv6 addresses missed on BSD
Fix git-log and git-log-s parsers for failure on empty author name
Update os-prober parser with split EFI partition fields

v1.22.4 Updates

Add iwconfig command parser
Add NeXTSTEP format support to the PLIST file parser
Fix proc parser magic signature detection for /proc/pid/stat hacks
Fix x509-cert parser for string serial numbers
Add category tags to parser metadata: generic, standard, file, string, binary, command
Add “list parsers by category” view to help
Fix python 3.6-related issues
Add python 3.6 to automated tests

v1.22.5 Updates

Add TOML file parser
Add INI with duplicate key support file parser
Add AIX support for the arp command parser
Add AIX support for the mount command parser
Fix lsusb command parser when extra hub port status information is output
Fix INI file parser to include top-level values with no section header
Fix INI file parser to not specially handle the [DEFAULT] section
Fix INI file and Key/Value parsers to only remove one quotation mark from the
beginning and end of values.

Happy parsing!

JC Version 1.22.0 Released

What’s New

New Features

Metadata Enhancement

New Parsers

`/proc` File Parsers

v1.22.1 Updates

`udevadm` command parser

`lspci` command parser

`pci.ids` file parser

v1.22.2 Updates

`sshd-conf` file and command parser

`findmnt` command parser

`git ls-remote` command parser

`os-prober` command parser

Semantic Version string parser

`ifconfig` command parser enhancements

v1.22.3 Updates

v1.22.4 Updates

v1.22.5 Updates

Like this:

Published by kellyjonbrazil

Leave a ReplyCancel reply

What’s New

New Features

Metadata Enhancement

New Parsers

/proc File Parsers

v1.22.1 Updates

udevadm command parser

lspci command parser

pci.ids file parser

v1.22.2 Updates

sshd-conf file and command parser

findmnt command parser

git ls-remote command parser

os-prober command parser

Semantic Version string parser

ifconfig command parser enhancements

v1.22.3 Updates

v1.22.4 Updates

v1.22.5 Updates

Like this:

Published by kellyjonbrazil

Leave a ReplyCancel reply

`/proc` File Parsers

`udevadm` command parser

`lspci` command parser

`pci.ids` file parser

`sshd-conf` file and command parser

`findmnt` command parser

`git ls-remote` command parser

`os-prober` command parser

`ifconfig` command parser enhancements