JC Version 1.17.0 Released

kellyjonbrazil

5 years ago

See below for v1.17.x updates

I’m excited to announce the release of jc version 1.17.0 available on github and pypi. This release includes streaming parser support, including three new streaming parsers (ls-s, ping-s, and vmstat-s) and one new standard parser (vmstat), bringing the total number of parsers to 78.

The streaming parsers output JSON Lines (aka NDJSON), which can be ingested by streaming processors like jq, elastic, Splunk, etc. These parsers use significantly less memory while converting large amounts of output (e.g. ls -lR /), and in some cases can be faster than standard parsers. Just like standard parsers, streaming parsers can be used both at the CLI and as Python libraries. When used as Python libraries, parse() is a generator function and returns an iterator which can be used in a loop for lazy processing of the stream.

The -u CLI option has been added to unbuffer the output. This is useful when piping jc output to another process like jq and the input stream is very slow (e.g. ping output). With the unbuffer option enabled you will be able to see the JSON output immediately when using streaming parsers in this scenario instead of waiting for the buffer to be filled.

Streaming parsers also have an ignore_exceptions option (-qq on the CLI) to allow uninterrupted processing in case any unexpected parsing errors occur. This can be used for long-lived streams so the pipe will not be broken if there is a hiccup in the stream. When this option is used, a _jc_meta object with a success attribute is added to each emitted JSON object. This allows the downstream application to decide whether to ignore the unparsable lines or further process those lines.

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, click here.

To upgrade with pip:

$ pip3 install --upgrade jc

Sections

New Features

Warning and Error messages now wrap to the terminal width.
Support for streaming parsers for much lower memory consumption when converting large amounts of command output.
-u CLI option unbuffers jc output. This is useful when converting slow output like ping through the ping-s streaming parser.
-qq CLI option makes jc “extra quiet” for streaming parsers. This equates to the ignore_exceptions argument in the streaming parser’s parse() function when using jc as a Python library.

When using this option, the streaming parser will not stop when parsing errors are encountered. Instead, a _jc_meta object included in the JSON output will have success set to false and the error and line attributes will be set to the error message and original line contents, respectively. Here are examples of the additional _jc_meta object:

Successfully parsed line with -qq option:

{
  "foo": "data1",
  "bar": "data2",
  "baz": "data3",
  "_jc_meta": {
    "success": true
  }
}

Unsuccessfully parsed line with -qq option:

{
  "_jc_meta": {
    "success": false,
    "error": "error message",
    "line": "original line data"
  }
}

New Parsers

jc now supports 78 parsers. New parsers include vmstat and three streaming parsers: ls-s, ping-s, and vmstat-s.

Streaming parsers are considered Beta quality. Even though the streaming parsers have gone through extensive testing, I would like to get more feedback from users before considering them 1.0. Please try them out and provide any feedback as a github issue.
Also, feel free to open a github issue if you have recommendations for other streaming parsers. Currently I’m thinking about adding streaming parsers for CSV, YAML, and XML documents in a future release.

Documentation and schemas for all parsers can be found here.

`vmstat` command parser

Linux support for the vmstat command. (Documentation):

$ vmstat | jc --vmstat -p          # or jc -p vmstat
[
  {
    "runnable_procs": 2,
    "uninterruptible_sleeping_procs": 0,
    "virtual_mem_used": 0,
    "free_mem": 2794468,
    "buffer_mem": 2108,
    "cache_mem": 741208,
    "inactive_mem": null,
    "active_mem": null,
    "swap_in": 0,
    "swap_out": 0,
    "blocks_in": 1,
    "blocks_out": 3,
    "interrupts": 29,
    "context_switches": 57,
    "user_time": 0,
    "system_time": 0,
    "idle_time": 99,
    "io_wait_time": 0,
    "stolen_time": 0,
    "timestamp": null,
    "timezone": null
  }
]

`vmstat-s` streaming command parser

Linux support for the vmstat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ vmstat | jc --vmstat-s
{"runnable_procs":2,"uninterruptible_sleeping_procs":...timestamp":null,"timezone":null}

`ls-s` streaming command parser

Linux, macOS, and BSD support for the ls command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ ls -l /usr/bin | jc --ls-s
{"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
{"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
{"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
...

`ping-s` streaming command parser

Linux, macOS, and BSD support for the ping and ping6 commands. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ ping 1.1.1.1 | jc --ping-s
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
...

Updated Parsers

No updated parsers in this release

Schema Changes

No schema changes in this release

Happy parsing!

For more information on the motivations for creating jc, see my blog post.

v1.17.1 Updates

Fix file parser for gzip files
Fix uname parser for cases where the ‘processor’ and/or ‘hardware_platform’ fields are missing on linux
Fix uname parser on FreeBSD
Add lsusb parser tested on linux
Add CSV file streaming parser
Add testing for Python 3.10.0

`lsusb` command parser

Linux support for the lsusb command. (Documentation):

$ lsusb -v | jc --lsusb -p          # or: jc -p lsusb -v
[
  {
    "bus": "002",
    "device": "001",
    "id": "1d6b:0001",
    "description": "Linux Foundation 1.1 root hub",
    "device_descriptor": {
      "bLength": {
        "value": "18"
      },
      "bDescriptorType": {
        "value": "1"
      },
      "bcdUSB": {
        "value": "1.10"
      },
      ...
      "bNumConfigurations": {
        "value": "1"
      },
      "configuration_descriptor": {
        "bLength": {
          "value": "9"
        },
        ...
        "iConfiguration": {
          "value": "0"
        },
        "bmAttributes": {
          "value": "0xe0",
          "attributes": [
            "Self Powered",
            "Remote Wakeup"
          ]
        },
        "MaxPower": {
          "description": "0mA"
        },
        "interface_descriptors": [
          {
            "bLength": {
              "value": "9"
            },
            ...
            "bInterfaceProtocol": {
              "value": "0",
              "description": "Full speed (or root) hub"
            },
            "iInterface": {
              "value": "0"
            },
            "endpoint_descriptors": [
              {
                "bLength": {
                  "value": "7"
                },
                ...
                "bmAttributes": {
                  "value": "3",
                  "attributes": [
                    "Transfer Type  Interrupt",
                    "Synch Type  None",
                    "Usage Type  Data"
                  ]
                },
                "wMaxPacketSize": {
                  "value": "0x0002",
                  "description": "1x 2 bytes"
                },
                "bInterval": {
                  "value": "255"
                }
              }
            ]
          }
        ]
      }
    },
    "hub_descriptor": {
      "bLength": {
        "value": "9"
      },
      ...
      "wHubCharacteristic": {
        "value": "0x000a",
        "attributes": [
          "No power switching (usb 1.0)",
          "Per-port overcurrent protection"
        ]
      },
      ...
      "hub_port_status": {
        "Port 1": {
          "value": "0000.0103",
          "attributes": [
            "power",
            "enable",
            "connect"
          ]
        },
        "Port 2": {
          "value": "0000.0103",
          "attributes": [
            "power",
            "enable",
            "connect"
          ]
        }
      }
    },
    "device_status": {
      "value": "0x0001",
      "description": "Self Powered"
    }
  }
]

`csv-s` streaming command parser

Support for CSV files. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ cat homes.csv
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3,  60, 0.28,  3167
175, 180, 18,  8, 4, 1,  12, 0.43,  4033
129, 132, 13,  6, 3, 1,  41, 0.33,  1471
...

$ cat homes.csv | jc --csv-s
{"Sell":"142","List":"160","Living":"28","Rooms":"10","Beds":"5","Baths":"3","Age":"60","Acres":"0.28","Taxes":"3167"}
{"Sell":"175","List":"180","Living":"18","Rooms":"8","Beds":"4","Baths":"1","Age":"12","Acres":"0.43","Taxes":"4033"}
{"Sell":"129","List":"132","Living":"13","Rooms":"6","Beds":"3","Baths":"1","Age":"41","Acres":"0.33","Taxes":"1471"}
...

v1.17.2 Updates

Fix ping parser to add Alpine linux support
Fix netstat parser for older versions of netstat on linux
Fix df parser for cases where the ‘filesystem’ field overflows the column length

v1.17.3 Updates

Update parsers to exit with error if non-string input is detected (raise TypeError)
Update streaming parsers to exit with error if non-iterable input is detected (raise TypeError)
Simplify quiet-checking in parsers
Add iostat parser tested on linux
Add iostat streaming parser tested on linux

`iostat` command parser

Linux support for the iostat command. (Documentation):

$ iostat | jc --iostat          # or: jc -p iostat
[
  {
      "percent_user": 0.15,
      "percent_nice": 0.0,
      "percent_system": 0.18,
      "percent_iowait": 0.0,
      "percent_steal": 0.0,
      "percent_idle": 99.67,
      "type": "cpu"
  },
  {
      "device": "sda",
      "tps": 0.29,
      "kb_read_s": 7.22,
      "kb_wrtn_s": 1.25,
      "kb_read": 194341,
      "kb_wrtn": 33590,
      "type": "device"
  },
  {
      "device": "dm-0",
      "tps": 0.29,
      "kb_read_s": 5.99,
      "kb_wrtn_s": 1.17,
      "kb_read": 161361,
      "kb_wrtn": 31522,
      "type": "device"
  },
  {
      "device": "dm-1",
      "tps": 0.0,
      "kb_read_s": 0.08,
      "kb_wrtn_s": 0.0,
      "kb_read": 2204,
      "kb_wrtn": 0,
      "type": "device"
  }
]

`iostat-s` streaming command parser

Linux support for the iostat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ iostat | jc --iostat-s
{"percent_user":0.14,"percent_nice":0.0,"percent_system":0.16,"percent_iowait":0.0,"percent_steal":0.0,"percent_idle":99.7,"type":"cpu"}
{"device":"sda","tps":0.24,"kb_read_s":5.28,"kb_wrtn_s":1.1,"kb_read":203305,"kb_wrtn":42368,"type":"device"}
...

v1.17.4 Updates

Add support for the NO_COLOR environment variable to set mono (http://no-color.org/)
Add -C option to force color output even when using pipes (overrides -m and NO_COLOR)

v1.17.5 Updates

Add zipinfo parser tested on linux and macOS

`zipinfo` command parser

Linux and macOS support for the zipinfo command. (Documentation):

$ zipinfo file.zip | jc --zipinfo -p
[
  {
    "archive": "file.zip",
    "size": 4116,
    "size_unit": "bytes",
    "number_entries": 1,
    "number_files": 1,
    "bytes_uncompressed": 11837,
    "bytes_compressed": 3966,
    "percent_compressed": 66.5,
    "files": [
      {
        "flags": "-rw-r--r--",
        "zipversion": "2.1",
        "zipunder": "unx",
        "filesize": 11837,
        "type": "bX",
        "method": "defN",
        "date": "21-Dec-08",
        "time": "20:50",
        "filename": "compressed_file"
      }
    ]
  }
]

v1.17.6 Updates

Add jar-manifest file parser for MANIFEST.MF files.
Fix CSV parsers for some files that include double-quotes

`jar-manifest` file parser

Support for Java JAR Manifest files. (Documentation):

$ cat MANIFEST.MF | jc --jar-manifest -p
[
  {
    "Import_Package": "com.conversantmedia.util.concurrent;resolution:=optional,com.fasterxml.jackson.annotation;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core.type;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.cor...",
    "Export_Package": "org.apache.logging.log4j.core;uses:=\"org.apache.logging.log4j,org.apache.logging.log4j.core.config,org.apache.logging.log4j.core.impl,org.apache.logging.log4j.core.layout,org.apache.logging.log4j.core.time,org.apache.logging.log4j.message,org.apache.logging.log4j.spi,org.apache.logging.log4j.status...",
    "Manifest_Version": "1.0",
    "Bundle_License": "https://www.apache.org/licenses/LICENSE-2.0.txt",
    "Bundle_SymbolicName": "org.apache.logging.log4j.core",
    "Built_By": "matt",
    "Bnd_LastModified": "1639373735804",
    "Implementation_Vendor_Id": "org.apache.logging.log4j",
    "Specification_Title": "Apache Log4j Core",
    "Log4jReleaseManager": "Matt Sicker",
    ...
  }
]

v1.17.7 Updates

Add stat-s streaming parser for the stat command.

`stat-s` streaming command parser

Linux, macOS, and FreeBSD support for the stat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ stat | jc --stat-s
{"file":"(stdin)","unix_device":1027739696,"inode":1155,"flags":"crw--w----","links":1,"user":"kbrazil","group":"tty","rdev":268435456,"size":0,"access_time":"Jan  4 15:27:44 2022","modify_time":"Jan  4 15:27:44 2022","change_time":"Jan  4 15:27:44 2022","birth_time":"Dec 31 16:00:00 1969","block_size":131072,"blocks":0,"unix_flags":"0","access_time_epoch":1641338864,"access_time_epoch_utc":null,"modify_time_epoch":1641338864,"modify_time_epoch_utc":null,"change_time_epoch":1641338864,"change_time_epoch_utc":null,"birth_time_epoch":null,"birth_time_epoch_utc":null}

New Features

New Parsers

vmstat command parser

vmstat-s streaming command parser

ls-s streaming command parser

ping-s streaming command parser