JC Version 1.17.0 Released

I’m excited to announce the release of jc version 1.17.0 available on github and pypi. This release includes streaming parser support, including three new streaming parsers (ls-s, ping-s, and vmstat-s) and one new standard parser (vmstat), bringing the total number of parsers to 78.

The streaming parsers output JSON Lines (aka NDJSON), which can be ingested by streaming processors like jq, elastic, Splunk, etc. These parsers use significantly less memory while converting large amounts of output (e.g. ls -lR /), and in some cases can be faster than standard parsers. Just like standard parsers, streaming parsers can be used both at the CLI and as Python libraries. When used as Python libraries, parse() is a generator function and returns an iterator which can be used in a loop for lazy processing of the stream.

The -u CLI option has been added to unbuffer the output. This is useful when piping jc output to another process like jq and the input stream is very slow (e.g. ping output). With the unbuffer option enabled you will be able to see the JSON output immediately when using streaming parsers in this scenario instead of waiting for the buffer to be filled.

Streaming parsers also have an ignore_exceptions option (-qq on the CLI) to allow uninterrupted processing in case any unexpected parsing errors occur. This can be used for long-lived streams so the pipe will not be broken if there is a hiccup in the stream. When this option is used, a _jc_meta object with a success attribute is added to each emitted JSON object. This allows the downstream application to decide whether to ignore the unparsable lines or further process those lines.

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, click here.

To upgrade with pip:

$ pip3 install --upgrade jc

New Features

  • Warning and Error messages now wrap to the terminal width.
  • Support for streaming parsers for much lower memory consumption when converting large amounts of command output.
  • -u CLI option unbuffers jc output. This is useful when converting slow output like ping through the ping-s streaming parser.
  • -qq CLI option makes jc “extra quiet” for streaming parsers. This equates to the ignore_exceptions argument in the streaming parser’s parse() function when using jc as a Python library.

    When using this option, the streaming parser will not stop when parsing errors are encountered. Instead, a _jc_meta object included in the JSON output will have success set to false and the error and line attributes will be set to the error message and original line contents, respectively. Here are examples of the additional _jc_meta object:

Successfully parsed line with -qq option:

{
  "foo": "data1",
  "bar": "data2",
  "baz": "data3",
  "_jc_meta": {
    "success": true
  }
}

Unsuccessfully parsed line with -qq option:

{
  "_jc_meta": {
    "success": false,
    "error": "error message",
    "line": "original line data"
  }
}

New Parsers

jc now supports 78 parsers. New parsers include vmstat and three streaming parsers: ls-s, ping-s, and vmstat-s.

Streaming parsers are considered Beta quality. Even though the streaming parsers have gone through extensive testing, I would like to get more feedback from users before considering them 1.0. Please try them out and provide any feedback as a github issue.

Also, feel free to open a github issue if you have recommendations for other streaming parsers. Currently I’m thinking about adding streaming parsers for CSV, YAML, and XML documents in a future release.

Documentation and schemas for all parsers can be found here.

vmstat command parser

Linux support for the vmstat command. (Documentation):

$ vmstat | jc --vmstat -p          # or jc -p vmstat
[
  {
    "runnable_procs": 2,
    "uninterruptible_sleeping_procs": 0,
    "virtual_mem_used": 0,
    "free_mem": 2794468,
    "buffer_mem": 2108,
    "cache_mem": 741208,
    "inactive_mem": null,
    "active_mem": null,
    "swap_in": 0,
    "swap_out": 0,
    "blocks_in": 1,
    "blocks_out": 3,
    "interrupts": 29,
    "context_switches": 57,
    "user_time": 0,
    "system_time": 0,
    "idle_time": 99,
    "io_wait_time": 0,
    "stolen_time": 0,
    "timestamp": null,
    "timezone": null
  }
]

vmstat-s streaming command parser

Linux support for the vmstat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ vmstat | jc --vmstat-s
{"runnable_procs":2,"uninterruptible_sleeping_procs":...timestamp":null,"timezone":null}

ls-s streaming command parser

Linux, macOS, and BSD support for the ls command. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ ls -l /usr/bin | jc --ls-s
{"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
{"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
{"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
...

ping-s streaming command parser

Linux, macOS, and BSD support for the ping and ping6 commands. This is a streaming parser and it outputs JSON Lines. (Documentation):

$ ping 1.1.1.1 | jc --ping-s
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
...

Updated Parsers

  • No updated parsers in this release

Schema Changes

  • No schema changes in this release

Happy parsing!

For more information on the motivations for creating jc, see my blog post.

Published by kellyjonbrazil

I'm a cybersecurity and cloud computing nerd.

Leave a Reply

%d bloggers like this: