See below for v1.17.x updates
I’m excited to announce the release of jc
version 1.17.0 available on github and pypi. This release includes streaming parser support, including three new streaming parsers (ls-s
, ping-s
, and vmstat-s
) and one new standard parser (vmstat
), bringing the total number of parsers to 78.
The streaming parsers output JSON Lines (aka NDJSON), which can be ingested by streaming processors like jq
, elastic
, Splunk, etc. These parsers use significantly less memory while converting large amounts of output (e.g. ls -lR /
), and in some cases can be faster than standard parsers. Just like standard parsers, streaming parsers can be used both at the CLI and as Python libraries. When used as Python libraries, parse()
is a generator function and returns an iterator which can be used in a loop for lazy processing of the stream.
The -u
CLI option has been added to unbuffer the output. This is useful when piping jc
output to another process like jq
and the input stream is very slow (e.g. ping
output). With the unbuffer option enabled you will be able to see the JSON output immediately when using streaming parsers in this scenario instead of waiting for the buffer to be filled.
Streaming parsers also have an ignore_exceptions
option (-qq
on the CLI) to allow uninterrupted processing in case any unexpected parsing errors occur. This can be used for long-lived streams so the pipe will not be broken if there is a hiccup in the stream. When this option is used, a _jc_meta
object with a success
attribute is added to each emitted JSON object. This allows the downstream application to decide whether to ignore the unparsable lines or further process those lines.
jc
can be installed via pip
or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc
, click here.
To upgrade with pip
:
$ pip3 install --upgrade jc
New Features
- Warning and Error messages now wrap to the terminal width.
- Support for streaming parsers for much lower memory consumption when converting large amounts of command output.
-u
CLI option unbuffersjc
output. This is useful when converting slow output likeping
through theping-s
streaming parser.-qq
CLI option makesjc
“extra quiet” for streaming parsers. This equates to theignore_exceptions
argument in the streaming parser’sparse()
function when usingjc
as a Python library.
When using this option, the streaming parser will not stop when parsing errors are encountered. Instead, a_jc_meta
object included in the JSON output will havesuccess
set tofalse
and theerror
andline
attributes will be set to the error message and original line contents, respectively. Here are examples of the additional_jc_meta
object:
Successfully parsed line with -qq
option:
{ "foo": "data1", "bar": "data2", "baz": "data3", "_jc_meta": { "success": true } }
Unsuccessfully parsed line with -qq
option:
{ "_jc_meta": { "success": false, "error": "error message", "line": "original line data" } }
New Parsers
jc
now supports 78 parsers. New parsers include vmstat
and three streaming parsers: ls-s
, ping-s
, and vmstat-s
.
Streaming parsers are considered Beta quality. Even though the streaming parsers have gone through extensive testing, I would like to get more feedback from users before considering them 1.0. Please try them out and provide any feedback as a github issue.
Also, feel free to open a github issue if you have recommendations for other streaming parsers. Currently I’m thinking about adding streaming parsers for CSV, YAML, and XML documents in a future release.
Documentation and schemas for all parsers can be found here.
vmstat
command parser
Linux support for the vmstat
command. (Documentation):
$ vmstat | jc --vmstat -p # or jc -p vmstat [ { "runnable_procs": 2, "uninterruptible_sleeping_procs": 0, "virtual_mem_used": 0, "free_mem": 2794468, "buffer_mem": 2108, "cache_mem": 741208, "inactive_mem": null, "active_mem": null, "swap_in": 0, "swap_out": 0, "blocks_in": 1, "blocks_out": 3, "interrupts": 29, "context_switches": 57, "user_time": 0, "system_time": 0, "idle_time": 99, "io_wait_time": 0, "stolen_time": 0, "timestamp": null, "timezone": null } ]
vmstat-s
streaming command parser
Linux support for the vmstat
command. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ vmstat | jc --vmstat-s {"runnable_procs":2,"uninterruptible_sleeping_procs":...timestamp":null,"timezone":null}
ls-s
streaming command parser
Linux, macOS, and BSD support for the ls
command. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ ls -l /usr/bin | jc --ls-s {"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"} {"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"} {"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"} ...
ping-s
streaming command parser
Linux, macOS, and BSD support for the ping
and ping6
commands. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ ping 1.1.1.1 | jc --ping-s {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703} {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862} {"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82} ...
Updated Parsers
- No updated parsers in this release
Schema Changes
- No schema changes in this release
Happy parsing!
For more information on the motivations for creating
jc
, see my blog post.
v1.17.1 Updates
- Fix
file
parser for gzip files - Fix
uname
parser for cases where the ‘processor’ and/or ‘hardware_platform’ fields are missing on linux - Fix
uname
parser on FreeBSD - Add
lsusb
parser tested on linux - Add CSV file streaming parser
- Add testing for Python 3.10.0
lsusb
command parser
Linux support for the lsusb
command. (Documentation):
$ lsusb -v | jc --lsusb -p # or: jc -p lsusb -v [ { "bus": "002", "device": "001", "id": "1d6b:0001", "description": "Linux Foundation 1.1 root hub", "device_descriptor": { "bLength": { "value": "18" }, "bDescriptorType": { "value": "1" }, "bcdUSB": { "value": "1.10" }, ... "bNumConfigurations": { "value": "1" }, "configuration_descriptor": { "bLength": { "value": "9" }, ... "iConfiguration": { "value": "0" }, "bmAttributes": { "value": "0xe0", "attributes": [ "Self Powered", "Remote Wakeup" ] }, "MaxPower": { "description": "0mA" }, "interface_descriptors": [ { "bLength": { "value": "9" }, ... "bInterfaceProtocol": { "value": "0", "description": "Full speed (or root) hub" }, "iInterface": { "value": "0" }, "endpoint_descriptors": [ { "bLength": { "value": "7" }, ... "bmAttributes": { "value": "3", "attributes": [ "Transfer Type Interrupt", "Synch Type None", "Usage Type Data" ] }, "wMaxPacketSize": { "value": "0x0002", "description": "1x 2 bytes" }, "bInterval": { "value": "255" } } ] } ] } }, "hub_descriptor": { "bLength": { "value": "9" }, ... "wHubCharacteristic": { "value": "0x000a", "attributes": [ "No power switching (usb 1.0)", "Per-port overcurrent protection" ] }, ... "hub_port_status": { "Port 1": { "value": "0000.0103", "attributes": [ "power", "enable", "connect" ] }, "Port 2": { "value": "0000.0103", "attributes": [ "power", "enable", "connect" ] } } }, "device_status": { "value": "0x0001", "description": "Self Powered" } } ]
csv-s
streaming command parser
Support for CSV files. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ cat homes.csv "Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes" 142, 160, 28, 10, 5, 3, 60, 0.28, 3167 175, 180, 18, 8, 4, 1, 12, 0.43, 4033 129, 132, 13, 6, 3, 1, 41, 0.33, 1471 ... $ cat homes.csv | jc --csv-s {"Sell":"142","List":"160","Living":"28","Rooms":"10","Beds":"5","Baths":"3","Age":"60","Acres":"0.28","Taxes":"3167"} {"Sell":"175","List":"180","Living":"18","Rooms":"8","Beds":"4","Baths":"1","Age":"12","Acres":"0.43","Taxes":"4033"} {"Sell":"129","List":"132","Living":"13","Rooms":"6","Beds":"3","Baths":"1","Age":"41","Acres":"0.33","Taxes":"1471"} ...
v1.17.2 Updates
- Fix
ping
parser to add Alpine linux support - Fix
netstat
parser for older versions ofnetstat
on linux - Fix
df
parser for cases where the ‘filesystem’ field overflows the column length
v1.17.3 Updates
- Update parsers to exit with error if non-string input is detected (raise
TypeError
) - Update streaming parsers to exit with error if non-iterable input is detected (raise
TypeError
) - Simplify quiet-checking in parsers
- Add
iostat
parser tested on linux - Add
iostat
streaming parser tested on linux
iostat
command parser
Linux support for the iostat
command. (Documentation):
$ iostat | jc --iostat # or: jc -p iostat [ { "percent_user": 0.15, "percent_nice": 0.0, "percent_system": 0.18, "percent_iowait": 0.0, "percent_steal": 0.0, "percent_idle": 99.67, "type": "cpu" }, { "device": "sda", "tps": 0.29, "kb_read_s": 7.22, "kb_wrtn_s": 1.25, "kb_read": 194341, "kb_wrtn": 33590, "type": "device" }, { "device": "dm-0", "tps": 0.29, "kb_read_s": 5.99, "kb_wrtn_s": 1.17, "kb_read": 161361, "kb_wrtn": 31522, "type": "device" }, { "device": "dm-1", "tps": 0.0, "kb_read_s": 0.08, "kb_wrtn_s": 0.0, "kb_read": 2204, "kb_wrtn": 0, "type": "device" } ]
iostat-s
streaming command parser
Linux support for the iostat
command. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ iostat | jc --iostat-s {"percent_user":0.14,"percent_nice":0.0,"percent_system":0.16,"percent_iowait":0.0,"percent_steal":0.0,"percent_idle":99.7,"type":"cpu"} {"device":"sda","tps":0.24,"kb_read_s":5.28,"kb_wrtn_s":1.1,"kb_read":203305,"kb_wrtn":42368,"type":"device"} ...
v1.17.4 Updates
- Add support for the
NO_COLOR
environment variable to set mono (http://no-color.org/) - Add
-C
option to force color output even when using pipes (overrides-m
andNO_COLOR
)
v1.17.5 Updates
- Add
zipinfo
parser tested on linux and macOS
zipinfo
command parser
Linux and macOS support for the zipinfo
command. (Documentation):
$ zipinfo file.zip | jc --zipinfo -p [ { "archive": "file.zip", "size": 4116, "size_unit": "bytes", "number_entries": 1, "number_files": 1, "bytes_uncompressed": 11837, "bytes_compressed": 3966, "percent_compressed": 66.5, "files": [ { "flags": "-rw-r--r--", "zipversion": "2.1", "zipunder": "unx", "filesize": 11837, "type": "bX", "method": "defN", "date": "21-Dec-08", "time": "20:50", "filename": "compressed_file" } ] } ]
v1.17.6 Updates
- Add
jar-manifest
file parser for MANIFEST.MF files. - Fix CSV parsers for some files that include double-quotes
jar-manifest
file parser
Support for Java JAR Manifest files. (Documentation):
$ cat MANIFEST.MF | jc --jar-manifest -p [ { "Import_Package": "com.conversantmedia.util.concurrent;resolution:=optional,com.fasterxml.jackson.annotation;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core.type;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.cor...", "Export_Package": "org.apache.logging.log4j.core;uses:=\"org.apache.logging.log4j,org.apache.logging.log4j.core.config,org.apache.logging.log4j.core.impl,org.apache.logging.log4j.core.layout,org.apache.logging.log4j.core.time,org.apache.logging.log4j.message,org.apache.logging.log4j.spi,org.apache.logging.log4j.status...", "Manifest_Version": "1.0", "Bundle_License": "https://www.apache.org/licenses/LICENSE-2.0.txt", "Bundle_SymbolicName": "org.apache.logging.log4j.core", "Built_By": "matt", "Bnd_LastModified": "1639373735804", "Implementation_Vendor_Id": "org.apache.logging.log4j", "Specification_Title": "Apache Log4j Core", "Log4jReleaseManager": "Matt Sicker", ... } ]
v1.17.7 Updates
- Add
stat-s
streaming parser for thestat
command.
stat-s
streaming command parser
Linux, macOS, and FreeBSD support for the stat
command. This is a streaming parser and it outputs JSON Lines. (Documentation):
$ stat | jc --stat-s {"file":"(stdin)","unix_device":1027739696,"inode":1155,"flags":"crw--w----","links":1,"user":"kbrazil","group":"tty","rdev":268435456,"size":0,"access_time":"Jan 4 15:27:44 2022","modify_time":"Jan 4 15:27:44 2022","change_time":"Jan 4 15:27:44 2022","birth_time":"Dec 31 16:00:00 1969","block_size":131072,"blocks":0,"unix_flags":"0","access_time_epoch":1641338864,"access_time_epoch_utc":null,"modify_time_epoch":1641338864,"modify_time_epoch_utc":null,"change_time_epoch":1641338864,"change_time_epoch_utc":null,"birth_time_epoch":null,"birth_time_epoch_utc":null}