JC Version 1.17.0 Released

Posted bykellyjonbrazilPosted inAll Posts, JC NewsTags:, , , , , , ,

See below for v1.17.x updates

I’m excited to announce the release of jc version 1.17.0 available on github and pypi. This release includes streaming parser support, including three new streaming parsers (ls-s, ping-s, and vmstat-s) and one new standard parser (vmstat), bringing the total number of parsers to 78.

The streaming parsers output JSON Lines (aka NDJSON), which can be ingested by streaming processors like jq, elastic, Splunk, etc. These parsers use significantly less memory while converting large amounts of output (e.g. ls -lR /), and in some cases can be faster than standard parsers. Just like standard parsers, streaming parsers can be used both at the CLI and as Python libraries. When used as Python libraries, parse() is a generator function and returns an iterator which can be used in a loop for lazy processing of the stream.

The -u CLI option has been added to unbuffer the output. This is useful when piping jc output to another process like jq and the input stream is very slow (e.g. ping output). With the unbuffer option enabled you will be able to see the JSON output immediately when using streaming parsers in this scenario instead of waiting for the buffer to be filled.

Streaming parsers also have an ignore_exceptions option (-qq on the CLI) to allow uninterrupted processing in case any unexpected parsing errors occur. This can be used for long-lived streams so the pipe will not be broken if there is a hiccup in the stream. When this option is used, a _jc_meta object with a success attribute is added to each emitted JSON object. This allows the downstream application to decide whether to ignore the unparsable lines or further process those lines.

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, click here.

To upgrade with pip:

$ pip3 install --upgrade jc
Sections

    New Features

    • Warning and Error messages now wrap to the terminal width.
    • Support for streaming parsers for much lower memory consumption when converting large amounts of command output.
    • -u CLI option unbuffers jc output. This is useful when converting slow output like ping through the ping-s streaming parser.
    • -qq CLI option makes jc “extra quiet” for streaming parsers. This equates to the ignore_exceptions argument in the streaming parser’s parse() function when using jc as a Python library.

      When using this option, the streaming parser will not stop when parsing errors are encountered. Instead, a _jc_meta object included in the JSON output will have success set to false and the error and line attributes will be set to the error message and original line contents, respectively. Here are examples of the additional _jc_meta object:

    Successfully parsed line with -qq option:

    {
  "foo": "data1",
  "bar": "data2",
  "baz": "data3",
  "_jc_meta": {
    "success": true
  }
}

    Unsuccessfully parsed line with -qq option:

    {
  "_jc_meta": {
    "success": false,
    "error": "error message",
    "line": "original line data"
  }
}

    New Parsers

    jc now supports 78 parsers. New parsers include vmstat and three streaming parsers: ls-s, ping-s, and vmstat-s.

    Streaming parsers are considered Beta quality. Even though the streaming parsers have gone through extensive testing, I would like to get more feedback from users before considering them 1.0. Please try them out and provide any feedback as a github issue.

    Also, feel free to open a github issue if you have recommendations for other streaming parsers. Currently I’m thinking about adding streaming parsers for CSV, YAML, and XML documents in a future release.

    Documentation and schemas for all parsers can be found here.

    vmstat command parser

    Linux support for the vmstat command. (Documentation):

    $ vmstat | jc --vmstat -p          # or jc -p vmstat
[
  {
    "runnable_procs": 2,
    "uninterruptible_sleeping_procs": 0,
    "virtual_mem_used": 0,
    "free_mem": 2794468,
    "buffer_mem": 2108,
    "cache_mem": 741208,
    "inactive_mem": null,
    "active_mem": null,
    "swap_in": 0,
    "swap_out": 0,
    "blocks_in": 1,
    "blocks_out": 3,
    "interrupts": 29,
    "context_switches": 57,
    "user_time": 0,
    "system_time": 0,
    "idle_time": 99,
    "io_wait_time": 0,
    "stolen_time": 0,
    "timestamp": null,
    "timezone": null
  }
]

    vmstat-s streaming command parser

    Linux support for the vmstat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ vmstat | jc --vmstat-s
{"runnable_procs":2,"uninterruptible_sleeping_procs":...timestamp":null,"timezone":null}

    ls-s streaming command parser

    Linux, macOS, and BSD support for the ls command. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ ls -l /usr/bin | jc --ls-s
{"filename":"2to3-","flags":"-rwxr-xr-x","links":4,"owner":"root","group":"wheel","size":925,"date":"Feb 22 2019"}
{"filename":"2to3-2.7","link_to":"../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/2to3-2.7","flags":"lrwxr-xr-x","links":1,"owner":"root","group":"wheel","size":74,"date":"May 4 2019"}
{"filename":"AssetCacheLocatorUtil","flags":"-rwxr-xr-x","links":1,"owner":"root","group":"wheel","size":55152,"date":"May 3 2019"}
...

    ping-s streaming command parser

    Linux, macOS, and BSD support for the ping and ping6 commands. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ ping 1.1.1.1 | jc --ping-s
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":0,"ttl":56,"time_ms":23.703}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":1,"ttl":56,"time_ms":22.862}
{"type":"reply","destination_ip":"1.1.1.1","sent_bytes":56,"pattern":null,"response_bytes":64,"response_ip":"1.1.1.1","icmp_seq":2,"ttl":56,"time_ms":22.82}
...

    Updated Parsers

    • No updated parsers in this release

    Schema Changes

    • No schema changes in this release

    Happy parsing!

    For more information on the motivations for creating jc, see my blog post.

    v1.17.1 Updates

    • Fix file parser for gzip files
    • Fix uname parser for cases where the ‘processor’ and/or ‘hardware_platform’ fields are missing on linux
    • Fix uname parser on FreeBSD
    • Add lsusb parser tested on linux
    • Add CSV file streaming parser
    • Add testing for Python 3.10.0

    lsusb command parser

    Linux support for the lsusb command. (Documentation):

    $ lsusb -v | jc --lsusb -p          # or: jc -p lsusb -v
[
  {
    "bus": "002",
    "device": "001",
    "id": "1d6b:0001",
    "description": "Linux Foundation 1.1 root hub",
    "device_descriptor": {
      "bLength": {
        "value": "18"
      },
      "bDescriptorType": {
        "value": "1"
      },
      "bcdUSB": {
        "value": "1.10"
      },
      ...
      "bNumConfigurations": {
        "value": "1"
      },
      "configuration_descriptor": {
        "bLength": {
          "value": "9"
        },
        ...
        "iConfiguration": {
          "value": "0"
        },
        "bmAttributes": {
          "value": "0xe0",
          "attributes": [
            "Self Powered",
            "Remote Wakeup"
          ]
        },
        "MaxPower": {
          "description": "0mA"
        },
        "interface_descriptors": [
          {
            "bLength": {
              "value": "9"
            },
            ...
            "bInterfaceProtocol": {
              "value": "0",
              "description": "Full speed (or root) hub"
            },
            "iInterface": {
              "value": "0"
            },
            "endpoint_descriptors": [
              {
                "bLength": {
                  "value": "7"
                },
                ...
                "bmAttributes": {
                  "value": "3",
                  "attributes": [
                    "Transfer Type  Interrupt",
                    "Synch Type  None",
                    "Usage Type  Data"
                  ]
                },
                "wMaxPacketSize": {
                  "value": "0x0002",
                  "description": "1x 2 bytes"
                },
                "bInterval": {
                  "value": "255"
                }
              }
            ]
          }
        ]
      }
    },
    "hub_descriptor": {
      "bLength": {
        "value": "9"
      },
      ...
      "wHubCharacteristic": {
        "value": "0x000a",
        "attributes": [
          "No power switching (usb 1.0)",
          "Per-port overcurrent protection"
        ]
      },
      ...
      "hub_port_status": {
        "Port 1": {
          "value": "0000.0103",
          "attributes": [
            "power",
            "enable",
            "connect"
          ]
        },
        "Port 2": {
          "value": "0000.0103",
          "attributes": [
            "power",
            "enable",
            "connect"
          ]
        }
      }
    },
    "device_status": {
      "value": "0x0001",
      "description": "Self Powered"
    }
  }
]

    csv-s streaming command parser

    Support for CSV files. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ cat homes.csv
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3,  60, 0.28,  3167
175, 180, 18,  8, 4, 1,  12, 0.43,  4033
129, 132, 13,  6, 3, 1,  41, 0.33,  1471
...

$ cat homes.csv | jc --csv-s
{"Sell":"142","List":"160","Living":"28","Rooms":"10","Beds":"5","Baths":"3","Age":"60","Acres":"0.28","Taxes":"3167"}
{"Sell":"175","List":"180","Living":"18","Rooms":"8","Beds":"4","Baths":"1","Age":"12","Acres":"0.43","Taxes":"4033"}
{"Sell":"129","List":"132","Living":"13","Rooms":"6","Beds":"3","Baths":"1","Age":"41","Acres":"0.33","Taxes":"1471"}
...

    v1.17.2 Updates

    • Fix ping parser to add Alpine linux support
    • Fix netstat parser for older versions of netstat on linux
    • Fix df parser for cases where the ‘filesystem’ field overflows the column length

    v1.17.3 Updates

    • Update parsers to exit with error if non-string input is detected (raise TypeError)
    • Update streaming parsers to exit with error if non-iterable input is detected (raise TypeError)
    • Simplify quiet-checking in parsers
    • Add iostat parser tested on linux
    • Add iostat streaming parser tested on linux

    iostat command parser

    Linux support for the iostat command. (Documentation):

    $ iostat | jc --iostat          # or: jc -p iostat
[
  {
      "percent_user": 0.15,
      "percent_nice": 0.0,
      "percent_system": 0.18,
      "percent_iowait": 0.0,
      "percent_steal": 0.0,
      "percent_idle": 99.67,
      "type": "cpu"
  },
  {
      "device": "sda",
      "tps": 0.29,
      "kb_read_s": 7.22,
      "kb_wrtn_s": 1.25,
      "kb_read": 194341,
      "kb_wrtn": 33590,
      "type": "device"
  },
  {
      "device": "dm-0",
      "tps": 0.29,
      "kb_read_s": 5.99,
      "kb_wrtn_s": 1.17,
      "kb_read": 161361,
      "kb_wrtn": 31522,
      "type": "device"
  },
  {
      "device": "dm-1",
      "tps": 0.0,
      "kb_read_s": 0.08,
      "kb_wrtn_s": 0.0,
      "kb_read": 2204,
      "kb_wrtn": 0,
      "type": "device"
  }
]

    iostat-s streaming command parser

    Linux support for the iostat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ iostat | jc --iostat-s
{"percent_user":0.14,"percent_nice":0.0,"percent_system":0.16,"percent_iowait":0.0,"percent_steal":0.0,"percent_idle":99.7,"type":"cpu"}
{"device":"sda","tps":0.24,"kb_read_s":5.28,"kb_wrtn_s":1.1,"kb_read":203305,"kb_wrtn":42368,"type":"device"}
...

    v1.17.4 Updates

    • Add support for the NO_COLOR environment variable to set mono (http://no-color.org/)
    • Add -C option to force color output even when using pipes (overrides -m and NO_COLOR)

    v1.17.5 Updates

    • Add zipinfo parser tested on linux and macOS

    zipinfo command parser

    Linux and macOS support for the zipinfo command. (Documentation):

    $ zipinfo file.zip | jc --zipinfo -p
[
  {
    "archive": "file.zip",
    "size": 4116,
    "size_unit": "bytes",
    "number_entries": 1,
    "number_files": 1,
    "bytes_uncompressed": 11837,
    "bytes_compressed": 3966,
    "percent_compressed": 66.5,
    "files": [
      {
        "flags": "-rw-r--r--",
        "zipversion": "2.1",
        "zipunder": "unx",
        "filesize": 11837,
        "type": "bX",
        "method": "defN",
        "date": "21-Dec-08",
        "time": "20:50",
        "filename": "compressed_file"
      }
    ]
  }
]

    v1.17.6 Updates

    • Add jar-manifest file parser for MANIFEST.MF files.
    • Fix CSV parsers for some files that include double-quotes

    jar-manifest file parser

    Support for Java JAR Manifest files. (Documentation):

    $ cat MANIFEST.MF | jc --jar-manifest -p
[
  {
    "Import_Package": "com.conversantmedia.util.concurrent;resolution:=optional,com.fasterxml.jackson.annotation;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.core.type;version=\"[2.12,3)\";resolution:=optional,com.fasterxml.jackson.cor...",
    "Export_Package": "org.apache.logging.log4j.core;uses:=\"org.apache.logging.log4j,org.apache.logging.log4j.core.config,org.apache.logging.log4j.core.impl,org.apache.logging.log4j.core.layout,org.apache.logging.log4j.core.time,org.apache.logging.log4j.message,org.apache.logging.log4j.spi,org.apache.logging.log4j.status...",
    "Manifest_Version": "1.0",
    "Bundle_License": "https://www.apache.org/licenses/LICENSE-2.0.txt",
    "Bundle_SymbolicName": "org.apache.logging.log4j.core",
    "Built_By": "matt",
    "Bnd_LastModified": "1639373735804",
    "Implementation_Vendor_Id": "org.apache.logging.log4j",
    "Specification_Title": "Apache Log4j Core",
    "Log4jReleaseManager": "Matt Sicker",
    ...
  }
]

    v1.17.7 Updates

    • Add stat-s streaming parser for the stat command.

    stat-s streaming command parser

    Linux, macOS, and FreeBSD support for the stat command. This is a streaming parser and it outputs JSON Lines. (Documentation):

    $ stat | jc --stat-s
{"file":"(stdin)","unix_device":1027739696,"inode":1155,"flags":"crw--w----","links":1,"user":"kbrazil","group":"tty","rdev":268435456,"size":0,"access_time":"Jan  4 15:27:44 2022","modify_time":"Jan  4 15:27:44 2022","change_time":"Jan  4 15:27:44 2022","birth_time":"Dec 31 16:00:00 1969","block_size":131072,"blocks":0,"unix_flags":"0","access_time_epoch":1641338864,"access_time_epoch_utc":null,"modify_time_epoch":1641338864,"modify_time_epoch_utc":null,"change_time_epoch":1641338864,"change_time_epoch_utc":null,"birth_time_epoch":null,"birth_time_epoch_utc":null}

    Published by kellyjonbrazil

    I'm a cybersecurity and cloud computing nerd.

    Leave a Reply

    %d bloggers like this: