JC Version 1.21.0 Released

I’m excited to announce the release of jc version 1.21.0 available on github and pypi. jc now supports over 100 standard and streaming parsers. Thank you to the Open Source community for making this possible!

jc can be installed via pip or through several official OS package repositories, including Debian, Ubuntu, Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, see the project README.

To upgrade with pip:

$ pip3 install --upgrade jc
Sections

    What’s New

    • New --meta-out or -M option to add metadata to the JSON output, including a UTC timestamp, parser name, magic command, and magic command exit code
    • IP Address string parser
    • Syslog standard and streaming string parsers (RFC 3164 and RFC 5424)
    • CEF standard and streaming string parsers
    • PLIST file parser (XML and binary support)
    • mdadm command parser
    • Add -n support to the traceroute parser
    • Other minor parser fixes

    New Features

    The new --meta-out command option adds a _jc_meta key to the output objects that contains the parser name, a UTC timestamp, and the magic command and exit code information if the magic syntax is used.

    Standard parser output can either be an array of objects (list of dictionaries) or a single object (dictionary). If the output is an array of objects, then each object in the array will have the _jc_meta field added. If the output is a singe object, then the _jc_meta field will be added to that single object. In the case of streaming parsers, discrete objects are emitted for each item. Each object will have a _jc_meta field added.

    Here is an example with the ping parser and the magic syntax.

    $ jc --meta-out -p ping -c3 192.168.1.252
    {
      "destination_ip": "192.168.1.252",
      "data_bytes": 56,
      "pattern": null,
      "destination": "192.168.1.252",
      "packets_transmitted": 3,
      "packets_received": 0,
      "packet_loss_percent": 100.0,
      "duplicates": 0,
      "responses": [
        {
          "type": "timeout",
          "icmp_seq": 0,
          "duplicate": false
        },
        {
          "type": "timeout",
          "icmp_seq": 1,
          "duplicate": false
        }
      ],
      "_jc_meta": {
        "parser": "ping",
        "timestamp": 1661128157.294033,
        "magic_command": [
          "ping",
          "-c3",
          "192.168.1.252"
        ],
        "magic_command_exit": 2
      }
    }

    New Parsers

    IP Address string parser

    Support for IPv4 and IPv6 CIDR strings. (Documentation)

    Standard and decimal IP notation is supported. The output includes subnet information in standard, decimal, hex, and binary notation.

    $ echo 192.168.2.10/24 | jc --ip-address -p
    {
      "version": 4,
      "max_prefix_length": 32,
      "ip": "192.168.2.10",
      "ip_compressed": "192.168.2.10",
      "ip_exploded": "192.168.2.10",
      "scope_id": null,
      "ipv4_mapped": null,
      "six_to_four": null,
      "teredo_client": null,
      "teredo_server": null,
      "dns_ptr": "10.2.168.192.in-addr.arpa",
      "network": "192.168.2.0",
      "broadcast": "192.168.2.255",
      "hostmask": "0.0.0.255",
      "netmask": "255.255.255.0",
      "cidr_netmask": 24,
      "hosts": 254,
      "first_host": "192.168.2.1",
      "last_host": "192.168.2.254",
      "is_multicast": false,
      "is_private": true,
      "is_global": false,
      "is_link_local": false,
      "is_loopback": false,
      "is_reserved": false,
      "is_unspecified": false,
      "int": {
        "ip": 3232236042,
        "network": 3232236032,
        "broadcast": 3232236287,
        "first_host": 3232236033,
        "last_host": 3232236286
      },
      "hex": {
        "ip": "c0:a8:02:0a",
        "network": "c0:a8:02:00",
        "broadcast": "c0:a8:02:ff",
        "hostmask": "00:00:00:ff",
        "netmask": "ff:ff:ff:00",
        "first_host": "c0:a8:02:01",
        "last_host": "c0:a8:02:fe"
      },
      "bin": {
        "ip": "11000000101010000000001000001010",
        "network": "11000000101010000000001000000000",
        "broadcast": "11000000101010000000001011111111",
        "hostmask": "00000000000000000000000011111111",
        "netmask": "11111111111111111111111100000000",
        "first_host": "11000000101010000000001000000001",
        "last_host": "11000000101010000000001011111110"
      }
    }
    
    $ echo 127:0:de::1%128/96 | jc --ip-address -p
    {
      "version": 6,
      "max_prefix_length": 128,
      "ip": "127:0:de::1",
      "ip_compressed": "127:0:de::1%128",
      "ip_exploded": "0127:0000:00de:0000:0000:0000:0000:0001",
      "scope_id": "128",
      "ipv4_mapped": null,
      "six_to_four": null,
      "teredo_client": null,
      "teredo_server": null,
      "dns_ptr": "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.....0.7.2.1.0.ip6.arpa",
      "network": "127:0:de::",
      "broadcast": "127:0:de::ffff:ffff",
      "hostmask": "::ffff:ffff",
      "netmask": "ffff:ffff:ffff:ffff:ffff:ffff::",
      "cidr_netmask": 96,
      "hosts": 4294967294,
      "first_host": "127:0:de::1",
      "last_host": "127:0:de::ffff:fffe",
      "is_multicast": false,
      "is_private": false,
      "is_global": true,
      "is_link_local": false,
      "is_loopback": false,
      "is_reserved": true,
      "is_unspecified": false,
      "int": {
        "ip": 1531727573536155682370944093904699393,
        "network": 1531727573536155682370944093904699392,
        "broadcast": 1531727573536155682370944098199666687,
        "first_host": 1531727573536155682370944093904699393,
        "last_host": 1531727573536155682370944098199666686
      },
      "hex": {
        "ip": "01:27:00:00:00:de:00:00:00:00:00:00:00:00:00:01",
        "network": "01:27:00:00:00:de:00:00:00:00:00:00:00:00:00:00",
        "broadcast": "01:27:00:00:00:de:00:00:00:00:00:00:ff:ff:ff:ff",
        "hostmask": "00:00:00:00:00:00:00:00:00:00:00:00:ff:ff:ff:ff",
        "netmask": "ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:00:00:00:00",
        "first_host": "01:27:00:00:00:de:00:00:00:00:00:00:00:00:00:01",
        "last_host": "01:27:00:00:00:de:00:00:00:00:00:00:ff:ff:ff:fe"
      },
      "bin": {
        "ip": "000000010010011100000000000000000000000011011110000000...",
        "network": "0000000100100111000000000000000000000000110111100...",
        "broadcast": "00000001001001110000000000000000000000001101111...",
        "hostmask": "000000000000000000000000000000000000000000000000...",
        "netmask": "1111111111111111111111111111111111111111111111111...",
        "first_host": "0000000100100111000000000000000000000000110111...",
        "last_host": "00000001001001110000000000000000000000001101111..."
      }
    }

    Syslog string parser (RFC 5424)

    Support for RFC 5424 Syslog strings. Multiple syslog strings separated by newline characters are supported. (Documentation)

    $ echo "<165>1 2003-08-24T05:14:15.000003-07:00 192.0.2.1 myproc 8710 - - %% It's time to make the do-nuts." | jc --syslog -p
    [
      {
        "priority": 165,
        "version": 1,
        "timestamp": "2003-08-24T05:14:15.000003-07:00",
        "hostname": "192.0.2.1",
        "appname": "myproc",
        "proc_id": 8710,
        "msg_id": null,
        "structured_data": null,
        "message": "%% It's time to make the do-nuts.",
        "timestamp_epoch": 1061727255,
        "timestamp_epoch_utc": null
      }
    ]

    Syslog string streaming parser (RFC 5424)

    Support for RFC 5424 Syslog strings. Multiple syslog strings separated by newline characters are supported. This is a streaming parser and it outputs JSON Lines. (Documentation)

    $ cat syslog.txt | jc --syslog-s -p
    {"priority":165,"version":1,"timestamp":"2003-08-24T05:14:15.000003-...}
    {"priority":165,"version":1,"timestamp":"2003-08-24T05:14:16.000003-...}
    ...

    Syslog string parser (BSD-style RFC 3164)

    Support for RFC 3164 Syslog strings. Multiple syslog strings separated by newline characters are supported. (Documentation)

    $ echo "<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8" | jc --syslog-bsd -p
    [
      {
        "priority": 34,
        "date": "Oct 11 22:14:15",
        "hostname": "mymachine",
        "tag": "su",
        "content": "'su root' failed for lonvick on /dev/pts/8"
      }
    ]

    Syslog string streaming parser (BSD-style RFC 3164)

    Support for RFC 3164 Syslog strings. Multiple syslog strings separated by newline characters are supported. This is a streaming parser and it outputs JSON Lines. (Documentation)

    $ cat syslog.txt | jc --syslog-bsd-s -p
    {"priority":34,"date":"Oct 11 22:14:15","hostname":"mymachine","t...}
    {"priority":34,"date":"Oct 11 22:14:16","hostname":"mymachine","t...}
    ...

    CEF string parser

    Support for standard CEF log lines as documented in the Microfocus Arcsight CEF specification. (Documentation)

    $ cat cef.log | jc --cef -p
    [
      {
        "deviceVendor": "Trend Micro",
        "deviceProduct": "Deep Security Agent",
        "deviceVersion": "<DSA version>",
        "deviceEventClassId": "4000000",
        "name": "Eicar_test_file",
        "agentSeverity": 6,
        "CEFVersion": 0,
        "dvchost": "hostname",
        "string": "hello \"world\"!",
        "start": "Nov 08 2020 12:30:00.111 UTC",
        "start_epoch": 1604867400,
        "start_epoch_utc": 1604838600,
        "Host_ID": 1,
        "Quarantine": 205,
        "myDate": "Nov 08 2022 12:30:00.111",
        "myDate_epoch": 1667939400,
        "myDate_epoch_utc": null,
        "myFloat": 3.14,
        "deviceEventClassIdNum": 4000000,
        "agentSeverityString": "Medium",
        "agentSeverityNum": 6
      }
    ]

    CEF string streaming parser

    Support for standard CEF log lines as documented in the Microfocus Arcsight CEF specification. This is a streaming parser and it outputs JSON Lines. (Documentation)

    $ cat cef.log | jc --cef-s
    {"deviceVendor":"Fortinet","deviceProduct":"FortiDeceptor","deviceV...}
    {"deviceVendor":"Trend Micro","deviceProduct":"Deep Security Agent"...}
    ...

    PLIST file parser

    Support for binary and XML PLIST files. (Documentation)

    $ cat info.plist | jc --plist -p
    {
      "NSAppleScriptEnabled": true,
      "LSMultipleInstancesProhibited": true,
      "CFBundleInfoDictionaryVersion": "6.0",
      "DTPlatformVersion": "GM",
      "CFBundleIconFile": "GarageBand.icns",
      "CFBundleName": "GarageBand",
      "DTSDKName": "macosx10.13internal",
      "NSSupportsAutomaticGraphicsSwitching": true,
      "RevisionDate": "2018-12-03_14:10:56",
      "UTImportedTypeDeclarations": [
        {
          "UTTypeConformsTo": [
            "public.data",
            "public.content"
      ...
    }

    mdadm command parser

    Linux support for mdadm command output. The --examine and --query options are supported. (Documentation)

    $ mdadm --query --detail /dev/md0 | jc --mdadm -p
    {
      "device": "/dev/md0",
      "version": "1.1",
      "creation_time": "Tue Apr 13 23:22:16 2010",
      "raid_level": "raid1",
      "array_size": "5860520828 (5.46 TiB 6.00 TB)",
      "used_dev_size": "5860520828 (5.46 TiB 6.00 TB)",
      "raid_devices": 2,
      "total_devices": 2,
      "persistence": "Superblock is persistent",
      "intent_bitmap": "Internal",
      "update_time": "Tue Jul 26 20:16:31 2022",
      "state": "clean",
      "active_devices": 2,
      "working_devices": 2,
      "failed_devices": 0,
      "spare_devices": 0,
      "consistency_policy": "bitmap",
      "name": "virttest:0",
      "uuid": "85c5b164:d58a5ada:14f5fe07:d642e843",
      "events": 2193679,
      "device_table": [
        {
          "number": 3,
          "major": 8,
          "minor": 17,
          "state": [
            "active",
            "sync"
          ],
          "device": "/dev/sdb1",
          "raid_device": 0
        },
        {
          "number": 2,
          "major": 8,
          "minor": 33,
          "state": [
            "active",
            "sync"
          ],
          "device": "/dev/sdc1",
          "raid_device": 1
        }
      ],
      "array_size_num": 5860520828,
      "used_dev_size_num": 5860520828,
      "name_val": "virttest:0",
      "uuid_val": "85c5b164:d58a5ada:14f5fe07:d642e843",
      "state_list": [
        "clean"
      ],
      "creation_time_epoch": 1271226136,
      "update_time_epoch": 1658891791
    }

    Happy parsing!

    Published by kellyjonbrazil

    I'm a cybersecurity and cloud computing nerd.

    Leave a Reply

    %d bloggers like this: