JC Version 1.8.0 Released

Posted bykellyjonbrazilPosted inAll Posts, JC NewsTags:, , ,

Try the jc web demo!

I’m excited to announce the release of jc version 1.8.0 available on github and pypi. See below for more information on the new features and parsers.

To upgrade, run:

$ pip3 install --upgrade jc

New Parsers

jc now includes 45 parsers! New parsers (tested on linux and OSX) include blkid, last, lastb, who, /etc/passwd files, /etc/shadow files, /etc/group files, /etc/gshadow files, and CSV files.

Documentation and schemas for all parsers can be found here.

blkid command parser

Linux support for the blkid command:

$ blkid | jc --blkid -p          # or:  jc -p blkid
[
  {
    "device": "/dev/sda1",
    "uuid": "05d927ab-5875-49e4-ada1-7f46cb32c932",
    "type": "xfs"
  },
  {
    "device": "/dev/sda2",
    "uuid": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "type": "LVM2_member"
  },
  {
    "device": "/dev/mapper/centos-root",
    "uuid": "07d718ff-950c-4e5b-98f0-42a1147c77d9",
    "type": "xfs"
  },
  {
    "device": "/dev/mapper/centos-swap",
    "uuid": "615eb89a-bcbf-46fd-80e3-c483ff5c931f",
    "type": "swap"
  }
]

$ sudo blkid -o udev -ip /dev/sda2 | jc --blkid -p          # or:  sudo jc -p blkid -o udev -ip /dev/sda2
[
  {
    "id_fs_uuid": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "id_fs_uuid_enc": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "id_fs_version": "LVM2\x20001",
    "id_fs_type": "LVM2_member",
    "id_fs_usage": "raid",
    "id_iolimit_minimum_io_size": 512,
    "id_iolimit_physical_sector_size": 512,
    "id_iolimit_logical_sector_size": 512,
    "id_part_entry_scheme": "dos",
    "id_part_entry_type": "0x8e",
    "id_part_entry_number": 2,
    "id_part_entry_offset": 2099200,
    "id_part_entry_size": 39843840,
    "id_part_entry_disk": "8:0"
  }
]

last and lastb command parsers

Linux and OSX support for the last command. Linux support for the lastb command.

$ last | jc --last -p          # or:  jc -p last
[
  {
    "user": "joeuser",
    "tty": "ttys002",
    "hostname": null,
    "login": "Thu Feb 27 14:31",
    "logout": "still logged in"
  },
  {
    "user": "joeuser",
    "tty": "ttys003",
    "hostname": null,
    "login": "Thu Feb 27 10:38",
    "logout": "10:38",
    "duration": "00:00"
  },
  {
    "user": "joeuser",
    "tty": "ttys003",
    "hostname": null,
    "login": "Thu Feb 27 10:18",
    "logout": "10:18",
    "duration": "00:00"
  },
  ...
]

$ sudo lastb | jc --last -p          # or:  sudo jc -p lastb
[
  {
    "user": "joeuser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  },
  {
    "user": "joeuser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  },
  {
    "user": "jouser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  }
]

who command parser

Linux and OSX support for the who command:

$ who | jc --who -p          # or:  jc -p who
[
  {
    "user": "joeuser",
    "tty": "ttyS0",
    "time": "2020-03-02 02:52"
  },
  {
    "user": "joeuser",
    "tty": "pts/0",
    "time": "2020-03-02 05:15",
    "from": "192.168.71.1"
  }
]

$ who -a | jc --who -p          # or:  jc -p who -a
[
  {
    "event": "reboot",
    "time": "Feb 7 23:31",
    "pid": 1
  },
  {
    "user": "joeuser",
    "writeable_tty": "-",
    "tty": "console",
    "time": "Feb 7 23:32",
    "idle": "old",
    "pid": 105
  },
  {
    "user": "joeuser",
    "writeable_tty": "+",
    "tty": "ttys000",
    "time": "Feb 13 16:44",
    "idle": ".",
    "pid": 51217,
    "comment": "term=0 exit=0"
  },
  {
    "user": "joeuser",
    "writeable_tty": "?",
    "tty": "ttys003",
    "time": "Feb 28 08:59",
    "idle": "01:36",
    "pid": 41402
  },
  {
    "user": "joeuser",
    "writeable_tty": "+",
    "tty": "ttys004",
    "time": "Mar 1 16:35",
    "idle": ".",
    "pid": 15679,
    "from": "192.168.1.5"
  }
]

CSV File Parser

Convert generic CSV files to JSON. The parser will attempt to automatically detect the delimiter character. If it cannot detect the delimiter it will use the comma (‘,‘) as the delimiter. The file must contain a header row as the first line:

$ cat homes.csv 
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3,  60, 0.28,  3167
175, 180, 18,  8, 4, 1,  12, 0.43,  4033
129, 132, 13,  6, 3, 1,  41, 0.33,  1471
...

$ cat homes.csv | jc --csv -p
[
  {
    "Sell": "142",
    "List": "160",
    "Living": "28",
    "Rooms": "10",
    "Beds": "5",
    "Baths": "3",
    "Age": "60",
    "Acres": "0.28",
    "Taxes": "3167"
  },
  {
    "Sell": "175",
    "List": "180",
    "Living": "18",
    "Rooms": "8",
    "Beds": "4",
    "Baths": "1",
    "Age": "12",
    "Acres": "0.43",
    "Taxes": "4033"
  },
  {
    "Sell": "129",
    "List": "132",
    "Living": "13",
    "Rooms": "6",
    "Beds": "3",
    "Baths": "1",
    "Age": "41",
    "Acres": "0.33",
    "Taxes": "1471"
  },
  ...
]

/etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow file parsers

Convert /etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow files to JSON format:

$ cat /etc/passwd | jc --passwd -p
[
  {
    "username": "nobody",
    "password": "*",
    "uid": -2,
    "gid": -2,
    "comment": "Unprivileged User",
    "home": "/var/empty",
    "shell": "/usr/bin/false"
  },
  {
    "username": "root",
    "password": "*",
    "uid": 0,
    "gid": 0,
    "comment": "System Administrator",
    "home": "/var/root",
    "shell": "/bin/sh"
  },
  {
    "username": "daemon",
    "password": "*",
    "uid": 1,
    "gid": 1,
    "comment": "System Services",
    "home": "/var/root",
    "shell": "/usr/bin/false"
  },
  ...
]

$ sudo cat /etc/shadow | jc --shadow -p
[
  {
    "username": "root",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  {
    "username": "daemon",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  {
    "username": "bin",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  ...
]

$ cat /etc/group | jc --group -p
[
  {
    "group_name": "nobody",
    "password": "*",
    "gid": -2,
    "members": []
  },
  {
    "group_name": "nogroup",
    "password": "*",
    "gid": -1,
    "members": []
  },
  {
    "group_name": "wheel",
    "password": "*",
    "gid": 0,
    "members": [
      "root"
    ]
  },
  {
    "group_name": "certusers",
    "password": "*",
    "gid": 29,
    "members": [
      "root",
      "_jabber",
      "_postfix",
      "_cyrus",
      "_calendar",
      "_dovecot"
    ]
  },
  ...
]

$ cat /etc/gshadow | jc --gshadow -p
[
  {
    "group_name": "root",
    "password": "*",
    "administrators": [],
    "members": []
  },
  {
    "group_name": "adm",
    "password": "*",
    "administrators": [],
    "members": [
      "syslog",
      "joeuser"
    ]
  },
  ...
]

Updated Parsers

  • The ls parser now supports filenames that contain newline characters when using ls -l or ls -b. A warning message will be sent to stderr if newlines are detected and ls -l or ls -b are not used:
$ ls | jc --ls

jc:  Warning - Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.

[{"filename": "this file has"}, {"filename": "a newline inside"}, {"filename": "this file has"}, {"filename": "four contiguous newlines inside"}, ...]
  • The ls parser now supports multiple directory listings, globbing, and recursive listings.
$ ls -R | jc --ls
[{"filename": "centos-7.7"}, {"filename": "create_fixtures.sh"}, {"filename": "generic"}, {"filename": "osx-10.11.6"}, {"filename": "osx-10.14.6"}, ...]

Alternative “Magic” Syntax

jc now accepts a simplified syntax for most command parsers. Instead of piping the data into jc you can now also prepend “jc” to the command you would like to convert. Note that command aliases are not supported:

$ jc dig www.example.com
[{"id": 31113, "opcode": "QUERY", "status": "NOERROR", "flags": ["qr", "rd", "ra"], "query_num": 1, "answer_num": 1, "authority_num": 0, "additional_num": 1, "question": {"name": "www.example.com.", "class": "IN", "type": "A"}, "answer": [{"name": "www.example.com.", "class": "IN", "type": "A", "ttl": 35366, "data": "93.184.216.34"}], "query_time": 37, "server": "2600", "when": "Mon Mar 02 16:13:31 PST 2020", "rcvd": 60}]

You can also insert jc options before the command:

$ jc -pqd dig www.example.com
[
  {
    "id": 7495,
    "opcode": "QUERY",
    "status": "NOERROR",
    "flags": [
      "qr",
      "rd",
      "ra"
    ],
    "query_num": 1,
    "answer_num": 1,
    "authority_num": 0,
    "additional_num": 1,
    "question": {
      "name": "www.example.com.",
      "class": "IN",
      "type": "A"
    },
    "answer": [
      {
        "name": "www.example.com.",
        "class": "IN",
        "type": "A",
        "ttl": 36160,
        "data": "93.184.216.34"
      }
    ],
    "query_time": 40,
    "server": "2600",
    "when": "Mon Mar 02 16:15:21 PST 2020",
    "rcvd": 60
  }
]

Schema Changes

There are no schema changes in this release.

Full Parser List

  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • du
  • env
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • /etc/passwd
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Published by kellyjonbrazil

I'm a cybersecurity and cloud computing nerd.

Leave a Reply

Discover more from Brazil's Blog

Subscribe now to keep reading and get access to the full archive.

Continue reading