Featured

Parsing Command Output in Saltstack with JC

In my last blog post I demonstrated how we can easily parse remote command output in Ansible. Since then it was requested that I demonstrate something similar using Saltstack.

Saltstack (or Salt, as it is known) is a little different than Ansible in that it primarily uses a pub/sub architecture vs. SSH and requires an agent, or a Minion, to be installed on the remote hosts you are managing.

It turns out it is fairly straightforward to add jc functionality to Saltstack via a custom Output Module and/or Serializer Module. We’ll go over both methods, plus a bonus method in this post.

For more information on the motivations for creating jc, see my blog post.

Output Module

With a Salt Output Module you can restructure the output of the command results that are written to STDOUT on the Master. The default output is typically YAML, but you can change it to JSON or other formats with builtin output modules.

Here is the default YAML output:

# salt '*' cmd.run 'uptime'
minion1:
     16:31:16 up 2 days,  3:04,  1 user,  load average: 0.03, 0.03, 0.00
minion2:
     16:31:16 up 2 days,  3:04,  1 user,  load average: 0.00, 0.00, 0.00

And here is the output using the builtin JSON ouputter:

# salt '*' cmd.run 'uptime' --out=json
{
    "minion2": " 16:33:02 up 2 days,  3:06,  1 user,  load average: 0.00, 0.00, 0.00"
}
{
    "minion1": " 16:33:02 up 2 days,  3:06,  1 user,  load average: 0.00, 0.02, 0.00"
}

But we can do better with jc by turning the uptime output into a JSON object:

# JC_PARSER=uptime salt '*' cmd.run 'uptime' --out=jc --out-indent=2
{
  "minion1": {
    "time": "16:36:04",
    "uptime": "2 days, 3:09",
    "users": 1,
    "load_1m": 0.07,
    "load_5m": 0.02,
    "load_15m": 0.0
  }
}
{
  "minion2": {
    "time": "16:36:04",
    "uptime": "2 days, 3:09",
    "users": 1,
    "load_1m": 0.0,
    "load_5m": 0.0,
    "load_15m": 0.0
  }
}

Now we can pipe this output to jq, jello, or any other JSON filter to more easily consume this data.

We’ll go over the Output Module installation and usage later in this post.

Serializer Module

With a Salt Serializer Module you can restructure the output of the command results during runtime on each Minion so they can be used as objects/variables within a Salt state. For example, If I only cared about the number of users currently logged into each minion and wanted to set that number as a variable for use elsewhere, we could do that with a jc Serializer Module.

Here is a simple, contrived example Salt state file to show how it works:

{% set uptime_out = salt.cmd.shell('uptime') %}
{% set uptime_jc = salt.slsutil.deserialize('jc', uptime_out, parser='uptime') %}

run_uptime:
  cmd.run:
    - name: >
        echo 'The number of users logged in is {{ uptime_jc.users }}'

And here is the output after applying this state file:

# salt '*' state.apply uptime-users
minion1:
----------
          ID: run_uptime
    Function: cmd.run
        Name: echo 'The number of users logged in is 1'

      Result: True
     Comment: Command "echo 'The number of users logged in is 1'
              " run
     Started: 17:01:43.992058
    Duration: 6.107 ms
     Changes:   
              ----------
              pid:
                  23208
              retcode:
                  0
              stderr:
              stdout:
                  The number of users logged in is 1

Summary for minion1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   6.107 ms
minion2:
----------
          ID: run_uptime
    Function: cmd.run
        Name: echo 'The number of users logged in is 2'

      Result: True
     Comment: Command "echo 'The number of users logged in is 2'
              " run
     Started: 17:01:44.005482
    Duration: 6.55 ms
     Changes:   
              ----------
              pid:
                  23371
              retcode:
                  0
              stderr:
              stdout:
                  The number of users logged in is 2

Summary for minion2
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   6.550 ms

Since jc deserialized the command output into an object, we can simply reference the object attributes in our Salt states. We’ll go over installation and usage of the jc Serializer Module later in this post.

Installation and Usage

To use the jc Output Module, you will need to install jc on the Master. To use the jc Serializer Module, you will need to install jc on the Minions. Depending on your use case you may decide to install one or the other or both modules.

Installing jc

You can install jc on the Master and Minions with the following command. Of course, this can also be automated via Salt!

$ pip3 install jc

Installing the Output Module

To install the Output Module on the Master, you need to place the Python module in a directory where the Master is configured to look for it.

First, edit the /etc/salt/master configuration file to configure a custom Module directory. In this example we will use /srv/modules by adding this line to the configuration file:

module_dirs: ["/srv/modules"]

Next we need to create the /srv/modules/output directory, if it doesn’t already exist:

# mkdir -p /srv/modules/output

Next, copy the python module into the directory. I have uploaded the code to Github as a Gist:

# curl https://gist.githubusercontent.com/kellyjonbrazil/24e10f0c3e438ea22fc1e2bfaee22efc/raw/263e4eaf8e51f974b34d44e0483540b163667bdf/jc.py -o /srv/modules/output/jc.py

Finally, restart the Salt Master:

# systemctl restart salt-master

Using the Output Module

To use the jc Output Module, you need to call it with the --out=jc option of the salt command.

Additionally, you need to tell the jc Output Module which parser to use. To do this, you can set the JC_PARSER environment variable inline with the command:

# JC_PARSER=date salt '*' cmd.run 'date' --out=jc
{"minion2": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}}
{"minion1": {"year": 2020, "month_num": 9, "day": 15, "hour": 18, "minute": 27, "second": 11, "month": "Sep", "weekday": "Tue", "weekday_num": 3, "timezone": "UTC"}}

For a list of jc parsers, see the parser documentation.

Additionally, you can add the --out-indent option to pretty-print the output:

# JC_PARSER=date salt '*' cmd.run 'date' --out=jc --out-indent=2
{
  "minion2": {
    "year": 2020,
    "month_num": 9,
    "day": 15,
    "hour": 18,
    "minute": 29,
    "second": 8,
    "month": "Sep",
    "weekday": "Tue",
    "weekday_num": 3,
    "timezone": "UTC"
  }
}
{
  "minion1": {
    "year": 2020,
    "month_num": 9,
    "day": 15,
    "hour": 18,
    "minute": 29,
    "second": 8,
    "month": "Sep",
    "weekday": "Tue",
    "weekday_num": 3,
    "timezone": "UTC"
  }
}

Installing the Serializer Module

To install the Serializer Module on the Minions, you can copy the Python module to the _serializers folder within your Salt fileserver directory on the Master (typically /srv/salt) and sync to the Minions.

First, create the /srv/salt/_serializers directory if it doesn’t already exist:

# mkdir -p /srv/salt/_serializers

Next, copy the Python module into the _serializers directory on the Master. I have uploaded the code to Github as a Gist:

# curl https://gist.githubusercontent.com/kellyjonbrazil/7d67cfa003735bf80ef43fe5652950dd/raw/1541a7d327aed0366ccfea91bd0533032111d11c/jc.py -o /srv/salt/_serializers/jc.py

Finally, sync the jc Serializer Module to the Minions:

# salt '*' saltutil.sync_all

Using the Serializer Module

To use the jc Serializer Module, invoke it with the salt.slsutil.deserialize() function within a Salt state file. The function requires three arguments to deserialize with jc:

  • Argument 1: 'jc'
    • This should always be the literal string 'jc' to call the jc Serializer Module
  • Argument 2: String data to be parsed
    • This is the STDOUT string output of the command you want to deserialize
  • Argument 3: parser='<parser>'
    • <parser> is the jc parser you want to use to parse the command output. For example, to use the ifconfig parser, Argument 3 would look like this: parser='ifconfig'. For a list of jc parsers, see the parser documentation.

For example, via Jinja2 template:

{% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %}

Then you can reference any attribute of the date object (Python dictionary) in any other part of the Salt state file. Here is a full example:

{% set date_stdout = salt.cmd.shell('date') %}
{% set date = salt.slsutil.deserialize('jc', date_stdout, parser='date') %}

run_date:
  cmd.run:
    - name: >
        echo 'The timezone is {{ date.timezone }}'

One More Thing

It is also possible to deserialize command output into objects using jc without using the jc Serializer Module. If jc is installed on the Minion, then you can pipe the command output to jc as you would normally do on the command line, then use the buit-in JSON Serializer Module to deserialize the jc JSON output into Python objects:

{% set date = salt.slsutil.deserialize('json', salt.cmd.shell('date | jc --date')) %}

run_date:
  cmd.run:
    - name: >
        echo 'The timezone is {{ date.timezone }}'

Happy parsing!

Featured

Parsing Command Output in Ansible with JC

Ansible is a popular automation framework that allows you to configure any number of remote hosts in a declarative and idempotent way. A common use-case is to run a shell command on the remote host, return the STDOUT output, loop through it and parse it.

Starting in Ansible 2.9 with the community.general collection, it is possible to use jc (https://github.com/kellyjonbrazil/jc) as a filter to automatically parse the command output for you so you can easily use the output as an object.

For more information on the motivations for creating jc, see my blog post.

Installation

To use the jc filter plugin, you just need to install jc and the community.general collection on the Ansible controller. Ansible version 2.9 or higher is required to install the community.general collection.

Installing jc:

$ pip3 install jc

Installing the community.general Ansible collection:

$ ansible-galaxy collection install community.general

Now we are ready to use the jc filter plugin!

Syntax

To use the jc filter plugin you just need to pipe the command output to the plugin and specify the parser as an argument. For example, this is how you would parse the output of ps on the remote host:

  tasks:
  - shell: ps aux
    register: result
  - set_fact:
      myvar: "{{ result.stdout | community.general.jc('ps') }}"

This will generate a myvar object that includes the exact same information you would have received by running jc ps aux on the remote host. Now you can use object notation to pull out the information you are interested in.

A Simple Example

Let’s put it all together with a very simple example. In this example we will run the date command on the remote host and print the timezone as a debug message:

- name: Get Timezone
  hosts: ubuntu
  tasks:
  - shell: date
    register: result
  - set_fact:
      myvar: "{{ result.stdout | community.general.jc('date') }}"
  - debug:
      msg: "The timezone is: {{ myvar.timezone }}"

Instead of parsing the STDOUT text manually, we used the timezone attribute of the myvar object that jc gave us. Let’s see this in action:

$ ansible-playbook get-timezone.yml 

PLAY [Get Timezone] *****************************************************************************

TASK [Gathering Facts] **************************************************************************
ok: [192.168.1.239]

TASK [shell] ************************************************************************************
changed: [192.168.1.239]

TASK [set_fact] *********************************************************************************
ok: [192.168.1.239]

TASK [debug] ************************************************************************************
ok: [192.168.1.239] => {
    "msg": "The timezone is: UTC"
}

PLAY RECAP **************************************************************************************
192.168.1.239              : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Simple – no more need to grep/awk/sed your way through the output to get what you are looking for!

For a complete list of jc parsers available and their associated schemas, see the parser documentation.

Happy parsing!

Featured

JC Version 1.13.1 Released

I’m happy to announce the release of jc version 1.13.1 available on github and pypi.

jc now supports over 55 commands and file-types, including the new ping, sysctl, traceroute, and tracepath command parsers. The INI file parser has been enhanced to support simple key/value text files and the route command parser now supports IPv6 tables.

Custom local parser plugins are now supported. This allows overriding existing parsers and rapid development of new parsers.

Other updates include verbose debugging, more consistent handling of empty data, and many parser fixes for FreeBSD.

jc can be installed via pip or through several new official OS package repositories, including Fedora, openSUSE, Arch Linux, NixOS Linux, Guix System Linux, FreeBSD, and macOS. For more information on how to get jc, click here.

To upgrade with pip:

$ pip3 install --upgrade jc

New Features

  • jc is now available on the official Fedora repository (dnf install jc)
  • jc is now available on the official Arch Linux repository (pacman -S jc)
  • jc is now available on the official NixOS repository (nix-env -iA nixpkgs.jc)
  • jc is now available on the official Guix System Linux repository (guix install jc)
  • jc is now available on the official FreeBSD ports repository (portsnap fetch update && cd /usr/ports/textproc/py-jc && make install clean)
  • jc is in process (Intent To Package) for Debian packaging.
  • Local custom parser plugins allow you to override packaged parsers or rapidly create your own.
  • Verbose debugging is now supported with the -dd command argument.
  • All parsers now correctly return empty objects when sent empty data.
  • Older versions of the pygments library (>=2.3.0) are now supported (for Debian packaging)

New Parsers

jc now supports 55 parsers. New parsers include ping, sysctl, tracepath, and traceroute.

Documentation and schemas for all parsers can be found here.

ping command parser

Linux, macOS, and FreeBSD support for the ping command:

$ ping 8.8.8.8 -c 3 | jc --ping -p          # or:  jc -p ping 8.8.8.8 -c 3
{
  "destination_ip": "8.8.8.8",
  "data_bytes": 56,
  "pattern": null,
  "destination": "8.8.8.8",
  "packets_transmitted": 3,
  "packets_received": 3,
  "packet_loss_percent": 0.0,
  "duplicates": 0,
  "time_ms": 2005.0,
  "round_trip_ms_min": 23.835,
  "round_trip_ms_avg": 30.46,
  "round_trip_ms_max": 34.838,
  "round_trip_ms_stddev": 4.766,
  "responses": [
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 1,
      "ttl": 118,
      "time_ms": 23.8,
      "duplicate": false
    },
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 2,
      "ttl": 118,
      "time_ms": 34.8,
      "duplicate": false
    },
    {
      "type": "reply",
      "timestamp": null,
      "bytes": 64,
      "response_ip": "8.8.8.8",
      "icmp_seq": 3,
      "ttl": 118,
      "time_ms": 32.7,
      "duplicate": false
    }
  ]
}

sysctl command parser

Linux, macOS, and FreeBSD support for the sysctl -a command:

$ sysctl -a | jc --sysctl -p          # or:  jc -p sysctl -a
{
  "user.cs_path": "/usr/bin:/bin:/usr/sbin:/sbin",
  "user.bc_base_max": 99,
  "user.bc_dim_max": 2048,
  "user.bc_scale_max": 99,
  "user.bc_string_max": 1000,
  "user.coll_weights_max": 2,
  "user.expr_nest_max": 32
  ...
}

tracepath command parser

Linux support for the tracepath command:

$ tracepath6 3ffe:2400:0:109::2 | jc --tracepath -p
{
  "pmtu": 1480,
  "forward_hops": 2,
  "return_hops": 2,
  "hops": [
    {
      "ttl": 1,
      "guess": true,
      "host": "[LOCALHOST]",
      "reply_ms": null,
      "pmtu": 1500,
      "asymmetric_difference": null,
      "reached": false
    },
    {
      "ttl": 1,
      "guess": false,
      "host": "dust.inr.ac.ru",
      "reply_ms": 0.411,
      "pmtu": null,
      "asymmetric_difference": null,
      "reached": false
    },
    {
      "ttl": 2,
      "guess": false,
      "host": "dust.inr.ac.ru",
      "reply_ms": 0.39,
      "pmtu": 1480,
      "asymmetric_difference": 1,
      "reached": false
    },
    {
      "ttl": 2,
      "guess": false,
      "host": "3ffe:2400:0:109::2",
      "reply_ms": 463.514,
      "pmtu": null,
      "asymmetric_difference": null,
      "reached": true
    }
  ]
}

traceroute command parser

Linux, macOS, and FreeBSD support for the traceroute command:

$ traceroute -m 3 8.8.8.8 | jc --traceroute -p          # or:  jc -p traceroute -m 3 8.8.8.8
{
  "destination_ip": "8.8.8.8",
  "destination_name": "8.8.8.8",
  "hops": [
    {
      "hop": 1,
      "probes": [
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.616
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.413
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "192.168.1.254",
          "name": "dsldevice.local.net",
          "rtt": 6.308
        }
      ]
    },
    {
      "hop": 2,
      "probes": [
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 29.367
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 40.197
        },
        {
          "annotation": null,
          "asn": null,
          "ip": "76.220.24.1",
          "name": "76-220-24-1.lightspeed.sntcca.sbcglobal.net",
          "rtt": 29.162
        }
      ]
    },
    {
      "hop": 3,
      "probes": [
        {
          "annotation": null,
          "asn": null,
          "ip": null,
          "name": null,
          "rtt": null
        }
      ]
    }
  ]
}

Updated Parsers

There have been many parser updates since v1.11.0. The INI file parser has been enhanced to support files and output that contains simple key/value pairs. The route command parser has been enhanced to add support for IPv6 routing tables. The uname parser provides more intuitive debug messages and an issue in the iptables command parser was fixed, allowing it to convert the last row of a table. Many other parser enhancements including the consistent handling of blank input, FreeBSD support, and minor field additions and fixes are included.

Key/Value Pair Files with the INI File Parser

The INI file parser has been enhanced to now support files containing simple key/value pairs. Files can include comments prepended with # or ; and keys and values can be delimited by = or : with or without spaces. Quotation marks are stripped from quoted values, though they can be kept with the -r (raw output) jc argument.

These types of files can be found in many places, including configuration files in /etc. (e.g. /etc/sysconfig/network-scripts).

$ cat keyvalue.txt
# this file contains key/value pairs
name = John Doe
address=555 California Drive
age: 34
; comments can include # or ;
# delimiter can be = or :
# quoted values have quotation marks stripped by default
# but can be preserved with the -r argument
occupation:"Engineer"

$ cat keyvalue.txt | jc --ini -p
{
  "name": "John Doe",
  "address": "555 California Drive",
  "age": "34",
  "occupation": "Engineer"
}

route Command Parser

The route command parser has been enhanced to support IPv6 tables.

$ route -6 | jc --route -p          # or: jc -p route -6
[
  {
    "destination": "[::]/96",
    "next_hop": "[::]",
    "flags": "!n",
    "metric": 1024,
    "ref": 0,
    "use": 0,
    "iface": "lo",
    "flags_pretty": [
      "REJECT"
    ]
  },
  {
    "destination": "0.0.0.0/96",
    "next_hop": "[::]",
    "flags": "!n",
    "metric": 1024,
    "ref": 0,
    "use": 0,
    "iface": "lo",
    "flags_pretty": [
      "REJECT"
    ]
  },
  {
    "destination": "2002:a00::/24",
    "next_hop": "[::]",
    "flags": "!n",
    "metric": 1024,
    "ref": 0,
    "use": 0,
    "iface": "lo",
    "flags_pretty": [
      "REJECT"
    ]
  },
  ...
]

Schema Changes

There are no schema changes in this release.

Full Parser List

  • airport -I
  • airport -s
  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • dmidecode
  • du
  • env
  • file
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • ntpq
  • /etc/passwd
  • ping
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • sysctl
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • timedatectl
  • tracepath
  • traceroute
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

More Comprehensive Tracebacks in Python

Python has great exception-handling with nice traceback messages that can help debug issues with your code. Here’s an example of a typical traceback message:

Traceback (most recent call last):
  File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module>
    load_entry_point('jc', 'console_scripts', 'jc')()
  File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main
    result = parser.parse(data, raw=raw, quiet=quiet)
  File "/Users/kbrazil/git/jc/jc/parsers/uname.py", line 108, in parse
    raw_output['kernel_release'] = parsed_line.pop(0)
IndexError: pop from empty list

I usually read these from the bottom-up to zero-in on the issue. Here I can see that my program is trying to pop the last item off a list called parsed_line, but the list is empty and Python doesn’t know what to do, so it quits with an IndexError exception.

The traceback conveniently includes the line number and snippet of the offending code. This is usually correct, but the line numbering can be off depending on the type of error or exception. This might be enough information for me to dig into the code and figure out why parsed_line is empty. But what about a more complex example?

Traceback (most recent call last):
  File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module>
    load_entry_point('jc', 'console_scripts', 'jc')()
  File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main
    result = parser.parse(data, raw=raw, quiet=quiet)
  File "/Users/kbrazil/git/jc/jc/parsers/arp.py", line 226, in parse
    'hwtype': line[4].lstrip('[').rstrip(']'),
IndexError: list index out of range

In this traceback I can see that the program is trying to pull the fifth item from the line list but Python can’t grab it – probably because the list doesn’t have that many items. This traceback doesn’t show me the state of the variables, so I can’t tell what input the function took or what the line variable looks like when causing this issue.

What I’d really like is to see more context (the code lines before and after the error) along with the variable state when the error occurred. Many times this is done with a debugger or with print() statements. But there is another way!

cgitb (deprecated)

Back in 1995 when CGI scripts were all the rage, Python added the cgi library along with its helper module, cgitb. This module would print more verbose traceback messages to the browser to help with troubleshooting. Conveniently, its traceback messages would include surrounding code context and variable state! cgitb is poorly named since it can drop-in replace standard tracebacks on any type of program. The name might be why it never really gained traction. Unfortunately, cgitb is set to be deprecated in Python 3.10, but let’s see how it works and then check out how to replace it:

IndexError
Python 3.7.6: /usr/local/opt/python/bin/python3.7
Mon Jul  6 12:09:08 2020

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /Users/kbrazil/Library/Python/3.7/bin/jc in <module>()
    2 # EASY-INSTALL-ENTRY-SCRIPT: 'jc','console_scripts','jc'
    3 __requires__ = 'jc'
    4 import re
    5 import sys
    6 from pkg_resources import load_entry_point
    7 
    8 if __name__ == '__main__':
    9     sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
   10     sys.exit(
   11         load_entry_point('jc', 'console_scripts', 'jc')()
   12     )
load_entry_point = <function load_entry_point>

 /Users/kbrazil/git/jc/jc/cli.py in main()
  391 
  392         if parser_name in parsers:
  393             # load parser module just in time so we don't need to load all modules
  394             parser = parser_module(arg)
  395             try:
  396                 result = parser.parse(data, raw=raw, quiet=quiet)
  397                 found = True
  398                 break
  399 
  400             except Exception:
  401                 if debug:
result undefined
parser = <module 'jc.parsers.arp' from '/Users/kbrazil/git/jc/jc/parsers/arp.py'>
parser.parse = <function parse>
data = "#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n"
raw = False
quiet = False

 /Users/kbrazil/git/jc/jc/parsers/arp.py in parse(data="#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n", raw=False, quiet=False)
  221             for line in cleandata:
  222                 line = line.split()
  223                 output_line = {
  224                     'name': line[0],
  225                     'address': line[1].lstrip('(').rstrip(')'),
  226                     'hwtype': line[4].lstrip('[').rstrip(']'),
  227                     'hwaddress': line[3],
  228                     'iface': line[6],
  229                 }
  230                 raw_output.append(output_line)
  231 
line = ['#!/usr/bin/env', 'python3']
].lstrip undefined
IndexError: list index out of range
    __cause__ = None
    __class__ = <class 'IndexError'>
    __context__ = None
    __delattr__ = <method-wrapper '__delattr__' of IndexError object>
    __dict__ = {}
    __dir__ = <built-in method __dir__ of IndexError object>
    __doc__ = 'Sequence index out of range.'
    __eq__ = <method-wrapper '__eq__' of IndexError object>
    __format__ = <built-in method __format__ of IndexError object>
    __ge__ = <method-wrapper '__ge__' of IndexError object>
    __getattribute__ = <method-wrapper '__getattribute__' of IndexError object>
    __gt__ = <method-wrapper '__gt__' of IndexError object>
    __hash__ = <method-wrapper '__hash__' of IndexError object>
    __init__ = <method-wrapper '__init__' of IndexError object>
    __init_subclass__ = <built-in method __init_subclass__ of type object>
    __le__ = <method-wrapper '__le__' of IndexError object>
    __lt__ = <method-wrapper '__lt__' of IndexError object>
    __ne__ = <method-wrapper '__ne__' of IndexError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of IndexError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of IndexError object>
    __repr__ = <method-wrapper '__repr__' of IndexError object>
    __setattr__ = <method-wrapper '__setattr__' of IndexError object>
    __setstate__ = <built-in method __setstate__ of IndexError object>
    __sizeof__ = <built-in method __sizeof__ of IndexError object>
    __str__ = <method-wrapper '__str__' of IndexError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __suppress_context__ = False
    __traceback__ = <traceback object>
    args = ('list index out of range',)
    with_traceback = <built-in method with_traceback of IndexError object>

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "/Users/kbrazil/Library/Python/3.7/bin/jc", line 11, in <module>
    load_entry_point('jc', 'console_scripts', 'jc')()
  File "/Users/kbrazil/git/jc/jc/cli.py", line 396, in main
    result = parser.parse(data, raw=raw, quiet=quiet)
  File "/Users/kbrazil/git/jc/jc/parsers/arp.py", line 226, in parse
    'hwtype': line[4].lstrip('[').rstrip(']'),
IndexError: list index out of range

This verbose traceback gives me just what I’m looking for! Though the default is 5, I told cgitb to print out 11 lines of context. Now I can see the two variables I’m particularly interested in to troubleshoot this issue: data and line.

data = "#!/usr/bin/env python3\n\nimport jc.parsers.ls\nimp...print(tabulate.tabulate(parsed, headers='keys'))\n"

(Notice how it snips the value if it’s too long. Pretty cool!)

line = ['#!/usr/bin/env', 'python3']

Now I can easily see that the data that was input into the function does not look like the type of data expected at all. (it is expecting text output from the arp command and instead it was fed in another Python script file) I can also see that the line list only has two items.

I included cgitb in jc to provide a verbose debug command option (-dd) to help speed up troubleshooting of parsing issues – typically during development of a new parser or to quickly identify an issue a user is having over email. It seemed perfect for my needs and aside from the weird name it worked well.

Then I noticed that cgitb was to be deprecated along with the cgi module with no replacement.

tracebackplus

I decided to vendorize the builtin cgitb library so it wouldn’t be orphaned in later versions of Python. After looking at the code I found it would be pretty easy to simplify the module by taking out all of the HTML rendering cruft. And why not rename it to something more descriptive while we’re at it? After not too much thought, I settled on tracebackplus.

Like cgitb, tracebackplus doesn’t require any external libraries and can easily replace standard tracebacks with the following code:

import tracebackplus
tracebackplus.enable(context=11)

Here is the code for tracebackplus along with the permissive MIT license. Feel free to use this code in your projects.

Here’s an example of how it is being used in jc to provide different levels of debugging using the -d (standard traceback) or -dd (tracebackplus) command line arguments:

try:
    result = parser.parse(data, raw=raw, quiet=quiet)
    found = True
    break

except Exception:
    if debug:
        if verbose_debug:
            import jc.tracebackplus
            jc.tracebackplus.enable(context=11)

        raise

    else:
        import jc.utils
        jc.utils.error_message(
            f'{parser_name} parser could not parse the input data. Did you use the correct parser?\n'
            '                 For details use the -d or -dd option.')
        sys.exit(1)

Happy debugging!

Featured

JC Version 1.11.1 Released

I’m happy to announce the release of jc version 1.11.1 available on github and pypi.

jc now supports over 50 commands and file-types and now can be installed via Homebrew (macOS) and zypper (OpenSUSE). In addition, jc can now be installed via DEB and RPM packages or run as a single binary on linux or macOS. You can set your own custom colors for jc to display and more command parsers are supported on macOS. See below for more information on the new features.

To upgrade, run:

$ pip3 install --upgrade jc

RPM/DEB packages and Binaries can also be found here.

OS package repositories (e.g. brew, zypper, etc.) will be updated with the latest version of jc on their own future release schedules.

New Features

  • jc now supports custom colors. You can customize the colors by setting the JC_COLORS environment variable.
  • jc is now available on macOS via Homebrew (brew install jc)
  • jc is now available on OpenSUSE via zypper
  • DEB, RPM, and Binary packages are now available for linux and macOS
  • Several back-end updates to support packaging on standard linux distribution package repositories in the future (e.g. Fedora)

New Parsers

jc now supports 51 parsers. The dmidecode command is now supported for linux platforms.

Documentation and schemas for all parsers can be found here.

dmidecode command parser

Linux support for the dmidecode command:

# jc -p dmidecode
[
  {
    "handle": "0x0000",
    "type": 0,
    "bytes": 24,
    "description": "BIOS Information",
    "values": {
      "vendor": "Phoenix Technologies LTD",
      "version": "6.00",
      "release_date": "04/13/2018",
      "address": "0xEA490",
      "runtime_size": "88944 bytes",
      "rom_size": "64 kB",
      "characteristics": [
        "ISA is supported",
        "PCI is supported",
        "PC Card (PCMCIA) is supported",
        "PNP is supported",
        "APM is supported",
        "BIOS is upgradeable",
        "BIOS shadowing is allowed",
        "ESCD support is available",
        "Boot from CD is supported",
        "Selectable boot is supported",
        "EDD is supported",
        "Print screen service is supported (int 5h)",
        "8042 keyboard services are supported (int 9h)",
        "Serial services are supported (int 14h)",
        "Printer services are supported (int 17h)",
        "CGA/mono video services are supported (int 10h)",
        "ACPI is supported",
        "Smart battery is supported",
        "BIOS boot specification is supported",
        "Function key-initiated network boot is supported",
        "Targeted content distribution is supported"
      ],
      "bios_revision": "4.6",
      "firmware_revision": "0.0"
    }
  },
  ...
]

Updated Parsers

The netstat command is now supported on macOS:

$ jc -p netstat
[
  {
    "proto": "tcp4",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "mylaptop.local",
    "foreign_address": "173.199.15.254",
    "state": "SYN_SENT   ",
    "kind": "network",
    "local_port": "57561",
    "foreign_port": "https",
    "transport_protocol": "tcp",
    "network_protocol": "ipv4",
    "local_port_num": 57561
  },
  {
    "proto": "tcp4",
    "recv_q": 0,
    "send_q": 0,
    "local_address": "mylaptop.local",
    "foreign_address": "192.0.71.3",
    "state": "ESTABLISHED",
    "kind": "network",
    "local_port": "57525",
    "foreign_port": "https",
    "transport_protocol": "tcp",
    "network_protocol": "ipv4",
    "local_port_num": 57525
  },
  ...
]

The netstat parser has been enhanced to support the -r (routes) and -i (interfaces) options on both linux and macOS.

$ jc -p netstat -r
[
  {
    "destination": "default",
    "gateway": "router.local",
    "route_flags": "UGSc",
    "route_refs": 102,
    "use": 24,
    "iface": "en0",
    "kind": "route"
  },
  {
    "destination": "127",
    "gateway": "localhost",
    "route_flags": "UCS",
    "route_refs": 0,
    "use": 0,
    "iface": "lo0",
    "kind": "route"
  },
  ...
]
$ jc -p netstat -i
[
  {
    "iface": "lo0",
    "mtu": 16384,
    "network": "<Link#1>",
    "address": null,
    "ipkts": 1777797,
    "ierrs": 0,
    "opkts": 1777797,
    "oerrs": 0,
    "coll": 0,
    "kind": "interface"
  },
  {
    "iface": "lo0",
    "mtu": 16384,
    "network": "127",
    "address": "localhost",
    "ipkts": 1777797,
    "ierrs": null,
    "opkts": 1777797,
    "oerrs": null,
    "coll": null,
    "kind": "interface"
  },
  {
    "iface": "lo0",
    "mtu": 16384,
    "network": "localhost",
    "address": "::1",
    "ipkts": 1777797,
    "ierrs": null,
    "opkts": 1777797,
    "oerrs": null,
    "coll": null,
    "kind": "interface"
  },
  ...
]

The stat command is now supported on macOS.

$ jc -p stat jc*
[
  {
    "file": "jc-1.11.1-linux.sha256",
    "device": "16778221",
    "inode": 82163627,
    "flags": "-rw-r--r--",
    "links": 1,
    "user": "joeuser",
    "group": "staff",
    "rdev": 0,
    "size": 69,
    "access_time": "May 26 08:27:44 2020",
    "modify_time": "May 24 18:47:25 2020",
    "change_time": "May 24 18:51:21 2020",
    "birth_time": "May 24 18:47:25 2020",
    "block_size": 4096,
    "blocks": 8,
    "osx_flags": "0"
  },
  {
    "file": "jc-1.11.1-linux.tar.gz",
    "device": "16778221",
    "inode": 82163628,
    "flags": "-rw-r--r--",
    "links": 1,
    "user": "joeuser",
    "group": "staff",
    "rdev": 0,
    "size": 20226936,
    "access_time": "May 26 08:27:44 2020",
    "modify_time": "May 24 18:47:25 2020",
    "change_time": "May 24 18:47:25 2020",
    "birth_time": "May 24 18:47:25 2020",
    "block_size": 4096,
    "blocks": 39512,
    "osx_flags": "0"
  },
  ...
]

Schema Changes

There are no schema changes in this release.

Full Parser List

  • airport -I
  • airport -s
  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • dmidecode
  • du
  • env
  • file
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • ntpq
  • /etc/passwd
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • timedatectl
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

JC Version 1.10.2 Released

I’m happy to announce the release of jc version 1.10.2 available on github and pypi. See below for more information on the new features.

To upgrade, run:

$ pip3 install --upgrade jc

New Features

jc now supports color output by default when printing to the terminal. Color is automatically disabled when piping to another program. The -m (monochrome) option can be used to disable color output to the terminal.

New Parsers

No new parsers in this release.

Updated Parsers

  • file command parser: minor fix for some edge cases
  • arp command parser: fix macOS detection for some edge cases
  • dig command parser: add axfr support

Schema Changes

The dig command parser now supports the axfr option. The schema has been updated to add this section:

$ jc -p dig @81.4.108.41 axfr zonetransfer.me
[
  {
    "axfr": [
      {
        "name": "zonetransfer.me.",
        "ttl": 7200,
        "class": "IN",
        "type": "SOA",
        "data": "nsztm1.digi.ninja. robin.digi.ninja. 2019100801 172800 900 1209600 3600"
      },
      {
        "name": "zonetransfer.me.",
        "ttl": 300,
        "class": "IN",
        "type": "HINFO",
        "data": "\"Casio fx-700G\" \"Windows XP\""
      },
      {
        "name": "zonetransfer.me.",
        "ttl": 301,
        "class": "IN",
        "type": "TXT",
        "data": "\"google-site-verification=tyP28J7JAUHA9fw2sHXMgcCC0I6XBmmoVi04VlMewxA\""
      },
      ...
    ],
    "query_time": 805,
    "server": "81.4.108.41#53(81.4.108.41)",
    "when": "Thu Apr 09 08:05:31 PDT 2020",
    "size": "50 records (messages 1, bytes 1994)"
  }
]

Full Parser List

  • airport -I
  • airport -s
  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • du
  • env
  • file
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • ntpq
  • /etc/passwd
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • timedatectl
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

Jello: The JQ Alternative for Pythonistas

Try the jello web demo!

I’m a big fan of using structured data at the command line. So much so that I’ve written a couple of utilities to promote JSON in the CLI:

Typically I use jq to filter and process the JSON output into submission until I get what I want. But if you’re anything like me, you spend a lot of time googling how to do what you want in jq because the syntax can get a little out of hand. In fact, I keep notes with example jq queries I’ve used before in case I need those techniques again.

jq is great for simple things, but sometimes when I want to iterate through a deeply nested structure with arrays of objects I find python’s list and dictionary syntax easier to comprehend.

Hello jello

That’s why I created jello. jello works similarly to jq but uses the python interpreter, so you can iterate with loops, comprehensions, variables, expressions, etc. just like you would in a full-fledged python script.

The nice thing about jello is that it removes a lot of the boilerplate code you would need to ingest and output the JSON or JSON Lines data so you can focus on the logic.

Let’s take the following output from jc -ap:

$ jc -ap
{
  "name": "jc",
  "version": "1.9.2",
  "description": "jc cli output JSON conversion tool",
  "author": "Kelly Brazil",
  "author_email": "kellyjonbrazil@gmail.com",
  "parser_count": 50,
  "parsers": [
    {
      "name": "airport",
      "argument": "--airport",
      "version": "1.0",
      "description": "airport -I command parser",
      "author": "Kelly Brazil",
      "author_email": "kellyjonbrazil@gmail.com",
      "compatible": [
        "darwin"
      ],
      "magic_commands": [
        "airport -I"
      ]
    },
    {
      "name": "airport_s",
      "argument": "--airport-s",
      "version": "1.0",
      "description": "airport -s command parser",
      "author": "Kelly Brazil",
      "author_email": "kellyjonbrazil@gmail.com",
      "compatible": [
        "darwin"
      ],
      "magic_commands": [
        "airport -s"
      ]
    },
    ...
]

Let’s say I want a list of the parser names that are compatible with macOS. Here is a jq query that will get down to that level:

$ jc -a | jq '[.parsers[] | select(.compatible[] | contains("darwin")) | .name]' 
[
  "airport",
  "airport_s",
  "arp",
  "crontab",
  "crontab_u",
  "csv",
  ...
]

This is not too terribly bad, but you need to be careful about bracket and parenthesis placements. Here’s the same query in jello:

$ jc -a | jello '[parser["name"] for parser in _["parsers"] if "darwin" in parser["compatible"]]'
[
  "airport",
  "airport_s",
  "arp",
  "crontab",
  "crontab_u",
  "csv",
  ...
]

As you can see, jello gives you the JSON or JSON Lines input as a dictionary or list of dictionaries assigned to ‘_‘. Then you process it as you’d like using standard python syntax. jello automatically takes care of slurping input and printing valid JSON or JSON Lines depending on the value of the last expression.

The example above is not quite as terse as using jq, but it’s more readable to someone who is familiar with python list comprehension. As with any programming language, there are multiple ways to skin a cat. We can also do a similar query with a for loop:

$ jc -a | jello '\
result = []
for parser in _["parsers"]:
  for k, v in parser.items():
    if "darwin" in v:
      result.append(parser["name"])
result'
[
  "airport",
  "airport_s",
  "arp",
  "crontab",
  "crontab_u",
  "csv",
  ...
]

Advanced JSON Processing

These are very simple examples and jq syntax might be ok here (though I prefer python syntax). But what if we try to do something more complex? Let’s take one of the advanced examples from the excellent jq tutorial by Matthew Lincoln.

Under Grouping and Counting, Matthew describes an advanced jq filter against a sample Twitter dataset that includes JSON Lines data. There he describes the following query:

“We can now create a table of users. Let’s create a table with columns for the user id, user name, followers count, and a column of their tweet ids separated by a semicolon.”

https://programminghistorian.org/en/lessons/json-and-jq

Here is the final jq query:

$ cat twitterdata.jlines | jq -s 'group_by(.user) | 
                                 .[] | 
                                 {
                                   user_id: .[0].user.id, 
                                   user_name: .[0].user.screen_name, 
                                   user_followers: .[0].user.followers_count, 
                                   tweet_ids: [.[].id | tostring] | join(";")
                                 }'
...
{
  "user_id": 47073035,
  "user_name": "msoltanm",
  "user_followers": 63,
  "tweet_ids": "619172275741298700"
}
{
  "user_id": 2569107372,
  "user_name": "SlavinOleg",
  "user_followers": 35,
  "tweet_ids": "501064198973960200;501064202794971140;501064214467731460;501064215759568900;501064220121632800"
}
{
  "user_id": 2369225023,
  "user_name": "SkogCarla",
  "user_followers": 10816,
  "tweet_ids": "501064217667960800"
}
{
  "user_id": 2477475030,
  "user_name": "bennharr",
  "user_followers": 151,
  "tweet_ids": "501064201503113200"
}
{
  "user_id": 42226593,
  "user_name": "shirleycolleen",
  "user_followers": 2114,
  "tweet_ids": "619172281294655500;619172179960328200"
}
...

This is a fantastic query! It’s actually deceptively simple looking – it takes quite a few paragraphs for Matthew to describe how it works and there are some tricky brackets, braces, and parentheses in there that need to be set just right. Let’s see how we could tackle this task with jello using standard python syntax:

$ cat twitterdata.jlines | jello -l '\
user_ids = set()
for tweet in _:
    user_ids.add(tweet["user"]["id"])
result = []
for user in user_ids:
    user_profile = {}
    tweet_ids = []
    for tweet in _:
        if tweet["user"]["id"] == user:
            user_profile.update({
                "user_id": user,
                "user_name": tweet["user"]["screen_name"],
                "user_followers": tweet["user"]["followers_count"]})
            tweet_ids.append(str(tweet["id"]))
    user_profile["tweet_ids"] = ";".join(tweet_ids)
    result.append(user_profile)
result'
...
{"user_id": 2696111005, "user_name": "EGEVER142", "user_followers": 1433, "tweet_ids": "619172303654518784"}
{"user_id": 42226593, "user_name": "shirleycolleen", "user_followers": 2114, "tweet_ids": "619172281294655488;619172179960328192"}
{"user_id": 106948003, "user_name": "MrKneeGrow", "user_followers": 172, "tweet_ids": "501064228627705857"}
{"user_id": 18270633, "user_name": "ahhthatswhy", "user_followers": 559, "tweet_ids": "501064204661850113"}
{"user_id": 14331818, "user_name": "edsu", "user_followers": 4220, "tweet_ids": "615973042443956225;618602288781860864"}
{"user_id": 2569107372, "user_name": "SlavinOleg", "user_followers": 35, "tweet_ids": "501064198973960192;501064202794971136;501064214467731457;501064215759568897;501064220121632768"}
{"user_id": 22668719, "user_name": "nodehyena", "user_followers": 294, "tweet_ids": "501064222772445187"}
...

So there’s 17 lines of python… again not as terse as jq, but for pythonistas this is probably a lot easier to understand what is going on. This is a pretty simple and naive implementation – there are probably much better approaches that are shorter, simpler, faster, etc. but the point is I can come back six months from now and understand what is going on if I need to debug or tweak it.

Just for fun, let’s pipe this result through jtbl to see what it looks like:

   user_id  user_name          user_followers  tweet_ids
----------  ---------------  ----------------  ----------------------------------------------------------------------------------------------
...
2481812382  SadieODoyle                    42  501064200035516416
2696111005  EGEVER142                    1433  619172303654518784
  42226593  shirleycolleen               2114  619172281294655488;619172179960328192
 106948003  MrKneeGrow                    172  501064228627705857
  18270633  ahhthatswhy                   559  501064204661850113
  14331818  edsu                         4220  615973042443956225;618602288781860864
2569107372  SlavinOleg                     35  501064198973960192;501064202794971136;501064214467731457;501064215759568897;501064220121632768
  22668719  nodehyena                     294  501064222772445187
  23598003  victoriasview                1163  501064228288364546
 851336634  20mUsa                      15643  50106414
...

Very cool! Find more examples at https://github.com/kellyjonbrazil/jello. I hope you find jello useful in your command line pipelines.

Try the jello web demo!

Featured

JC Version 1.9.0 Released

I’m happy to announce the release of jc version 1.9.0 available on github and pypi. See below for more information on the new features and parsers.

To upgrade, run:

$ pip3 install --upgrade jc

jc In The News!

The Linux Unplugged podcast gave a shoutout to jc on their February 18, 2020 episode for their App Pick segment. The discussion starts at 45:47. Go check out the podcast!

New Parsers

jc now includes 50 parsers! New parsers (tested on linux and OSX) include airport -I, airport -s, file, ntpq -p, and timedatectl commands.

Documentation and schemas for all parsers can be found here.

airport -I command parser

OSX support for the airport -I command:

$ airport -I | jc --airport -p          # or:  jc -p airport -I
{
  "agrctlrssi": -66,
  "agrextrssi": 0,
  "agrctlnoise": -90,
  "agrextnoise": 0,
  "state": "running",
  "op_mode": "station",
  "lasttxrate": 195,
  "maxrate": 867,
  "lastassocstatus": 0,
  "802_11_auth": "open",
  "link_auth": "wpa2-psk",
  "bssid": "3c:37:86:15:ad:f9",
  "ssid": "SnazzleDazzle",
  "mcs": 0,
  "channel": "48,80"
}

airport -s command parser

OSX support for the airport -s command.

$ airport -s | jc --airport-s -p          or: jc -p airport -s
[
  {
    "ssid": "DIRECT-4A-HP OfficeJet 3830",
    "bssid": "00:67:eb:2a:a7:3b",
    "rssi": -90,
    "channel": "6",
    "ht": true,
    "cc": "--",
    "security": [
      "WPA2(PSK/AES/AES)"
    ]
  },
  {
    "ssid": "Latitude38",
    "bssid": "c0:ff:d5:d2:7a:f3",
    "rssi": -85,
    "channel": "11",
    "ht": true,
    "cc": "US",
    "security": [
      "WPA2(PSK/AES/AES)"
    ]
  },
  {
    "ssid": "xfinitywifi",
    "bssid": "6e:e3:0e:b8:45:99",
    "rssi": -83,
    "channel": "11",
    "ht": true,
    "cc": "US",
    "security": [
      "NONE"
    ]
  },
  ...
]

file command parser

Linux and OSX support for the file command:

$ file * | jc --file -p          or:  jc -p file *
[
  {
    "filename": "Applications",
    "type": "directory"
  },
  {
    "filename": "another file with spaces",
    "type": "empty"
  },
  {
    "filename": "argstest.py",
    "type": "Python script text executable, ASCII text"
  },
  {
    "filename": "blkid-p.out",
    "type": "ASCII text"
  },
  {
    "filename": "blkid-pi.out",
    "type": "ASCII text, with very long lines"
  },
  {
    "filename": "cd_catalog.xml",
    "type": "XML 1.0 document text, ASCII text, with CRLF line terminators"
  },
  {
    "filename": "centosserial.sh",
    "type": "Bourne-Again shell script text executable, UTF-8 Unicode text"
  },
  ...
]

ntpq command parser

Linux support for the ntpq -p command.

$ ntpq -p | jc --ntpq -p          # or:  jc -p ntpq -p
[
  {
    "remote": "44.190.6.254",
    "refid": "127.67.113.92",
    "st": 2,
    "t": "u",
    "when": 1,
    "poll": 64,
    "reach": 1,
    "delay": 23.399,
    "offset": -2.805,
    "jitter": 2.131,
    "state": null
  },
  {
    "remote": "mirror1.sjc02.s",
    "refid": "216.218.254.202",
    "st": 2,
    "t": "u",
    "when": 2,
    "poll": 64,
    "reach": 1,
    "delay": 29.325,
    "offset": 1.044,
    "jitter": 4.069,
    "state": null
  }
]

timedatectl command parser

Linux support for the timedatectl command:

$ timedatectl | jc --timedatectl -p          # or:  jc -p timedatectl
{
  "local_time": "Tue 2020-03-10 17:53:21 PDT",
  "universal_time": "Wed 2020-03-11 00:53:21 UTC",
  "rtc_time": "Wed 2020-03-11 00:53:21",
  "time_zone": "America/Los_Angeles (PDT, -0700)",
  "ntp_enabled": true,
  "ntp_synchronized": true,
  "rtc_in_local_tz": false,
  "dst_active": true
}

Updated Parsers

No updated parsers in this release.

Schema Changes

There are no schema changes in this release.

Full Parser List

  • airport -I
  • airport -s
  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • du
  • env
  • file
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • ntpq
  • /etc/passwd
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • timedatectl
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

JSON Tables in the Terminal

The other day I was looking around for a simple command-line tool to print JSON and JSON Lines data to a table in the terminal. I found a few programs that can do it with some massaging of the data, like visidata, jt, and json-table, but these really didn’t meet my requirements.

I wanted to pipe JSON or JSON Lines data into a program and get a nicely formatted table with correct headers without any additional configuration or arguments. I also wanted it to automatically fit the terminal width and wrap or truncate the columns to fit the data with no complicated configuration. Basically, I just wanted it to “do the right thing” so I can view JSON data in a tabular format without any fuss.

I ended up creating a little command-line utility called jtbl that does exactly that:

$ cat cities.json | jtbl 
  LatD    LatM    LatS  NS      LonD    LonM    LonS  EW    City               State
------  ------  ------  ----  ------  ------  ------  ----  -----------------  -------
    41       5      59  N         80      39       0  W     Youngstown         OH
    42      52      48  N         97      23      23  W     Yankton            SD
    46      35      59  N        120      30      36  W     Yakima             WA
    42      16      12  N         71      48       0  W     Worcester          MA
    43      37      48  N         89      46      11  W     Wisconsin Dells    WI
    36       5      59  N         80      15       0  W     Winston-Salem      NC
    49      52      48  N         97       9       0  W     Winnipeg           MB

jtbl is simple and elegant. It just takes in piped JSON or JSON Lines data and prints a table. There’s only one option to turn on column truncation vs. wrapping columns if the terminal width is too narrow to display the complete table.

$ jtbl -h
jtbl:   Converts JSON and JSON Lines to a table

Usage:  <JSON Data> | jtbl [OPTIONS]

        -t  truncate data instead of wrapping if too long for the terminal width
        -v  version info
        -h  help

Here’s an example using a relatively slim terminal width of 75:

$ jc dig www.cnn.com | jq '.[].answer' | jtbl 
+----------------+---------+--------+-------+----------------+
| name           | class   | type   |   ttl | data           |
+================+=========+========+=======+================+
| www.cnn.com.   | IN      | CNAME  |   143 | turner-tls.map |
|                |         |        |       | .fastly.net.   |
+----------------+---------+--------+-------+----------------+
| turner-tls.map | IN      | A      |     4 | 151.101.1.67   |
| .fastly.net.   |         |        |       |                |
+----------------+---------+--------+-------+----------------+
| turner-tls.map | IN      | A      |     4 | 151.101.129.67 |
| .fastly.net.   |         |        |       |                |
+----------------+---------+--------+-------+----------------+
| turner-tls.map | IN      | A      |     4 | 151.101.65.67  |
| .fastly.net.   |         |        |       |                |
+----------------+---------+--------+-------+----------------+
| turner-tls.map | IN      | A      |     4 | 151.101.193.67 |
| .fastly.net.   |         |        |       |                |
+----------------+---------+--------+-------+----------------+

or with truncation enabled:

$ jc dig www.cnn.com | jq '.[].answer' | jtbl -t 
name                  class    type      ttl  data
--------------------  -------  ------  -----  --------------------
www.cnn.com.          IN       CNAME      48  turner-tls.map.fastl
turner-tls.map.fastl  IN       A          13  151.101.129.67
turner-tls.map.fastl  IN       A          13  151.101.65.67
turner-tls.map.fastl  IN       A          13  151.101.193.67
turner-tls.map.fastl  IN       A          13  151.101.1.67

Here’s an example using it to print the result of an XML API query response, converted to JSON with jc, and filtered with jq:

$ curl -X GET --basic -u "testuser:testpassword" https://reststop.randomhouse.com/resources/works/19306 | jc --xml | jq '.work' | jtbl
+----------+----------+----------+------------+------------+----------+------------+------------+------------+------------+
| titles   |   workid | @uri     | authorwe   | onsaleda   | series   | titleAut   | titleSub   | titlesho   | titleweb   |
|          |          |          | b          | te         |          | h          | titleAut   | rt         |            |
|          |          |          |            |            |          |            | h          |            |            |
+==========+==========+==========+============+============+==========+============+============+============+============+
|          |    19306 | https:// | BROWN, D   | 2003-09-   | ROBERT L | Angels &   | Angels &   | ANGELS &   | Angels &   |
|          |          | reststop | AN         | 02T00:00   | ANGDON   |  Demons    |  Demons    |  DEMON(L   |  Demons    |
|          |          | .randomh |            | :00-04:0   |          | : Dan Br   | :  : Dan   | PTP)(REI   |            |
|          |          | ouse.com |            | 0          |          | own        |  Brown     | )(MTI)     |            |
|          |          | /resourc |            |            |          |            |            |            |            |
|          |          | es/works |            |            |          |            |            |            |            |
|          |          | /19306   |            |            |          |            |            |            |            |
+----------+----------+----------+------------+------------+----------+------------+------------+------------+------------+

Again, with truncation enabled:

$ curl -X GET --basic -u "testuser:testpassword" https://reststop.randomhouse.com/resources/works/19306 | jc --xml | jq '.work' | jtbl -t
authorweb    titles      workid  @uri        onsaledate    series      titleAuth    titleSubti    titleshort    titleweb
-----------  --------  --------  ----------  ------------  ----------  -----------  ------------  ------------  ----------
BROWN, DAN                19306  https://re  2003-09-02    ROBERT LAN  Angels & D   Angels & D    ANGELS & D    Angels & D

I found that having the ability to quickly see the JSON data in a tabular, horizontal format can sometimes help me visualize ‘where I am’ in the data more easily than looking at long vertical lists of JSON.

I hope you enjoy it!

Featured

JC Version 1.8.0 Released

I’m excited to announce the release of jc version 1.8.0 available on github and pypi. See below for more information on the new features and parsers.

To upgrade, run:

$ pip3 install --upgrade jc

New Parsers

jc now includes 45 parsers! New parsers (tested on linux and OSX) include blkid, last, lastb, who, /etc/passwd files, /etc/shadow files, /etc/group files, /etc/gshadow files, and CSV files.

Documentation and schemas for all parsers can be found here.

blkid command parser

Linux support for the blkid command:

$ blkid | jc --blkid -p          # or:  jc -p blkid
[
  {
    "device": "/dev/sda1",
    "uuid": "05d927ab-5875-49e4-ada1-7f46cb32c932",
    "type": "xfs"
  },
  {
    "device": "/dev/sda2",
    "uuid": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "type": "LVM2_member"
  },
  {
    "device": "/dev/mapper/centos-root",
    "uuid": "07d718ff-950c-4e5b-98f0-42a1147c77d9",
    "type": "xfs"
  },
  {
    "device": "/dev/mapper/centos-swap",
    "uuid": "615eb89a-bcbf-46fd-80e3-c483ff5c931f",
    "type": "swap"
  }
]

$ sudo blkid -o udev -ip /dev/sda2 | jc --blkid -p          # or:  sudo jc -p blkid -o udev -ip /dev/sda2
[
  {
    "id_fs_uuid": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "id_fs_uuid_enc": "3klkIj-w1kk-DkJi-0XBJ-y3i7-i2Ac-vHqWBM",
    "id_fs_version": "LVM2\x20001",
    "id_fs_type": "LVM2_member",
    "id_fs_usage": "raid",
    "id_iolimit_minimum_io_size": 512,
    "id_iolimit_physical_sector_size": 512,
    "id_iolimit_logical_sector_size": 512,
    "id_part_entry_scheme": "dos",
    "id_part_entry_type": "0x8e",
    "id_part_entry_number": 2,
    "id_part_entry_offset": 2099200,
    "id_part_entry_size": 39843840,
    "id_part_entry_disk": "8:0"
  }
]

last and lastb command parsers

Linux and OSX support for the last command. Linux support for the lastb command.

$ last | jc --last -p          # or:  jc -p last
[
  {
    "user": "joeuser",
    "tty": "ttys002",
    "hostname": null,
    "login": "Thu Feb 27 14:31",
    "logout": "still logged in"
  },
  {
    "user": "joeuser",
    "tty": "ttys003",
    "hostname": null,
    "login": "Thu Feb 27 10:38",
    "logout": "10:38",
    "duration": "00:00"
  },
  {
    "user": "joeuser",
    "tty": "ttys003",
    "hostname": null,
    "login": "Thu Feb 27 10:18",
    "logout": "10:18",
    "duration": "00:00"
  },
  ...
]

$ sudo lastb | jc --last -p          # or:  sudo jc -p lastb
[
  {
    "user": "joeuser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  },
  {
    "user": "joeuser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  },
  {
    "user": "jouser",
    "tty": "ssh:notty",
    "hostname": "127.0.0.1",
    "login": "Tue Mar 3 00:48",
    "logout": "00:48",
    "duration": "00:00"
  }
]

who command parser

Linux and OSX support for the who command:

$ who | jc --who -p          # or:  jc -p who
[
  {
    "user": "joeuser",
    "tty": "ttyS0",
    "time": "2020-03-02 02:52"
  },
  {
    "user": "joeuser",
    "tty": "pts/0",
    "time": "2020-03-02 05:15",
    "from": "192.168.71.1"
  }
]

$ who -a | jc --who -p          # or:  jc -p who -a
[
  {
    "event": "reboot",
    "time": "Feb 7 23:31",
    "pid": 1
  },
  {
    "user": "joeuser",
    "writeable_tty": "-",
    "tty": "console",
    "time": "Feb 7 23:32",
    "idle": "old",
    "pid": 105
  },
  {
    "user": "joeuser",
    "writeable_tty": "+",
    "tty": "ttys000",
    "time": "Feb 13 16:44",
    "idle": ".",
    "pid": 51217,
    "comment": "term=0 exit=0"
  },
  {
    "user": "joeuser",
    "writeable_tty": "?",
    "tty": "ttys003",
    "time": "Feb 28 08:59",
    "idle": "01:36",
    "pid": 41402
  },
  {
    "user": "joeuser",
    "writeable_tty": "+",
    "tty": "ttys004",
    "time": "Mar 1 16:35",
    "idle": ".",
    "pid": 15679,
    "from": "192.168.1.5"
  }
]

CSV File Parser

Convert generic CSV files to JSON. The parser will attempt to automatically detect the delimiter character. If it cannot detect the delimiter it will use the comma (‘,‘) as the delimiter. The file must contain a header row as the first line:

$ cat homes.csv 
"Sell", "List", "Living", "Rooms", "Beds", "Baths", "Age", "Acres", "Taxes"
142, 160, 28, 10, 5, 3,  60, 0.28,  3167
175, 180, 18,  8, 4, 1,  12, 0.43,  4033
129, 132, 13,  6, 3, 1,  41, 0.33,  1471
...

$ cat homes.csv | jc --csv -p
[
  {
    "Sell": "142",
    "List": "160",
    "Living": "28",
    "Rooms": "10",
    "Beds": "5",
    "Baths": "3",
    "Age": "60",
    "Acres": "0.28",
    "Taxes": "3167"
  },
  {
    "Sell": "175",
    "List": "180",
    "Living": "18",
    "Rooms": "8",
    "Beds": "4",
    "Baths": "1",
    "Age": "12",
    "Acres": "0.43",
    "Taxes": "4033"
  },
  {
    "Sell": "129",
    "List": "132",
    "Living": "13",
    "Rooms": "6",
    "Beds": "3",
    "Baths": "1",
    "Age": "41",
    "Acres": "0.33",
    "Taxes": "1471"
  },
  ...
]

/etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow file parsers

Convert /etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow files to JSON format:

$ cat /etc/passwd | jc --passwd -p
[
  {
    "username": "nobody",
    "password": "*",
    "uid": -2,
    "gid": -2,
    "comment": "Unprivileged User",
    "home": "/var/empty",
    "shell": "/usr/bin/false"
  },
  {
    "username": "root",
    "password": "*",
    "uid": 0,
    "gid": 0,
    "comment": "System Administrator",
    "home": "/var/root",
    "shell": "/bin/sh"
  },
  {
    "username": "daemon",
    "password": "*",
    "uid": 1,
    "gid": 1,
    "comment": "System Services",
    "home": "/var/root",
    "shell": "/usr/bin/false"
  },
  ...
]

$ sudo cat /etc/shadow | jc --shadow -p
[
  {
    "username": "root",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  {
    "username": "daemon",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  {
    "username": "bin",
    "password": "*",
    "last_changed": 18113,
    "minimum": 0,
    "maximum": 99999,
    "warn": 7,
    "inactive": null,
    "expire": null
  },
  ...
]

$ cat /etc/group | jc --group -p
[
  {
    "group_name": "nobody",
    "password": "*",
    "gid": -2,
    "members": []
  },
  {
    "group_name": "nogroup",
    "password": "*",
    "gid": -1,
    "members": []
  },
  {
    "group_name": "wheel",
    "password": "*",
    "gid": 0,
    "members": [
      "root"
    ]
  },
  {
    "group_name": "certusers",
    "password": "*",
    "gid": 29,
    "members": [
      "root",
      "_jabber",
      "_postfix",
      "_cyrus",
      "_calendar",
      "_dovecot"
    ]
  },
  ...
]

$ cat /etc/gshadow | jc --gshadow -p
[
  {
    "group_name": "root",
    "password": "*",
    "administrators": [],
    "members": []
  },
  {
    "group_name": "adm",
    "password": "*",
    "administrators": [],
    "members": [
      "syslog",
      "joeuser"
    ]
  },
  ...
]

Updated Parsers

  • The ls parser now supports filenames that contain newline characters when using ls -l or ls -b. A warning message will be sent to stderr if newlines are detected and ls -l or ls -b are not used:
$ ls | jc --ls

jc:  Warning - Newline characters detected. Filenames probably corrupted. Use ls -l or -b instead.

[{"filename": "this file has"}, {"filename": "a newline inside"}, {"filename": "this file has"}, {"filename": "four contiguous newlines inside"}, ...]
  • The ls parser now supports multiple directory listings, globbing, and recursive listings.
$ ls -R | jc --ls
[{"filename": "centos-7.7"}, {"filename": "create_fixtures.sh"}, {"filename": "generic"}, {"filename": "osx-10.11.6"}, {"filename": "osx-10.14.6"}, ...]

Alternative “Magic” Syntax

jc now accepts a simplified syntax for most command parsers. Instead of piping the data into jc you can now also prepend “jc” to the command you would like to convert. Note that command aliases are not supported:

$ jc dig www.example.com
[{"id": 31113, "opcode": "QUERY", "status": "NOERROR", "flags": ["qr", "rd", "ra"], "query_num": 1, "answer_num": 1, "authority_num": 0, "additional_num": 1, "question": {"name": "www.example.com.", "class": "IN", "type": "A"}, "answer": [{"name": "www.example.com.", "class": "IN", "type": "A", "ttl": 35366, "data": "93.184.216.34"}], "query_time": 37, "server": "2600", "when": "Mon Mar 02 16:13:31 PST 2020", "rcvd": 60}]

You can also insert jc options before the command:

$ jc -pqd dig www.example.com
[
  {
    "id": 7495,
    "opcode": "QUERY",
    "status": "NOERROR",
    "flags": [
      "qr",
      "rd",
      "ra"
    ],
    "query_num": 1,
    "answer_num": 1,
    "authority_num": 0,
    "additional_num": 1,
    "question": {
      "name": "www.example.com.",
      "class": "IN",
      "type": "A"
    },
    "answer": [
      {
        "name": "www.example.com.",
        "class": "IN",
        "type": "A",
        "ttl": 36160,
        "data": "93.184.216.34"
      }
    ],
    "query_time": 40,
    "server": "2600",
    "when": "Mon Mar 02 16:15:21 PST 2020",
    "rcvd": 60
  }
]

Schema Changes

There are no schema changes in this release.

Full Parser List

  • arp
  • blkid
  • crontab
  • crontab-u
  • CSV
  • df
  • dig
  • du
  • env
  • free
  • fstab
  • /etc/group
  • /etc/gshadow
  • history
  • /etc/hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • last and lastb
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • /etc/passwd
  • pip list
  • pip show
  • ps
  • route
  • /etc/shadow
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • uname -a
  • uptime
  • w
  • who
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

Applying Orchestration and Choreography to Cybersecurity Automation

Imagine a world where most of your security stack seamlessly integrates with each other, has access to the latest threat intelligence from internal and external sources, and automatically mitigates the most severe incidents. Suspicious files found in emails get sent to the closest sandbox for detonation, where the hash and other IOCs are sent to endpoints, NGFWs, proxies, etc. to inoculate the organization, and then send all of the relevant information to the SOC as an incident ticket.

Many organizations can at least do the above with a Security Orchestration Automation and Response (SOAR) platform implementation. Several vendors offer this type of Orchestration platform, including Splunk (Phantom), Palo Alto Networks (Demisto), Fortinet (Cybersponse), and IBM (Resilient). These platforms have become mainstream within the past few years and with more and more cybersecurity professionals learning the python programming language it has become easier to implement and customize them. In fact, no programming experience is needed at all for many use cases since playbooks can be created and maintained with a graphical builder.

Cybersponse Graphical Playbook Editor

I’m a big fan of using Orchestration to automate workflows with playbooks – in fact I’ve written a couple of these integrations before for Phantom and Demisto. But there is another automation paradigm that doesn’t get talked about as much in the cybersecurity realm: Choreography.

Orchestration

So we already have an idea of what Orchestration is: it’s a central repository of vendor integrations and associated actions that can be connected together in clever and novel ways to create playbooks. Playbooks are like scripts that run based on incoming events, schedules, or can even be run manually to automate repetitive tasks. This automation removes the human-error factor and can reduce the workload of the Security team.

Centralized Automation with Orchestration

The key piece about Orchestration is that it is centralized. There is typically a central server that has all of the vendor integration information and playbooks. Alarms, logs, alerts, etc. get sent to this server so it can act as the conductor and tell each security device in the stack what to do and when to do it.

This approach has pros and cons:

Pros:

  • Very flexible – you can make a playbook do almost anything you can think of
  • Can version control the playbooks in a central repository like git
  • Large libraries of vendor apps
  • Typically have a good user communities

Cons:

  • Can be brittle if APIs change, unsupported vendors are introduced, or if there are connectivity issues to the central Orchestrator
  • Vendor lock-in to a SOAR platform / not open source
  • Can require python programming experience to onboard an unsupported security service or to create a complex playbook

Let’s compare this to Choreography – the other, lesser-known automation paradigm available to us.

Choreography

Choreography? Where did that come from? Well, the concepts of Orchestration and Choreography come from the world of Service Oriented Architecture (SOA). SOA had some good ideas, but it didn’t really take off until it recently morphed and rebranded as Microservice Architecture. (Yes, this is an over-simplification for the scope of this post)

We almost take microservice architectures for granted now. Cloud application delivery and containerization of services are not as bleeding-edge as they were just a couple of years ago. We intuitively understand that microservices act independently yet are connected to other microservices to make up an application. The way these microservices are connected can be described as Orchestration or Choreography.

Now we are just extending the metaphor and considering each piece of our security stack as a ‘microservice’. For example, your NGFW, sandbox, email security gateway, NAC, Intel feed, etc. are all cybersecurity microservices that need to be configured to talk to one another to enable your cybersecurity ‘application’.

Distributed Automation with Choreography

In the case of Choreography, each of these security ‘microservices’ (or security appliances) knows what they are supposed to do by subscribing to one or more channels on a message bus. This bus allows the service to receive alerts and IOC information in near-real-time and then publish their results on one or more channels on that same bus. It’s almost like layer 7 multicast for you router geeks out there.

In this paradigm, there is no need for a central repository of rules or playbooks for many standard use-cases because the ‘fabric’ gets smarter as more and more different types of security services join. Unlike an orchestra, which follows the lead of the conductor, each service works independently based on its own configuration. Each service knows its own dance moves and works harmoniously in relation to the other services.

The Message Bus

How does this work in the real world?

There are a couple examples of the Choreography approach being used in the Cybersecurity realm. A proprietary implementation by Fortinet (disclaimer: I am a Fortinet employee) is called the “Security Fabric”.

Fortinet Security Fabric

The Fortinet Security Fabric

Fortinet’s Security Fabric is a proprietary implementation that behaves like a message bus to learn about new Fortinet and Fabric Ready ecosystem partner appliances and services as soon as they connect to the fabric.  These services are configured to connect to the Security Fabric and take appropriate action when a security incident is identified.

For example, after installing a FortiSandbox appliance and adding it to the Security Fabric, other Fortinet or “Fabric-Ready” partner appliances, such as the NGFW and Secure Email Gateway can send suspicious files they detect to the Security Fabric where the sandbox service is listening. The FortiSandbox, in turn, can publish the IOC results of the scans it performs to the Security Fabric so other Fortinet or Fabric-Ready partner appliances (e.g. NGFW, FortiGuard, FortiEDR) can ingest them and take appropriate action.

This is very powerful. As more services are connected to the Security Fabric, it gets smarter, more capable, and scales – automatically.

OpenDXL

Another open-source, multi-vendor example of a message bus being used for cybersecurity choreography is OpenDXL. OpenDXL was originally developed by McAfee, as a security-specific message bus, but it was open-sourced under the Organization for the Advancement of Structured Information Standards (OASIS) Open Cybersecurity Alliance (OCA) project. (Disclaimer: Fortinet is a sponsor of OCA) This project brings together the message bus concept to integrate multiple cybersecurity services using well-known formats like STIX2 to influence its ontology.

OpenDXL Architecture

Some of the pros and cons of the Choreography approach:

Pros:

  • The ‘fabric’ automatically gets smarter and more capable as more security services are connected
  • No need for dozens of boilerplate playbooks
  • Open-source and proprietary options available
  • No reliance on a central conductor – less brittle to Orchestrator outages or misconfigurations.
  • Integrations “just work” together if they are part of the ecosystem

Cons:

  • Less granular control over automation workflows
  • Open-source options are still maturing
  • Typically, no central repository for service configurations

Which Way is the Best?

We know that automation will improve our security operations, but which approach is best? Since Orchestration and Choreography both have their own pros and cons that don’t overlap too much it probably makes sense to use both.

Choreography can reduce the amount of boilerplate playbooks you need to bootstrap your automation initiative, while Orchestration can be used to automate higher-level business or incident response workflows.

By applying the application architecture concepts of SOA and microservices to cybersecurity we can take security automation to the next level.

Featured

JC Version 1.7.1 Released

I’m happy to announce that jc version 1.7.1 has been released and is available on github and pypi. In addition to the new and updated parsers and features outlined below, some back-end code cleanup to improve performance along with minor bug fixes were completed.

To upgrade, run:

$ pip3 install --upgrade jc

New Parsers

jc now includes 37 parsers! New parsers (tested on linux and OSX) include id, crontab-u, INI, XML, and YAML:

id parser

Linux and OSX support for the id command:

$ id | jc --id -p
{
  "uid": {
    "id": 1000,
    "name": "joeuser"
  },
  "gid": {
    "id": 1000,
    "name": "joeuser"
  },
  "groups": [
    {
      "id": 1000,
      "name": "joeuser"
    },
    {
      "id": 10,
      "name": "wheel"
    }
  ],
  "context": {
    "user": "unconfined_u",
    "role": "unconfined_r",
    "type": "unconfined_t",
    "level": "s0-s0:c0.c1023"
  }
}

crontab files with user defined

Some crontab files contain the user field. In this case, use the new crontab-u parser:

$ cat /etc/crontab | jc --crontab-u -p
{
  "variables": [
    {
      "name": "MAILTO",
      "value": "root"
    },
    {
      "name": "PATH",
      "value": "/sbin:/bin:/usr/sbin:/usr/bin"
    },
    {
      "name": "SHELL",
      "value": "/bin/bash"
    }
  ],
  "schedule": [
    {
      "minute": [
        "5"
      ],
      "hour": [
        "10-11",
        "22"
      ],
      "day_of_month": [
        "*"
      ],
      "month": [
        "*"
      ],
      "day_of_week": [
        "*"
      ],
      "user": "root",
      "command": "/var/www/devdaily.com/bin/mk-new-links.php"
    },
    {
      "minute": [
        "30"
      ],
      "hour": [
        "4/2"
      ],
      "day_of_month": [
        "*"
      ],
      "month": [
        "*"
      ],
      "day_of_week": [
        "*"
      ],
      "user": "root",
      "command": "/var/www/devdaily.com/bin/create-all-backups.sh"
    },
    {
      "occurrence": "yearly",
      "user": "root",
      "command": "/home/maverick/bin/annual-maintenance"
    },
    {
      "occurrence": "reboot",
      "user": "root",
      "command": "/home/cleanup"
    },
    {
      "occurrence": "monthly",
      "user": "root",
      "command": "/home/maverick/bin/tape-backup"
    }
  ]
}

INI file parser

Convert generic INI files to JSON:

$ cat example.ini
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes

[bitbucket.org]
User = hg

[topsecret.server.com]
Port = 50022
ForwardX11 = no

$ cat example.ini | jc --ini -p
{
  "bitbucket.org": {
    "serveraliveinterval": "45",
    "compression": "yes",
    "compressionlevel": "9",
    "forwardx11": "yes",
    "user": "hg"
  },
  "topsecret.server.com": {
    "serveraliveinterval": "45",
    "compression": "yes",
    "compressionlevel": "9",
    "forwardx11": "no",
    "port": "50022"
  }
}

XML file parser

Convert generic XML files to JSON:

$ cat cd_catalog.xml 
<?xml version="1.0" encoding="UTF-8"?>
<CATALOG>
  <CD>
    <TITLE>Empire Burlesque</TITLE>
    <ARTIST>Bob Dylan</ARTIST>
    <COUNTRY>USA</COUNTRY>
    <COMPANY>Columbia</COMPANY>
    <PRICE>10.90</PRICE>
    <YEAR>1985</YEAR>
  </CD>
  <CD>
    <TITLE>Hide your heart</TITLE>
    <ARTIST>Bonnie Tyler</ARTIST>
    <COUNTRY>UK</COUNTRY>
    <COMPANY>CBS Records</COMPANY>
    <PRICE>9.90</PRICE>
    <YEAR>1988</YEAR>
  </CD>
  ...

$ cat cd_catalog.xml | jc --xml -p
{
  "CATALOG": {
    "CD": [
      {
        "TITLE": "Empire Burlesque",
        "ARTIST": "Bob Dylan",
        "COUNTRY": "USA",
        "COMPANY": "Columbia",
        "PRICE": "10.90",
        "YEAR": "1985"
      },
      {
        "TITLE": "Hide your heart",
        "ARTIST": "Bonnie Tyler",
        "COUNTRY": "UK",
        "COMPANY": "CBS Records",
        "PRICE": "9.90",
        "YEAR": "1988"
      },
  ...
}

YAML file parser

Convert YAML files to JSON – even files that contain multiple YAML documents:

$ cat istio-mtls-permissive.yaml 
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "default"
spec:
  peers:
  - mtls: {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

$ cat istio-mtls-permissive.yaml | jc --yaml -p
[
  {
    "apiVersion": "authentication.istio.io/v1alpha1",
    "kind": "Policy",
    "metadata": {
      "name": "default",
      "namespace": "default"
    },
    "spec": {
      "peers": [
        {
          "mtls": {}
        }
      ]
    }
  },
  {
    "apiVersion": "networking.istio.io/v1alpha3",
    "kind": "DestinationRule",
    "metadata": {
      "name": "default",
      "namespace": "default"
    },
    "spec": {
      "host": "*.default.svc.cluster.local",
      "trafficPolicy": {
        "tls": {
          "mode": "ISTIO_MUTUAL"
        }
      }
    }
  }
]

Updated Parsers

  • history parser now outputs line fields as integers
  • crontab parser bug fix for an issue that sometimes lost a row of data
  • Updated the compatibility information for du and history parsers

__version__ Attribute Added

Python programmers can now call the __version__ attribute on all parsers when running them as modules.

>>> import jc.parsers.arp
>>> print(jc.parsers.arp.__version__)
1.1

Added Exit Codes

jc will now provide an exit code (1) if it did not successfully exit.

Schema Changes

The history parser now outputs line fields as integers

$ history | jc --history -p
[
  {
    "line": 118,
    "command": "sleep 100"
  },
  ...
]
 

Full Parser List

  • arp
  • crontab
  • crontab-u
  • df
  • dig
  • du
  • env
  • free
  • fstab
  • history
  • hosts
  • id
  • ifconfig
  • INI
  • iptables
  • jobs
  • ls
  • lsblk
  • lsmod
  • lsof
  • mount
  • netstat
  • pip list
  • pip show
  • ps
  • route
  • ss
  • stat
  • systemctl
  • systemctl list-jobs
  • systemctl list-sockets
  • systemctl list-unit-files
  • uname -a
  • uptime
  • w
  • XML
  • YAML

For more information on the motivations for creating jc, see my blog post.

Happy parsing!

Featured

Microservice Security Design Patterns for Kubernetes (Part 5)

The Service Mesh Sidecar-on-Sidecar Pattern

In Part 4 of of my series on Microservice Security Patterns for Kubernetes we dove into the Sidecar Security Pattern and configured a working application with micro-segmentation enforcement and deep inspection for application-layer protection. The Sidecar Security Pattern is nice and clean, but what if you are running a Service Mesh like Istio with Envoy?

In this post we will take the Sidecar Security Pattern from Part 4 and apply it in an Istio Service Mesh using Envoy sidecars. This is essentially a Sidecar-on-Sidecar Pattern that will allow us to not only use the native encryption and segmentation capabilities of the Service Mesh, but will allow us to layer on L7 application security for OWASP top 10 type of attacks against the microservices.

How does the Service Mesh Sidecar-on-Sidecar Pattern work?

It’s Sidecars All The Way Down

As we discussed in Part 4, you can have multiple containers in a Pod. We used the modsecurity container as a sidecar to intercept HTTP requests and inspect them before forwarding them on to the microsimserver container in the same pod. But with an Istio Service Mesh, there will also be an Envoy container injected into the Pod and it will do the egress and ingress traffic interception. Can we have two sidecars in a Pod?

The answer is yes. In the case of Envoy using the sidecar injection functionality, it configures itself based on the existing Pod spec in the deployment manifest. This means that we can use a manifest nearly identical to what we used in Part 4 and Envoy will correctly configure itself to send intercepted traffic on to the modsecurity container, which will then send the traffic to the microsimserver container.

In this post we will be demonstrating this in action. There are surprisingly few changes that need to be made to the Security Sidecar Pattern deployment file to make this work. Also, we’ll be able to easily see how this works using the Kiali dashboard which provides visualization for the Istio Service Mesh.

The Sidecar-on-Sidecar Pattern

We’ll be using this deployment manifest that is nearly identical to the Security Sidecar Pattern manifest from Part 4. Here is what the design looks like:

First we’ll enable service-to-service encryption, then strict mutual TLS (mTLS) with RBAC to provide micro-segmentation. Finally, we’ll configure Istio ingress gateway so we can access the app from the public internet.

But first, let’s just deploy the modified Sidecar Pattern manifest with a vanilla Istio configuration.

Spinning up the Cluster in GKE

We’ll spin up a kubernetes cluster in GKE similar to how we did previously in Part 2 except this time we’ll use 4 nodes of n1-standard-2 machine type instead of 3. Since we’ll be using Istio to control service-to-service traffic (East/West flows) we no longer need to check the Enable Network Policy box. Instead, we will need to check the Enable Istio (beta) box under Additional Features.

We’ll start with setting Enable mTLS (beta) to Permissive. We will change this later via configuration files as we try out some scenarios.

I’m not going to give a complete tutorial on how to complete the set up of Istio on GKE, but I basically used the instructions documented in the following links to enable Prometheus and Grafana. I used the same idea to enable the Kiali dashboard to visualize the Service Mesh. We’ll be using the Kiali service graphs to verify the status of the application.

Once you have Kiali enabled, you can configure port forwarding on the Service so you can browse to the dashboard using your laptop.

Click the https://ssh.cloud.google.com/devshell/proxy?port=8080 link and then append /kiali at the end of the translated link in your browser. You should see a login screen. Use the default credentials or the ones you specified with a kubernetes secret during setup. You should see a blank service graph:

Make sure to check the Security checkbox under the Display menu:

Finally, we want to enable automatic sidecar injection for the Envoy proxy by running this command within Cloud Shell:

$ kubectl label namespace default istio-injection=enabled

Alright! Now let’s deploy the app.

Deploying the Sidecar-on-Sidecar Manifest

There are only a few minor differences between the sidecar.yaml manifest used in Part 4 and the istio-sidecar.yaml that we will be using for the following examples. Let’s take a look:

Service Accounts

apiVersion: v1
kind: ServiceAccount
metadata:
  name: www
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: db
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: auth

First, we have added these ServiceAccount objects. This is what Istio uses to differentiate services within the mesh and affects how the certificates used in mTLS are generated. You’ll see how we bind these ServiceAccount objects to the Pods next.

Deployments

We’ll just take a look at the www Deployment since the same changes are required for all of the Deployments.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: www
spec:
  replicas: 3
  selector:
    matchLabels:
      app: www
  template:
    metadata:
      labels:
        app: www
        version: v1.0       # add version
    spec:
      serviceAccountName: www      # add serviceAccountName
      containers:
      - name: modsecurity
        image: owasp/modsecurity-crs:v3.2-modsec2-apache
        ports:
        - containerPort: 80
        env:
        - name: SETPROXY
          value: "True"
        - name: PROXYLOCATION
          value: "http://127.0.0.1:8080/"
      - name: microsimserver
        image: kellybrazil/microsimserver
        ports:
        - containerPort: 8080       # add microsimserver port
        env:
        - name: STATS_PORT
          value: "5000"
      - name: microsimclient
        image: kellybrazil/microsimclient
        env:
        - name: STATS_PORT
          value: "5001"
        - name: REQUEST_URLS
          value: "http://auth.default.svc.cluster.local:8080/,http://db.default.svc.cluster.local:8080/"
        - name: SEND_SQLI
          value: "True"

The only difference from the original sidecar.yaml is:

  • We have added a version label. Istio requires this label to be included.
  • We associated the Pods with the appropriate ServiceAccountName. This will be important for micro-segmentation later on.
  • We add the containerPort configuration for the microsimserver containers. This is important so the Envoy proxy sidecar can configure itself properly.

Services

Now let’s see the minor changes to the Services. Since they are all very similar, we will just take a look at the www Service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: www
  name: www
spec:
  # externalTrafficPolicy: Local      # remove externalTrafficPolicy
  ports:
  - port: 8080
    targetPort: 80
    name: http         # add port name
  selector:
    app: www
  sessionAffinity: None
  # type: LoadBalancer          # remove LoadBalancer type

We have removed a couple of items from the www service: externalTrafficPolicy and type. This is because the www service is no longer directly exposed to the public internet. We’ll expose it later using an Istio Ingress Gateway.

Also, we have added the port name field. This is required so Istio can correctly configure Envoy to listen for the correct protocol and produce the correct telemetry for the inter-service traffic.

Deploy the App

Now let’s deploy the application using kubectl. Copy/paste the manifest to a file called istio-sidecar.yaml within Cloud Shell using vi. Then run:

$ kubectl apply -f istio-sidecar.yaml
serviceaccount/www created
serviceaccount/db created
serviceaccount/auth created
deployment.apps/www created
deployment.apps/auth created
deployment.apps/db created
service/www created
service/auth created
service/db created

After a couple of minutes you should see this within the Kiali dashboard:

Excellent! You’ll notice the services will alternate between green and orange. This is because the www service is sending SQLi attacks to the db and auth services every so often and those are being blocked with HTTP 403 errors being returned by the modsecurity WAF container.

Voila! We have application layer security in Istio!

But you may have noticed that there is no encryption between services enabled yet. Also, all services can talk to each other, so we don’t have proper micro-segmentation. We can illustrate that with a curl from auth to db:

$ kubectl exec auth-cf6f45fb-9k678 -c microsimserver curl http://db:8080
<snip>
sufH1FhoMgvXvbPOkE3O0H3MwNAN
Tue Jan 28 01:16:48 2020   hostname: db-55747d84d8-jlz7z   ip: 10.8.0.13   remote: 127.0.0.1   hostheader: 127.0.0.1:8080   path: /

Let’s fix these issues.

Encrypting the East/West Traffic

It is fairly easy to encrypt East/West traffic using Istio. First we’ll demonstrate permissive mTLS and then we’ll advance to strict mTLS with RBAC to enforce micro-segmentation.

Here’s what the manifest for this configuration looks like:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "default"
spec:
  peers:
  - mtls: {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

The Policy manifest specifies that all Pods in the default namespace will only accept encrypted requests using TLS. The DestinationRule manifest specifies how the client-side outbound connections are handled. Here we see that connections to any services in the default namespace will use TLS (*.default.svc.cluster.local) This effectively disables plaintext traffic between services in the namespace.

Copy/paste the manifest text to a file called istio-mtls-permissive.yaml. Then apply it with kubectl:

$ kubectl apply -f istio-mtls-permissive.yaml
policy.authentication.istio.io/default created
destinationrule.networking.istio.io/default created

After 30 seconds or so you should start to see the padlocks between the services in the Kiali Dashboard indicating that the communications are encrypted. (Ensure you checked the Security checkbox under the Display drop-down)

Nice! We have successfully encrypted traffic between our services.

Enforcing micro-segmentation

Even though the communications between services is now encrypted, we still don’t have effective micro-segmentation between Pods running the Envoy sidecar. We can test this again with a curl from an auth pod to a db pod:

$ kubectl exec auth-cf6f45fb-9k678 -c microsimserver curl http://db:8080
<snip>
2S76Q83lFt3eplRkAHoHkqUl1PhX
Tue Jan 28 03:47:03 2020   hostname: db-55747d84d8-9bhwx   ip: 10.8.1.5   remote: 127.0.0.1   hostheader: 127.0.0.1:8080   path: /

And here is the connection displayed in Kiali:

So the good news is that the connection is encrypted. The bad news is that auth shouldn’t be able to communicate with db. Let’s implement micro-segmentation.

The first step is to enforce strict mTLS and enable Role Based Access Control (RBAC) for the default namespace. First copy/paste the manifest to a file called istio-mtls-strict.yaml with vi. Let’s take a look at the configuration:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "default"
spec:
  peers:
  - mtls:
      mode: STRICT
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "default"
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: 'ON_WITH_INCLUSION'
  inclusion:
    namespaces: ["default"]

The important bits here are:

  • Line 9: mode: STRICT in the Policy, which disallows any plaintext communications
  • Line 27: mode: 'ON_WITH_INCLUSION', which requires RBAC policies to be satisfied before allowing connections between services for the namespaces defined in line 29
  • Line 29: namespaces: ["default"], which are the namespaces that have the RBAC policies applied

Let’s apply this by deleting the old config and applying the new one:

$ kubectl delete -f istio-mtls-permissive.yaml
policy.authentication.istio.io "default" deleted
destinationrule.networking.istio.io "default" deleted

$ kubectl apply -f istio-mtls-strict.yaml
policy.authentication.istio.io/default created
destinationrule.networking.istio.io/default created
clusterrbacconfig.rbac.istio.io/default created

Hmm… the entire application is broken now. No worries – this is expected! We did this to illustrate that policies need to be explicitly defined to allow any service-to-service (East/West) communications.

Let’s add one service at a time to see these policies in action. Copy paste this manifest to a file called istio-rbac-policy-test.yaml with vi:

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: www-access-role
  namespace: default
spec:
  rules:
  - services: ["db.default.svc.cluster.local"]
    methods: ["GET", "POST"]
    paths: ["*"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: www-to-db
  namespace: default
spec:
  subjects:
  - user: "cluster.local/ns/default/sa/www"
  roleRef:
    kind: ServiceRole
    name: "www-access-role"

Remember those serviceAccounts we created in the beginning? Now we are tying them to an RBAC policy. In this case we are allowing GET and POST requests to db.default.svc.cluster.local from Pods that offer client certificates identifying themselves as www.

The user field takes an entry in the form of cluster.local/ns/<namespace>/sa/<serviceAcountName>. In this case cluster.local/ns/default/sa/www refers to the www Service Account we created earlier.

Let’s apply this:

$ kubectl apply -f istio-rbac-policy-test.yaml
servicerole.rbac.istio.io/www-access-role created
servicerolebinding.rbac.istio.io/www-to-db created

It worked! www can now talk to db. Now we can fix auth by updating the policy to look like this:

spec:
  rules:
  - services: ["db.default.svc.cluster.local", "auth.default.svc.cluster.local"]

Let’s do that, plus allow the Istio Ingress Gateway service istio-ingressgateway-service-account to access www. This will allow public access to the service when we configure the Ingress Gateway later. Copy/paste this manifest to a file called istio-rbac-policy-final.yaml and apply it:

$ kubectl delete -f istio-rbac-policy-test.yaml
servicerole.rbac.istio.io "www-access-role" deleted
servicerolebinding.rbac.istio.io "www-to-db" deleted

$ kubectl apply -f istio-rbac-policy-final.yaml
servicerole.rbac.istio.io/www-access-role created
servicerolebinding.rbac.istio.io/www-to-db created
servicerole.rbac.istio.io/pub-access-role created
servicerolebinding.rbac.istio.io/pub-to-www created

Very good! We’re back up and running. Let’s verify that micro-segmentation is in place and that requests cannot get through even by using IP addresses instead of Service names. We’ll try connecting from an auth Pod to a db Pod:

$ kubectl exec auth-cf6f45fb-9k678 -c microsimserver curl http://db:8080
RBAC: access denied

$ kubectl exec auth-cf6f45fb-9k678 -c microsimserver curl 10.4.3.10:8080
upstream connect error or disconnect/reset before headers. reset reason: connection termination

Success!

Exposing the App to the Internet

Now that we have secured the app internally, we can expose it to the internet. If you try to visit the site now it will fail since the Istio Ingress has not been configured to forward traffic to the www service.

In Cloud Shell, copy/paste this manifest to a file called istio-ingress.yaml with vi:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: www-gateway
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
    release: istio
  servers:
  - port:
      number: 80
      name: http2
      protocol: HTTP2
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: www-vservice
spec:
  hosts:
  - "*"
  gateways:
  - www-gateway
  http:
  - match:
    - uri:
        prefix: "/"
    route:
    - destination:
        port:
          number: 8080
        host: www.default.svc.cluster.local

Here we’re telling Istio Ingress to listen on port 80 using HTTP2 protocol and then we attach our www service to that gateway. We allowed the Ingress Gateway to communicate with the www service earlier via RBAC policy so we should be good to apply this:

$ kubectl apply -f istio-ingress.yaml
gateway.networking.istio.io/www-gateway created
virtualservice.networking.istio.io/www-vservice created

Now we should be able to reach the application from the internet:

$ kubectl get services -n istio-system
NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                                                                                                      AGE
grafana                  ClusterIP      10.70.12.231   <none>          3000/TCP                                                                                                                                     83m
istio-citadel            ClusterIP      10.70.2.197    <none>          8060/TCP,15014/TCP                                                                                                                           87m
istio-galley             ClusterIP      10.70.11.184   <none>          443/TCP,15014/TCP,9901/TCP                                                                                                                   87m
istio-ingressgateway     LoadBalancer   10.70.10.196   34.68.212.250   15020:30100/TCP,80:31596/TCP,443:32314/TCP,31400:31500/TCP,15029:32208/TCP,15030:31368/TCP,15031:31242/TCP,15032:31373/TCP,15443:30451/TCP   87m
istio-pilot              ClusterIP      10.70.3.210    <none>          15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       87m
istio-policy             ClusterIP      10.70.4.74     <none>          9091/TCP,15004/TCP,15014/TCP                                                                                                                 87m
istio-sidecar-injector   ClusterIP      10.70.3.147    <none>          443/TCP                                                                                                                                      87m
istio-telemetry          ClusterIP      10.70.10.55    <none>          9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       87m
kiali                    ClusterIP      10.70.15.2     <none>          20001/TCP                                                                                                                                    86m
prometheus               ClusterIP      10.70.7.187    <none>          9090/TCP                                                                                                                                     84m
promsd                   ClusterIP      10.70.8.70     <none>          9090/TCP     

$ curl 34.68.212.250
<snip>
ja1IO2Hm2GJAqKBPao2YyccDAVrd
Wed Jan 29 01:24:46 2020   hostname: www-74f9dc9df8-j54k4   ip: 10.4.3.9   remote: 127.0.0.1   hostheader: 127.0.0.1:8080   path: /

Excellent! Our simple App is secured internally and exposed to the Internet.

Conclusion

I really enjoyed this challenge and I see great potential in using a Service Mesh along with a security sidecar proxy like modsecurity. Though, I have to say that things are changing quickly, including the best practices and configuration syntax.

For example, in this proof of concept I used the default version of Istio that was installed on my GKE cluster (1.1.16) which already seems old since version 1.4 has deprecated the RBAC configuration I used for a new style called AuthorizationPolicy. Unfortunately, this option was not available in my version of Istio but it does look more straightforward than RBAC.

There is a great deal more complexity in a Service Mesh deployment and troubleshooting connectivity issues can be difficult.

One thing that would probably need to be addressed in a production environment would be the Envoy proxy sidecar configuration. In my simple scenario I was getting very strange connectivity results until I exposed port 8080 on the microsimserver container in the Deployment. Without that configuration (which worked fine without Istio) Envoy didn’t properly grab all of the ports, so it was possible to completely bypass Envoy altogether which meant broken micro-segmentation and WAF bypass when connecting directly to the Pod IP address.

There is a traffic management configuration called sidecar which allows you to fine-tune how the Envoy sidecar configures itself. Fortunately, I ended up not needing to do this in this example, though I did go through some iterations of experimenting with it to get micro-segmentation working without exposing port 8080 on the Pod.

So in the end, the Service Mesh Sidecar-on-Sidecar Pattern may work for you, but you might end up tearing out a fair bit of your hair getting it to work in your environment.

I’m looking forward to doing a proof of concept of the Service Mesh Security Plugin Pattern in the future, which will require compiling a custom version of Envoy that automatically filters traffic through modsecurity. I may let the versions of Istio and Envoy mature a bit before attempting that, though.

What do you think about the Sidecar-on-Sidecar Pattern?

Featured

Explaining Kubernetes to a Five Year Old

A friend of mine pointed me to a twitter thread on how to explain Kubernetes to a five year old. Since I have a two year old, this immediately popped into my head.

I’ve seen the Lonely Goatherd scene from The Sound of Music many a time – my daughter absolutely loves it. And it seems to be a fairly good explanation for Kubernetes. Hear me out:

Stage = Kubernetes Cluster

The stage is the Kubernetes cluster where the application is deployed. This includes the Nodes, environment, config maps, secrets, etc.

Puppets = Containers/Pods/Microservices

The puppets are the actual microservices made up of Pods and Containers.

Julie Andrews = DevOps

Julie Andrews (Maria) is the poor DevOps soul who is staving off disaster with kubectl, helm charts, APIs, etc.

Kids = Kubernetes Scheduler

The Kids are (mostly) doing what Julie (DevOps) is telling them to do. They are adding and removing the puppets (containers) as she has directed.

Audience = End Users

The Audience is the end users of the application… but let’s not kid ourselves – this app is not in production, so the audience is really QA. 🙂

Featured

Silly Terminal Plotting with jc, jq, and jp

I ran across a cool little utility called jp that takes JSON input and plots bar, line, histogram, and scatterplot graphs right in the terminal. I’m always looking for ways to hone my jq skills so I found some time to play around with it.

I figured it would be fun to plot some system stats like CPU and Memory utilization in the terminal, so I started with some simple plots piping jc, and jp together. For example, here’s a bar graph of the output of df:

df | jc --df | jp -type bar -canvas full-escape -x ..filesystem -y ..used

Not super useful, but we’re just having fun here! How about graphing the relative sizes of files in a directory using ls?

ls -l Documents/lab\ license/ | jc --ls | jp -type bar -canvas full-escape -x ..filename -y ..size

Not bad! Let’s get a little fancier by filtering results through jq. We’ll plot the output of ps to see the CPU utilization of processes with more than .5% CPU utilization:

ps axu | jc --ps | jq '[.[] | select (.cpu_percent > 0.5)]' | jp -type bar -canvas full-escape -x ..pid -y ..cpu_percent

That’s a nice static bar chart of the most active PIDs on the system. But we can do better. Let’s make the graph dynamic by enclosing the above in a while true loop:

while true; do ps axu | jc --ps | jq '[.[] | select (.cpu_percent > 0.5)]' | jp -type bar -canvas full-escape -x ..pid -y ..cpu_percent; sleep 3; done

Fancy! Of course we could have plotted mem_percent instead to plot memory utilization by PID. By the way, I made the animated GIF above using ttyrec and ttygif.

Ok, one last dynamic graph. This time, let’s track system load over time using the output of uptime. To pull this off we’ll need to keep a history of load values over time, so we’ll move from a one-liner to a small bash script:

#!/bin/bash

rm /tmp/load.json
SECONDS=0

while true; do 

    uptime | jc --uptime | jq --arg sec "$SECONDS" '{"seconds": $sec | tonumber, "load": .load_1m}' >> /tmp/load.json
    cat /tmp/load.json | jq -s . | jp -canvas full-escape -x ..seconds -y ..load
    sleep 2

done

Fun! We got to do a couple of neat things with jq here.

We pulled in the uptime output converted to JSON with jc and rebuilt the JSON to use only the load_1m value and the SECONDS environment variable. We used tonumber to convert the SECONDS variable into a number that could be plotted by jp. We redirect the output to a temporary text file called /tmp/load.json so jp can read it later and build out the line graph.

I know, I know – I’m piping cat output into jq but I just wanted to make the script readable. The interesting thing here is that we are using the -s or “slurp” option of jq, which essentially reformats the JSON lines output in /tmp/load.json into a proper JSON array so jp can consume it.

By the way, the graphs animate a little nicer in real life since you don’t get the artificial delay between frames you see in the animated GIF.

I thought that was pretty fun and I got to try a couple different things in jq I haven’t tried before. Happy JSON plotting!

Featured

Microservice Security Design Patterns for Kubernetes (Part 4)

The Security Sidecar Pattern

In Part 3 of my series on Microservice Security Patterns for Kubernetes we dove into the Security Service Layer Pattern and configured a working application with micro-segmentation enforcement and deep inspection for application-layer protection. We were able to secure the application with that configuration, but, as we saw, the micro-segmentation configuration can get a bit unwieldy when you have more than a couple services.

In this post we’ll configure a Security Sidecar Pattern which will provide the same level of security but with a simpler configuration. I really like the Security Sidecar Pattern because it tightly couples the application security layer with the application without requiring any changes to the application.

This also means you can scale the application and your security together, so you don’t have to worry about scaling the security layer separately as your application needs grow. The only downside to this is that the application security layer (we’ll be using the Modsecurity WAF) may be overprovisioned and could waste cluster resources if not kept in check.

Let’s find out how the Security Sidecar Pattern works.

Sidecar where art thou?

One of the really cool things about Kubernetes is that the smallest workload unit is a Pod and a Pod can be made up of multiple containers. Even better, these containers share the loopback network interface. (127.0.0.1) This means you can communicate between containers using normal network protocols without needing to expose these ports to the rest of the cluster.

In practice, what this means is that you can deploy a reverse proxy, such as the one we have been using in Part 3, but instead of setting the origin server as the Kubernetes cluster DNS name of the service, we can just use localhost or 127.0.0.1. Pretty neat!

Sidecar Injection

Another cool thing about Pods is that there are multiple ways to define how the containers within the Pod are defined. In the most basic scenario (and the one we will be deploying in this post) you can simply manually define the application and the WAF container in the Deployment YAML.

But there are fancier ways to automatically inject a sidecar container, like the WAF, by using Mutating Webhooks. Some examples of how this can be done can be found here and here. The nice thing about automatic sidecar injection is that the developers or DevOps team can define their Deployment YAML per usual and the sidecar will be injected without them needing to change their process. Automatic application layer protection!

One more thing about automatic sidecar injection – this is how the Envoy dataplane proxy sidecar is typically injected in an Istio Service Mesh deployment. Istio has its own sidecar injection service, but you can also manually configure the Envoy sidecar if you would like.

The Security Sidecar Pattern

Let’s dive in and see how to configure the Security Sidecar Pattern. We will be using the same application that we set up in Part 2, so go ahead and take a look there to refresh your memory on how things are set up. Here is the diagram:

Figure 1: Insecure Application

As demonstrated before, all microsim services can communicate with each other and there is no deep inspection implemented to block application layer attacks like SQLi. In this post, we will be implementing this sidecar.yaml deployment that adds modsecurity reverse proxy WAF containers with the Core Rule Set as sidecars in front of the microsim services. modsecurity will perform deep inspection on the JSON/HTTP traffic and block application layer attacks.

Then we will add on a Kubernetes Network Policy to enforce segmentation between the services.

Security Sidecar Pattern Deployment Spec

We’ll immediately notice how much smaller and simpler the Security Sidecar Pattern configuration is compared to the Security Service Layer Pattern. We went from 238 lines of configuration down to 142!

Instead of creating separate security deployments and services to secure the application like we did in the Security Service Layer Pattern, we will simply add the WAF container to the same Pod as the application. We will need to make sure the WAF and the application listen on different TCP Ports since they share the loopback interface which doesn’t allow overlapping ports.

In this case, the WAF will become the front-end and will be listening on behalf of the application and will forward on the clean, inspected traffic to the application via the loopback interface. We will only need to expose the WAF listening port to the cluster. Since we don’t want to allow bypassing the WAF we don’t want to expose the application port directly any longer.

Note: Container TCP and UDP ports are still accessible via IP within the Kubernetes cluster even if they are not explicitly configured in the deployment YAML via containerPort configuration. To completely lock down direct access to the application TCP port so the WAF cannot be bypassed we will need to configure Network Policy.

Figure 2: Security Sidecar Pattern

Let’s take a closer look at the spec.

www Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: www
spec:
  replicas: 3
  selector:
    matchLabels:
      app: www
  template:
    metadata:
      labels:
        app: www
    spec:
      containers:
      - name: modsecurity
        image: owasp/modsecurity-crs:v3.2-modsec2-apache
        ports:
        - containerPort: 80
        env:
        - name: SETPROXY
          value: "True"
        - name: PROXYLOCATION
          value: "http://127.0.0.1:8080/"
      - name: microsimserver
        image: kellybrazil/microsimserver
        env:
        - name: STATS_PORT
          value: "5000"
      - name: microsimclient
        image: kellybrazil/microsimclient
        env:
        - name: STATS_PORT
          value: "5001"
        - name: REQUEST_URLS
          value: "http://auth.default.svc.cluster.local:8080/,http://db.default.svc.cluster.local:8080/"
        - name: SEND_SQLI
          value: "True"

We see three replicas of the www pods that are made up of both the official OWASP modsecurity container available on Docker Hub configured as a reverse proxy WAF listening on TCP port 80. The microsimserver application container listening on TCP port 8080 remains unchanged. Note that it is important that services listen on different ports since they are sharing the same loopback interface in the Pod.

All requests that go to the WAF containers will be inspected and proxied to the microsimserver application container within the same Pod at http://127.0.0.1:8080/.

These WAF containers are effectively impersonating the original service so the user or application does not need to modify its configuration. One nice thing about this design is that it allows you to scale the security layer along with the application, so as you scale up the application, security scales along with it automatically.

The microsimclient container configuration remains unchanged from the original, which is nice. This shows that you can implement the Security Sidecar Pattern with little to no application logic changes if you are careful about how you set up the ports.

Now, let’s take a look at the www Service that points to this deployment.

www Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: www
  name: www
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 8080
    targetPort: 80
  selector:
    app: www
  sessionAffinity: None
  type: LoadBalancer

Here we are just forwarding TCP port 8080 application traffic to TCP port 80 on the www Pods since that is the port the modsecurity reverse proxy containers listen on. Since this is an externally facing service we are using type: LoadBalancer and externalTrafficPolicy: Local just like the original Service did.

Next we’ll take a look at the internal microservices. Since the auth and db deployments and services are configured identically we’ll just go over the db configuration.

db Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
spec:
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: modsecurity
        image: owasp/modsecurity-crs:v3.2-modsec2-apache
        ports:
        - containerPort: 80
        env:
        - name: SETPROXY
          value: "True"
        - name: PROXYLOCATION
          value: "http://127.0.0.1:8080/"
      - name: microsimserver
        image: kellybrazil/microsimserver
        env:
        - name: STATS_PORT
          value: "5000"

Again, we have just added the modsecurity WAF container to the Pod listening on TCP Port 80. Since this is different than the listening port of the microsimserver container we are good to go without any changes to the app. Just like on the www Deployment, we have configured the modsecurity reverse proxy to send inspected traffic locally within the Pod to http://127.0.0.1:8080/.

Note that even though we aren’t explicitly configuring the microsimserver TCP port 8080 via containerPort in the Deployment spec, this port is still technically available on the cluster via direct IP access. To fully lock down connectivity, we will be using Network Policy later on.

db Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: db
  name: db
spec:
  ports:
  - port: 8080
    targetPort: 80
  selector:
    app: db
  sessionAffinity: None

Nothing fancy here – just listening on TCP port 8080 and forwarding to port 80, which is what the modsecurity WAF containers listen on. This is an internal service so no need for type: LoadBalancer or externalTrafficPolicy: Local.

Now that we understand how the Deployment and Service specs work, let’s apply them on our Kubernetes cluster.

See Part 2 for more information on setting up the cluster.

Applying the Deployments and Services

First, let’s delete the original insecure deployment in Cloud Shell if it is still running:

$ kubectl delete -f simple.yaml

Your Pods, Deployments, and Services should be empty before you proceed:

$ kubectl get pods
No resources found.
$ kubectl get deploy
No resources found.
$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.12.0.1    <none>        443/TCP   3m46s

Next, copy/paste the deployment text into a file called sidecar.yaml using vi. Then apply the deployment with kubectl:

$ kubectl create -f sidecar.yaml
deployment.apps/www created
deployment.apps/auth created
deployment.apps/db created
service/www created
service/auth created
service/db created

Testing the Deployment

Once the www service has an external IP, you can send an HTTP GET or POST request to it from Cloud Shell or your laptop:

$ kubectl get services
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
auth         ClusterIP      10.12.7.96    <none>          8080/TCP         90m
db           ClusterIP      10.12.8.118   <none>          8080/TCP         90m
kubernetes   ClusterIP      10.12.0.1     <none>          443/TCP          93m
www          LoadBalancer   10.12.14.67   35.238.35.208   8080:32032/TCP   90m
$ curl 35.238.35.208:8080
...vME2NtSGaTBnt2zsprKdes5KKXCCAG9pk0yUr4K
Thu Jan  9 22:09:27 2020   hostname: www-5bfc744996-tdzsk   ip: 10.8.2.3   remote: 127.0.0.1   hostheader: 127.0.0.1:8080   path: /

The originating IP address is now the IP address of the local WAF in the Pod that handled the request. (always 127.0.0.1, since it is a sidecar). Since the WAF is deployed as a reverse proxy, the only way to get the originating IP information will be via HTTP headers, such as X-Forwarded-For (XFF). Also, the host header has now changed, so keep this in mind if the application is expecting certain values in the headers.

We can do a quick check to see if the modsecurity WAF is inspecting traffic by sending an HTTP POST request to an IP address with no data or size information. This will be seen as an anomalous request and blocked:

$ curl -X POST 35.238.35.208:8080
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.<br />
</p>
</body></html>

Excellent! Now let’s take a look at the microsim stats to see if the WAF layers are blocking the East/West SQLi attacks. Let’s open two tabs in Cloud Shell: one for shell access to a www microsimclient container and another for shell access to a db microsimserver container.

In the first tab, use kubectl to find the name of one of the www pods and shell into the microsimclient container running in it:

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
auth-7559599f89-d8tnw   2/2     Running   0          102m
auth-7559599f89-k8qht   2/2     Running   0          102m
auth-7559599f89-wfbp4   2/2     Running   0          102m
db-59f8d84df-4kbvg      2/2     Running   0          102m
db-59f8d84df-5csh8      2/2     Running   0          102m
db-59f8d84df-ncksp      2/2     Running   0          102m
www-5bfc744996-6jbr7    3/3     Running   0          102m
www-5bfc744996-bgh9h    3/3     Running   0          102m
www-5bfc744996-tdzsk    3/3     Running   0          102m
$ kubectl exec www-5bfc744996-6jbr7 -c microsimclient -it sh
/app #

Then curl to the microsimclient stats server on localhost:5001:

/app # curl localhost:5001
{
  "time": "Thu Jan  9 22:23:25 2020",
  "runtime": 6349,
  "hostname": "www-5bfc744996-6jbr7",
  "ip": "10.8.0.4",
  "stats": {
    "Requests": 6320,
    "Sent Bytes": 6547520,
    "Received Bytes": 112275897,
    "Internet Requests": 0,
    "Attacks": 64,
    "SQLi": 64,
    "XSS": 0,
    "Directory Traversal": 0,
    "DGA": 0,
    "Malware": 0,
    "Error": 0
  },
  "config": {
    "STATS_PORT": 5001,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "REQUEST_URLS": "http://auth.default.svc.cluster.local:8080/,http://db.default.svc.cluster.local:8080/",
    "REQUEST_INTERNET": false,
    "REQUEST_MALWARE": false,
    "SEND_SQLI": true,
    "SEND_DIR_TRAVERSAL": false,
    "SEND_XSS": false,
    "SEND_DGA": false,
    "REQUEST_WAIT_SECONDS": 1.0,
    "REQUEST_BYTES": 1024,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0,
    "REQUEST_PROBABILITY": 1.0,
    "EGRESS_PROBABILITY": 0.1,
    "ATTACK_PROBABILITY": 0.01
  }
}

Here we see 64 SQLi attacks have been sent to the auth and db services in the last 6349 seconds.

Now, let’s see if the attacks are getting through like they did in the insecure deployment. In the other tab, find the name of one of the db pods and shell into the microsimserver container running in it:

$ kubectl exec db-59f8d84df-4kbvg -c microsimserver -it sh
/app #
/app # curl localhost:5000
{
  "time": "Thu Jan  9 22:39:30 2020",
  "runtime": 7316,
  "hostname": "db-59f8d84df-4kbvg",
  "ip": "10.8.0.5",
  "stats": {
    "Requests": 3659,
    "Sent Bytes": 60563768,
    "Received Bytes": 3790724,
    "Attacks": 0,
    "SQLi": 0,
    "XSS": 0,
    "Directory Traversal": 0
  },
  "config": {
    "LISTEN_PORT": 8080,
    "STATS_PORT": 5000,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "RESPOND_BYTES": 16384,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0
  }

In the insecure deployment we saw the SQLi value incrementing. Now that the modsecurity WAF is inspecting the East/West traffic, the SQLi attacks are no longer getting through, though we still see normal RequestsSent Bytes, and Received Bytes incrementing.

modsecurity Logs

Now, let’s check the modsecurity logs to see how the East/West application attacks are being identified. To see the modsecurity audit log we’ll need to shell into one of the WAF containers and look at the /var/log/modsec_audit.log file:

$ kubectl exec db-59f8d84df-4kbvg -c modsecurity -it sh
# grep -C 60 sql /var/log/modsec_audit.log
<snip>
--a05a312e-A--
[09/Jan/2020:23:41:46 +0000] Xhe6OmUpgBRl4hgX8QIcmAAAAIE 10.8.0.4 50990 10.8.0.5 80
--a05a312e-B--
GET /?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1-- HTTP/1.1
Host: db.default.svc.cluster.local:8080
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

--a05a312e-F--
HTTP/1.1 403 Forbidden
Content-Length: 209
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--a05a312e-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.<br />
</p>
</body></html>

--a05a312e-H--
Message: Warning. Pattern match "(?i:(?:[\"'`](?:;?\\s*?(?:having|select|union)\\b\\s*?[^\\s]|\\s*?!\\s*?[\"'`\\w])|(?:c(?:onnection_id|urrent_user)|database)\\s*?\\([^\\)]*?|u(?:nion(?:[\\w(\\s]*?select| select @)|ser\\s*?\\([^\\)]*?)|s(?:chema\\s*?\\([^\\)]*?|elect.*?\\w?user\\()|in ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "190"] [id "942190"] [msg "Detects MSSQL code execution and information gathering attempts"] [data "Matched Data: UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Message: Warning. Pattern match "(?i:(?:^[\\W\\d]+\\s*?(?:alter\\s*(?:a(?:(?:pplication\\s*rol|ggregat)e|s(?:ymmetric\\s*ke|sembl)y|u(?:thorization|dit)|vailability\\s*group)|c(?:r(?:yptographic\\s*provider|edential)|o(?:l(?:latio|um)|nversio)n|ertificate|luster)|s(?:e(?:rv(?:ice|er)| ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "471"] [id "942360"] [msg "Detects concatenated basic SQL injection and SQLLFI attempts"] [data "Matched Data: ;UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "91"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "86"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10 - SQLI=10,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 10, 0, 0, 0"] [tag "event-correlation"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.0.4] ModSecurity: Warning. Pattern match "(?i:(?:[\\\\"'`](?:;?\\\\\\\\s*?(?:having|select|union)\\\\\\\\b\\\\\\\\s*?[^\\\\\\\\s]|\\\\\\\\s*?!\\\\\\\\s*?[\\\\"'`\\\\\\\\w])|(?:c(?:onnection_id|urrent_user)|database)\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?|u(?:nion(?:[\\\\\\\\w(\\\\\\\\s]*?select| select @)|ser\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?)|s(?:chema\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?|elect.*?\\\\\\\\w?user\\\\\\\\()|in ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "190"] [id "942190"] [msg "Detects MSSQL code execution and information gathering attempts"] [data "Matched Data: UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "Xhe6OmUpgBRl4hgX8QIcmAAAAIE"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.0.4] ModSecurity: Warning. Pattern match "(?i:(?:^[\\\\\\\\W\\\\\\\\d]+\\\\\\\\s*?(?:alter\\\\\\\\s*(?:a(?:(?:pplication\\\\\\\\s*rol|ggregat)e|s(?:ymmetric\\\\\\\\s*ke|sembl)y|u(?:thorization|dit)|vailability\\\\\\\\s*group)|c(?:r(?:yptographic\\\\\\\\s*provider|edential)|o(?:l(?:latio|um)|nversio)n|ertificate|luster)|s(?:e(?:rv(?:ice|er)| ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "471"] [id "942360"] [msg "Detects concatenated basic SQL injection and SQLLFI attempts"] [data "Matched Data: ;UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "Xhe6OmUpgBRl4hgX8QIcmAAAAIE"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.0.4] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "91"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "Xhe6OmUpgBRl4hgX8QIcmAAAAIE"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.0.4] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "86"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10 - SQLI=10,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 10, 0, 0, 0"] [tag "event-correlation"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "Xhe6OmUpgBRl4hgX8QIcmAAAAIE"]
Action: Intercepted (phase 2)
Apache-Handler: proxy-server
Stopwatch: 1578613306195047 3522 (- - -)
Stopwatch2: 1578613306195047 3522; combined=2944, p1=904, p2=1734, p3=0, p4=0, p5=306, sr=353, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.2.0.
Server: Apache
Engine-Mode: "ENABLED"

--a05a312e-Z--

Here we see modsecurity has blocked and logged the East/West SQLi attack from one of the www Pods to a db Pod. Sweet!

Yet, we’re still not done. Even though we are now inspecting and protecting traffic at the application layer, we are not yet enforcing micro-segmentation between the services. That means that, even with the WAFs in place, any auth Pod can communicate with any db Pod. We can demonstrate this by opening a shell on any auth microsimserver container and attempting to send a request to a db Pod from it:

/app # curl 'http://db:8080'
...JsHT4A8GK8H0Am47jSG7MppM3o7BOlTrRZl4EEA9bNzsjND
Thu Jan  9 23:57:54 2020   hostname: db-59f8d84df-5csh8   ip: 10.8.2.5   remote: 127.0.0.1   hostheader: 127.0.0.1:8080   path: /

Even worse, if I know the IP address of the db pod, I can even bypass the WAF and send a successful SQLi attack:

/app # curl 'http://10.8.2.5:8080/?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--'
...7Z7Kw2JxEgXipBnDZyyoZI4TK3RswBuZ509y2WY1wJTsERJFoRW6ZYY1QiA
Fri Jan 10 00:01:37 2020   hostname: db-59f8d84df-5csh8   ip: 10.8.2.5   remote: 10.8.2.4   hostheader: 10.8.2.5:8080   path: /?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--

Not good! Now, let’s add Network Policy to provide micro-segmentation and button this thing up.

Adding Micro-segmentation

Here is a simple Network Policy spec that will control the ingress to each internal service. I tried to keep the rules simple, but in a production deployment a tighter policy would likely be desired. For example, you would probably also want to include Egress policies.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: auth
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: www
    to:
    ports:
    - protocol: TCP
      port: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: www
    to:
    ports:
    - protocol: TCP
      port: 80

Another big difference here is the simplicity of the Network Policy when compared to the Security Service Layer Pattern. We went from 104 lines of configuration down to 41.

This policy is says:

  • On the auth Pods only accept traffic from the www Pods that is destined to TCP port 80
  • On the db Pods only accept traffic from the www Pods that is destined to TCP port 80

Let’s try it out. Copy the Nework Policy text to a file named sidecar-network-policy.yaml in vi and apply the Network Policy to the cluster with kubectl:

$ kubectl create -f sidecar-network-policy.yaml
networkpolicy.networking.k8s.io/auth-ingress created
networkpolicy.networking.k8s.io/db-ingress created

Next, let’s try that simulated SQLi attack again from auth to db:

$ kubectl exec auth-7559599f89-d8tnw -c microsimserver -it sh
/app #
/app # curl 'http://10.8.2.5:8080/?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--'
curl: (7) Failed to connect to 10.8.2.5 port 8080: Operation timed out

Good stuff – no matter how you try to connect from auth to db it will now fail.

Finally, let’s ensure that the rest of the application is still working correctly by checking the db logs. If we are still getting legitimate requests then we should be good to go:

$ kubectl logs -f db-59f8d84df-4kbvg microsimserver
<snip>
127.0.0.1 - - [10/Jan/2020 00:27:57] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2020 00:27:58] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2020 00:27:59] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2020 00:28:02] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2020 00:28:04] "POST / HTTP/1.1" 200 -
{"Total": {"Requests": 6987, "Sent Bytes": 115648879, "Received Bytes": 7235424, "Attacks": 1, "SQLi": 1, "XSS": 0, "Directory Traversal": 0}, "Last 30 Seconds": {"Requests": 15, "Sent Bytes": 248280, "Received Bytes": 15540, "Attacks": 0, "SQLi": 0, "XSS": 0, "Directory Traversal": 0}}
127.0.0.1 - - [10/Jan/2020 00:28:04] "POST / HTTP/1.1" 200 -

The service is still getting requests with the Network Policy in place. We can even see the test SQLi request we sent earlier when we bypassed the WAF, but no SQLi attacks are seen since the Network Policy was applied.

Conclusion

We have successfully secured the intra-cluster service communication (East/West communications) via micro-segmentation and WAF utilizing the Sidecar Security Pattern. This pattern is great for quickly and easily adding security to your cluster without creating a lot of overhead for the developers or DevOps teams. The configuration is also smaller and simpler than the Security Service Layer Pattern. It is also possible to automate the injection of the security sidecar with Mutating Webhooks. The nice thing about this pattern is that the security layer scales alongside the application automatically, though one downside to this pattern is that you could waste cluster resources if the WAF containers are not being fully utilized.

What’s next?

My goal is to demonstrate the Service Mesh Security Plugin Pattern in a future post. There are a couple of commercial and open source projects that provide this option, but it’s still early days in this space. In my opinion this pattern makes the most sense since it tightly integrates security with the cluster and cleanly provides both micro-segmentation and application layer security as code, which is the direction everything is moving.

I’m also looking at implementing a Security Sidecar Pattern in conjunction with Istio Service Mesh. This is effectively a Sidecar on Sidecar Pattern. (The Envoy container and WAF container are both added to the application Pod) We’ll see how that goes, and if successful I’ll write that one up as well.

I hope this series has been helpful and if you have suggestions for future topics, please feel free to let me know!

Next in the series: Part 5

Featured

Tools of the Trade for Security Systems Engineers in 2020

Happy New Year, everyone! As we begin a new decade and I reflect on the last quarter century of networking and security I thought it would be cool to see how the tools of the trade for pre-sales Systems Engineers in the network security field have changed and which tools the SE’s SE will need to be proficient with in 2020.

As an SE in the 90’s and early 2000’s I remember carrying a heavy laptop bag filled with now obsolete dongles, serial converters, null-modem cables, ethernet patch cables and crossover cables, screw drivers, papers and excerpts of manuals. I probably couldn’t get through TSA with that bag these days!

Networking and security has changed so much from those years. My early days were spent learning the opaque details of Windows NT and the black art of IPv4 subnetting (and CIDR!). I was obsessed with linux, OSPF, and BGP and made sure I understood the details of how encryption and key exchanges work for IPSEC VPNs.

Obviously all of those foundational skills have served all of us well, but in the past few years we’ve seen the security industry change quite dramatically. Stateful inspection firewalls have given way to Defense in Depth and Zero Trust, which includes so much more. (EDR, NDR, IPS, VM/Cloud/Micro Services, UEBA, Deception, SOAR… whew!) To that end, here are a few tools that I have added to my toolbox in the past few years that I look for SEs to at least have some familiarity with on my high-performing teams.

Cloud Providers

Every SE should have accounts in all of the major cloud providers. Each has its own flavor, advantages, and APIs. Cloud accounts are perfect for setting up temporary labs to test out a configuration or a quick POC. You never know which combination of providers your customers will be using these days so you really need to be familiar with at least these:

The good news is that all of the providers have free signups and the monthly bill is usually very low for lab usage.

Integrations and Automation

A lot of SEs have at least some background in scripting and programming and those skills are becoming more important now with everything becoming more connected and integrated. Integrations are the name of the game and if you can make a POC successful by building one yourself in a pinch it will make you that much more valuable to the customer and your company.

Python has become so popular in the past few years that it’s definitely something that I look for in SE candidates, but BASH, and PowerShell skills are still very relevant. Extra credit for learning Go! Here are some of the more important tools to help in this area:

  • Proper IDE or text editor (I like Sublime, but there are many options, including old-school vi!)
  • git (open some sort of git account, like github, and share your code, )
    • I’m not a git expert, by any means, so I use Sourcetree to keep me sane
  • SOAR Platforms (Phantom, Demisto, Cybersponse)
    • These typically have free community editions
  • SIEM (Elastic Search, Splunk, etc.)
    • Again, set up the free community editions in your lab

APIs

In line with Integrations and Automation, some of the lower-level skills that will be needed is to understand the different flavors of APIs. You’ll find that RESTful or REST-like APIs are very common these days, which makes things easy, but you’ll definitely need to understand JSON format.

Here are some helpful tools for navigating APIs:

  • Online JSON pretty printer and validator
  • Online encoder/decoder (Cyberchef)
  • Postman – I love using this tool to learn a new API or to share quick python/BASH snippets with a customer.
  • jq – one of my favorite command line tools. It’s like sed or awk for JSON. Also, a quick and dirty JSON pretty printer/validator at the command line.

Containers and Microservices

Don’t worry, all of your legacy networking skills (OSI 1-7) aren’t obsolete, but a lot of the lower levels are becoming more abstracted and more emphasis is being laid on layer 7 for security.

I think it’s a good exercise to write a small, simple app in Python and package it up as a Docker container running standalone or in a Kubernetes cluster. Extra credit for learning Service Mesh technologies like Istio/Envoy and CI/CD Pipelines and tools like Jenkins.

It’s a big topic and a lot of things are changing rapidly, so this is an opportunity to learn something a bit bleeding edge, but quickly becoming mainstream. The SEs that understand these technologies will be the most relevant in 2020 and beyond as their customers transition to them.

To get started, make sure these tools are in your tool belt:

Penetration Testing/Hacking

Of course, we can’t forget the basics of security, including pen testing and hacking tools that will enable you to test and demonstrate your technologies and solutions.

  • netcat (aka ncat or nc) – this is one of the first command line tools I install on my laptop. It’s a Swiss army knife for network testing.
  • nmap – another must have at the command line – tried and true for many years.
  • Kali Linux
  • Application security test tools available from the OWASP site.
  • Virus Total – just be careful you don’t upload sensitive files or compromise an ongoing investigation by uploading a file the incident responders are still reversing.

There are so many more tools for this section but they will typically be dependent on the type of security products you support.

2020 and Beyond!

There’s no shortage of things to learn and tools in the toolbox, though I have noticed that my laptop bag is a lot lighter these days! What are your favorite tools that I have missed?

Featured

Microservice Security Design Patterns for Kubernetes (Part 3)

The Security Service Layer Pattern

In Part 1 of this series on microservices security patterns for Kubernetes we went over three design patterns that enable micro-segmentation and deep inspection of the application and API traffic between microservices:

  1. Security Service Layer Pattern
  2. Security Sidecar Pattern
  3. Service Mesh Security Plugin Pattern

In Part 2 we set up a simple, insecure deployment and demonstrated application layer attacks and the lack of micro-segmentation. In this post we will take that insecure deployment and implement a Security Service Layer Pattern to block application layer attacks and enforce strict segmentation between services.

The Insecure Deployment

Let’s take a quick look at the insecure deployment from Part 2:

Figure 1: Insecure Deployment

insecure deployment

As demonstrated before, all microsim services can communicate with each other and there is no deep inspection implemented to block application layer attacks like SQLi. In this post, we will be implementing this servicelayer.yaml deployment that adds modsecurity reverse proxy WAF Pods with the Core Rule Set in front of the microsim services. modsecurity will perform deep inspection on the JSON/HTTP traffic and block application layer attacks.

Then we will add on a Kubernetes Network Policy to enforce segmentation between the services. In the end, the deployment will look like this:

Figure 2: Security Service Layer Pattern

Security Service Layer Deployment Spec

You’ll notice that each original service has been split into two services: a modsecurity WAF service (in orange) and the original service (in blue). Let’s take a look at the deployment YAML file to understand how this pattern works.

The Security Service Layer Pattern does add quite a bit of lines to our deployment file, but they are simple additions. We’ll just need to keep our port numbers and service names straight as we add the WAF layers into the deployment.

Let’s take a closer look at the components that have changed from the insecure deployment.

www Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: www
spec:
  replicas: 3
  selector:
    matchLabels:
      app: www
  template:
    metadata:
      labels:
        app: www
    spec:
      containers:
      - name: modsecurity
        image: owasp/modsecurity-crs:v3.2-modsec2-apache
        ports:
        - containerPort: 80
        env:
        - name: SETPROXY
          value: "True"
        - name: PROXYLOCATION
          value: "http://wwworigin.default.svc.cluster.local:8080/"

We see three replicas of the official OWASP modsecurity container available on Docker Hub configured as a reverse proxy WAF listening on TCP port 80. All requests that go to any of these WAF instances will be inspected and proxied to the origin service, wwworigin, on TCP port 8080. wwworigin is the original Service and Deployment from the insecure deployment.

These WAF containers are effectively impersonating the original service so the user or application does not need to modify its configuration. One nice thing about this design is that it allows you to scale the security layer independent from the application. For instance, you might only require two modsecurity Pods to secure 10 of your application Pods.

Now, let’s take a look at the www Service that points to this WAF deployment.

www Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: www
  name: www
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: www
  sessionAffinity: None
  type: LoadBalancer

Nothing too fancy here – just forwarding TCP port 80 application traffic to TCP port 80 on the modsecurity WAF Pods since that is the port they listen on. Since this is an externally facing service we are using type: LoadBalancer and externalTrafficPolicy: Local just like the original Service did.

Next, let’s check out the wwworigin Deployment spec where the original application Pods are defined.

wwworigin Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wwworigin
spec:
  replicas: 3
  selector:
    matchLabels:
      app: wwworigin
  template:
    metadata:
      labels:
        app: wwworigin
    spec:
      containers:
      - name: microsimserver
        image: kellybrazil/microsimserver
        env:
        - name: STATS_PORT
          value: "5000"
        ports:
        - containerPort: 8080
      - name: microsimclient
        image: kellybrazil/microsimclient
        env:
        - name: REQUEST_URLS
          value: "http://auth.default.svc.cluster.local:80,http://db.default.svc.cluster.local:80"
        - name: SEND_SQLI
          value: "True"
        - name: STATS_PORT
          value: "5001"

There’s a lot going on here, but basically it’s nearly identical to what we had in the insecure deployment. The only thing that has changed is the name of the deployment from www to wwworigin and we changed the REQUEST_URLS destination ports from 8080 to 80. This is because the modsecurity WAF containers listen on port 80 and they are the true front-end to the auth and db services.

Next, let’s take a look at the wwworigin Service spec.

wwworigin Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: wwworigin
  name: wwworigin
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: wwworigin
  sessionAffinity: None

The only change to the original deployment here is that we changed the name from www to wwworigin and the port from 80 to 8080 since the origin Pods are now internal and not directly exposed to the internet.

Now we need to repeat this process for the auth and db services. Since they are configured the same way, we will only go over the db Deployment and Service. Remember, there is now a db (WAF) and dborigin (application) Deployment and Service that we need to define.

db Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
spec:
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: modsecurity
        image: owasp/modsecurity-crs:v3.2-modsec2-apache
        ports:
        - containerPort: 80
        env:
        - name: SETPROXY
          value: "True"
        - name: PROXYLOCATION
          value: "http://dborigin.default.svc.cluster.local:8080/"

This is essentially the same as the www Deployment except we are proxying to dborigin. The WAF containers listen on port 80 and then they proxy the traffic to port 8080 on the origin application service.

db Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: db
  name: db
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: db
  sessionAffinity: None

Again, nothing fancy here – just listening on TCP port 80, which is what the modsecurity WAF containers listen on. This is an internal service so no need for type: LoadBalancer or externalTrafficPolicy: Local.

Finally, let’s take a look at the dborigin Deployment and Service.

dborigin Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dborigin
spec:
  replicas: 3
  selector:
    matchLabels:
      app: dborigin
  template:
    metadata:
      labels:
        app: dborigin
    spec:
      containers:
      - name: microsimserver
        image: kellybrazil/microsimserver
        ports:
        - containerPort: 8080
        env:
        - name: STATS_PORT
          value: "5000"

This Deployment is essentially the same as the original, except the name has been changed from db to dborigin.

dborigin Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: dborigin
  name: dborigin
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: dborigin
  sessionAffinity: None

Again, the only change from the original here is the name from db to dborigin.

Now that we understand how the Deployment and Service specs work, let’s apply them on our Kubernetes cluster.

See Part 2 for more information on setting up the cluster.

Applying the Deployments and Services

First, let’s delete the original insecure deployment in Cloud Shell if it is still running:

$ kubectl delete -f simple.yaml

Your Pods, Deployments, and Services should be empty before you proceed:

$ kubectl get pods
No resources found.
$ kubectl get deploy
No resources found.
$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.12.0.1    <none>        443/TCP   3m46s

Next, copy/paste the deployment text into a file called servicelayer.yaml using vi. Then apply the deployment with kubectl:

$ kubectl apply -f servicelayer.yaml
deployment.apps/www created
deployment.apps/wwworigin created
deployment.apps/auth created
deployment.apps/authorigin created
deployment.apps/db created
deployment.apps/dborigin created
service/www created
service/auth created
service/db created
service/wwworigin created
service/authorigin created
service/dborigin created

Testing the Deployment

Once the www service has an external IP, you can send an HTTP GET or POST request to it from Cloud Shell or your laptop:

$ kubectl get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
auth         ClusterIP      10.12.14.41    <none>        80/TCP         52s
authorigin   ClusterIP      10.12.5.222    <none>        8080/TCP       52s
db           ClusterIP      10.12.9.224    <none>        80/TCP         52s
dborigin     ClusterIP      10.12.13.80    <none>        8080/TCP       51s
kubernetes   ClusterIP      10.12.0.1      <none>        443/TCP        7m43s
www          LoadBalancer   10.12.13.193   34.66.99.16   80:30394/TCP   52s
wwworigin    ClusterIP      10.12.6.122    <none>        8080/TCP       52s
$ curl 34.66.99.16
...o7yXXg70Olfu2MvVsm9kos8ksEXyzX4oYnZ7wQh29FaqSF
Thu Dec 19 00:58:15 2019   hostname: wwworigin-6c8fb48f79-frmk9   ip: 10.8.1.9   remote: 10.8.0.7   hostheader: wwworigin.default.svc.cluster.local:8080   path: /

You can probably already see some interesting side effects of this deployment. The originating IP address is now the IP address of the WAF that handled the request. (10.8.0.7 in this case). Since the WAF is deployed as a reverse proxy, the only way to get the originating IP information will be via HTTP headers, such as X-Forwarded-For (XFF). Also, the host header has now changed, so keep this in mind if the application is expecting certain values in the headers.

We can do a quick check to see if the modsecurity WAF is inspecting traffic by sending an HTTP POST request with no data or size information. This will be seen as an anomalous request and blocked:

$ curl -X POST http://34.66.99.16
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.<br />
</p>
</body></html>

That looks good! Now let’s take a look at the microsim stats to see if the WAF layers are blocking the East/West SQLi attacks. Let’s open two tabs in Cloud Shell: one for shell access to a wwworigin container and another for shell access to a dborigin container.

In the first tab, use kubectl to find the name of one of the wwworigin pods and shell into the microsimclient container running in it:

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
auth-865675dd7f-4nld7         1/1     Running   0          23m
auth-865675dd7f-7xsks         1/1     Running   0          23m
auth-865675dd7f-lzdzg         1/1     Running   0          23m
authorigin-5f6b795dcd-47gwn   1/1     Running   0          23m
authorigin-5f6b795dcd-r5lr2   1/1     Running   0          23m
authorigin-5f6b795dcd-xb68n   1/1     Running   0          23m
db-dc6f6f5f9-b2j2f            1/1     Running   0          23m
db-dc6f6f5f9-kb5q9            1/1     Running   0          23m
db-dc6f6f5f9-wmj4n            1/1     Running   0          23m
dborigin-7dc8d69f86-6mj2d     1/1     Running   0          23m
dborigin-7dc8d69f86-bvpdn     1/1     Running   0          23m
dborigin-7dc8d69f86-n42vg     1/1     Running   0          23m
www-7cdc675f9-bhrhp           1/1     Running   0          23m
www-7cdc675f9-dldhq           1/1     Running   0          23m
www-7cdc675f9-rlqwv           1/1     Running   0          23m
wwworigin-6c8fb48f79-9tq5t    2/2     Running   0          23m
wwworigin-6c8fb48f79-frmk9    2/2     Running   0          23m
wwworigin-6c8fb48f79-tltzd    2/2     Running   0          23m
$ kubectl exec wwworigin-6c8fb48f79-9tq5t -c microsimclient -it sh
/app #

Then curl to the microsimclient stats server on localhost:5001:

/app # curl localhost:5001
{
  "time": "Thu Dec 19 01:26:24 2019",
  "runtime": 1855,
  "hostname": "wwworigin-6c8fb48f79-9tq5t",
  "ip": "10.8.0.10",
  "stats": {
    "Requests": 1848,
    "Sent Bytes": 1914528,
    "Received Bytes": 30650517,
    "Internet Requests": 0,
    "Attacks": 18,
    "SQLi": 18,
    "XSS": 0,
    "Directory Traversal": 0,
    "DGA": 0,
    "Malware": 0,
    "Error": 0
  },
  "config": {
    "STATS_PORT": 5001,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "REQUEST_URLS": "http://auth.default.svc.cluster.local:80,http://db.default.svc.cluster.local:80",
    "REQUEST_INTERNET": false,
    "REQUEST_MALWARE": false,
    "SEND_SQLI": true,
    "SEND_DIR_TRAVERSAL": false,
    "SEND_XSS": false,
    "SEND_DGA": false,
    "REQUEST_WAIT_SECONDS": 1.0,
    "REQUEST_BYTES": 1024,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0,
    "REQUEST_PROBABILITY": 1.0,
    "EGRESS_PROBABILITY": 0.1,
    "ATTACK_PROBABILITY": 0.01
  }
}

Here we see 18 SQLi attacks have been sent to the auth and db services in the last 1855 seconds.

Now, let’s see if the attacks are getting through like they did in the insecure deployment. In the other tab, find the name of one of the dborigin pods and shell into the microsimserver container running in it:

$ kubectl exec dborigin-7dc8d69f86-6mj2d -c microsimserver -it sh
/app #

Then curl to the microsimserver stats server on localhost:5000:

/app # curl localhost:5000
{
  "time": "Thu Dec 19 01:29:00 2019",
  "runtime": 2013,
  "hostname": "dborigin-7dc8d69f86-6mj2d",
  "ip": "10.8.2.10",
  "stats": {
    "Requests": 1009,
    "Sent Bytes": 16733599,
    "Received Bytes": 1045324,
    "Attacks": 0,
    "SQLi": 0,
    "XSS": 0,
    "Directory Traversal": 0
  },
  "config": {
    "LISTEN_PORT": 8080,
    "STATS_PORT": 5000,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "RESPOND_BYTES": 16384,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0
  }
}

Remember in the insecure deployment we saw the SQLi value incrementing. Now that the modsecurity WAF is inspecting the East/West traffic, the SQLi attacks are no longer getting through, though we still see normal Requests, Sent Bytes, and Received Bytes incrementing.

modsecurity Logs

Let’s check the modsecurity logs to see how the East/West application attacks are being identified. To see the modsecurity audit log we’ll need to shell into one of the WAF containers and look at the /var/log/modsec_audit.log file:

$ kubectl exec db-dc6f6f5f9-b2j2f -it sh
/app # grep -C 60 sql /var/log/modsec_audit.log
<snip>
--fa628b64-A--
[19/Dec/2019:03:06:44 +0000] XfrpRArFgedF@mTDKh9QvAAAAI4 10.8.1.9 60612 10.8.2.9 80
--fa628b64-B--
GET /?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1-- HTTP/1.1
Host: db.default.svc.cluster.local
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

--fa628b64-F--
HTTP/1.1 403 Forbidden
Content-Length: 209
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--fa628b64-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.<br />
</p>
</body></html>

--fa628b64-H--
Message: Warning. Pattern match "(?i:(?:[\"'`](?:;?\\s*?(?:having|select|union)\\b\\s*?[^\\s]|\\s*?!\\s*?[\"'`\\w])|(?:c(?:onnection_id|urrent_user)|database)\\s*?\\([^\\)]*?|u(?:nion(?:[\\w(\\s]*?select| select @)|ser\\s*?\\([^\\)]*?)|s(?:chema\\s*?\\([^\\)]*?|elect.*?\\w?user\\()|in ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "190"] [id "942190"] [msg "Detects MSSQL code execution and information gathering attempts"] [data "Matched Data: UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Message: Warning. Pattern match "(?i:(?:^[\\W\\d]+\\s*?(?:alter\\s*(?:a(?:(?:pplication\\s*rol|ggregat)e|s(?:ymmetric\\s*ke|sembl)y|u(?:thorization|dit)|vailability\\s*group)|c(?:r(?:yptographic\\s*provider|edential)|o(?:l(?:latio|um)|nversio)n|ertificate|luster)|s(?:e(?:rv(?:ice|er)| ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "471"] [id "942360"] [msg "Detects concatenated basic SQL injection and SQLLFI attempts"] [data "Matched Data: ;UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "91"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "86"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10 - SQLI=10,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 10, 0, 0, 0"] [tag "event-correlation"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.1.9] ModSecurity: Warning. Pattern match "(?i:(?:[\\\\"'`](?:;?\\\\\\\\s*?(?:having|select|union)\\\\\\\\b\\\\\\\\s*?[^\\\\\\\\s]|\\\\\\\\s*?!\\\\\\\\s*?[\\\\"'`\\\\\\\\w])|(?:c(?:onnection_id|urrent_user)|database)\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?|u(?:nion(?:[\\\\\\\\w(\\\\\\\\s]*?select| select @)|ser\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?)|s(?:chema\\\\\\\\s*?\\\\\\\\([^\\\\\\\\)]*?|elect.*?\\\\\\\\w?user\\\\\\\\()|in ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "190"] [id "942190"] [msg "Detects MSSQL code execution and information gathering attempts"] [data "Matched Data: UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "XfrpRArFgedF@mTDKh9QvAAAAI4"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.1.9] ModSecurity: Warning. Pattern match "(?i:(?:^[\\\\\\\\W\\\\\\\\d]+\\\\\\\\s*?(?:alter\\\\\\\\s*(?:a(?:(?:pplication\\\\\\\\s*rol|ggregat)e|s(?:ymmetric\\\\\\\\s*ke|sembl)y|u(?:thorization|dit)|vailability\\\\\\\\s*group)|c(?:r(?:yptographic\\\\\\\\s*provider|edential)|o(?:l(?:latio|um)|nversio)n|ertificate|luster)|s(?:e(?:rv(?:ice|er)| ..." at ARGS:password. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "471"] [id "942360"] [msg "Detects concatenated basic SQL injection and SQLLFI attempts"] [data "Matched Data: ;UNION SELECT found within ARGS:password: ;UNION SELECT 1, version() limit 1,1--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.2.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "XfrpRArFgedF@mTDKh9QvAAAAI4"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.1.9] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "91"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "XfrpRArFgedF@mTDKh9QvAAAAI4"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 10.8.1.9] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity.d/owasp-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "86"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10 - SQLI=10,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): individual paranoia level scores: 10, 0, 0, 0"] [tag "event-correlation"] [hostname "db.default.svc.cluster.local"] [uri "/"] [unique_id "XfrpRArFgedF@mTDKh9QvAAAAI4"]
Action: Intercepted (phase 2)
Apache-Handler: proxy-server
Stopwatch: 1576724804853810 2752 (- - -)
Stopwatch2: 1576724804853810 2752; combined=2296, p1=669, p2=1340, p3=0, p4=0, p5=287, sr=173, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/); OWASP_CRS/3.2.0.
Server: Apache
Engine-Mode: "ENABLED"

--fa628b64-Z--

Here we see modsecurity has blocked and logged the East/West SQLi attack from one of the wwworigin containers to a dborigin container. Excellent!

But there’s still a bit more to do. Even though we are now inspecting and protecting traffic at the application layer, we are not yet enforcing micro-segmentation between the services. That means that, even with the WAFs in place, any authorigin container can communicate with any dborigin container. We can demonstrate this by opening a shell on a authorigin container and attempting to send a simulated SQLi to a dborigin container from it:

# curl 'http://dborigin:8080/?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--'
X7fJ4MnlHo5gzJFQ1...
Thu Dec 19 04:54:25 2019   hostname: dborigin-7dc8d69f86-6mj2d   ip: 10.8.2.10   remote: 10.8.2.5   hostheader: dborigin:8080   path: /?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--

Not only can they communicate – we have completely bypassed the WAF! Let’s fix this with Network Policy.

Network Policy

Here is a Network Policy spec that will control the ingress to each internal pod. I tried to keep the rules simple, but in a production deployment a tighter policy would likely be desired. For example, you would probably also want to include Egress policies.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: wwworigin-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: wwworigin
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: www
    to:
    ports:
    - protocol: TCP
      port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: auth
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: wwworigin
    to:
    ports:
    - protocol: TCP
      port: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: wwworigin
    to:
    ports:
    - protocol: TCP
      port: 80
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: authorigin-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: authorigin
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: auth
    to:
    ports:
    - protocol: TCP
      port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dborigin-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: dborigin
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: db
    to:
    ports:
    - protocol: TCP
      port: 8080

Even with a simple Network Policy you can see one of the downsides to the Security Services Layer Pattern: it can be tedious to set the proper micro-segmentation policy without making errors.

Basically what this policy is saying is:

  • On the wwworigin containers only accept traffic from the www containers that is destined to TCP port 8080
  • On the auth containers only accept traffic from the wwworigin containers that is destined to TCP port 80
  • On the db containers only accept traffic from the wwworigin containers that is destined to TCP port 80
  • On the authorigin containers only accept traffic from the auth containers that is destined to TCP port 8080
  • On the dborigin containers only accept traffic from the db containers that is destined to TCP port 8080

Not Fun! In a large deployment with many services, this can quickly get out of hand and errors will be easy to make as you trace the traffic flow between each service. That’s why a Service Mesh is probably a better choice for an application with more than a few services.

So let’s see if this works. Let’s copy the Nework Policy text to a file named servicelayer-network-policy.yaml in vi and apply the Network Policy to the cluster with kubectl:

$ kubectl create -f servicelayer-network-policy.yaml
networkpolicy.networking.k8s.io/wwworigin-ingress created
networkpolicy.networking.k8s.io/auth-ingress created
networkpolicy.networking.k8s.io/db-ingress created
networkpolicy.networking.k8s.io/authorigin-ingress created
networkpolicy.networking.k8s.io/dborigin-ingress created

And now let’s try that simulated SQLi attack again from authorigin to dborigin:

/var/log # curl 'http://dborigin:8080/?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--'
curl: (7) Failed to connect to dborigin port 8080: Operation timed out

Success!

Finally, let’s doublecheck that the rest of the application is still working by checking the dborigin logs. If we are still getting legitimate requests then we should be good to go:

$ kubectl logs -f dborigin-7dc8d69f86-6mj2d
<snip>
10.8.2.6 - - [19/Dec/2019 05:23:26] "POST / HTTP/1.1" 200 -
10.8.2.6 - - [19/Dec/2019 05:23:28] "POST / HTTP/1.1" 200 -
10.8.2.9 - - [19/Dec/2019 05:23:31] "POST / HTTP/1.1" 200 -
10.8.2.6 - - [19/Dec/2019 05:23:33] "POST / HTTP/1.1" 200 -
10.8.0.11 - - [19/Dec/2019 05:23:34] "POST / HTTP/1.1" 200 -
10.8.2.9 - - [19/Dec/2019 05:23:34] "POST / HTTP/1.1" 200 -
10.8.2.6 - - [19/Dec/2019 05:23:35] "POST / HTTP/1.1" 200 -
10.8.2.9 - - [19/Dec/2019 05:23:39] "POST / HTTP/1.1" 200 -
10.8.2.9 - - [19/Dec/2019 05:23:40] "POST / HTTP/1.1" 200 -
10.8.2.9 - - [19/Dec/2019 05:23:41] "POST / HTTP/1.1" 200 -
{"Total": {"Requests": 8056, "Sent Bytes": 133603375, "Received Bytes": 8342908, "Attacks": 1, "SQLi": 1, "XSS": 0, "Directory Traversal": 0}, "Last 30 Seconds": {"Requests": 17, "Sent Bytes": 281932, "Received Bytes": 17612, "Attacks": 0, "SQLi": 0, "XSS": 0, "Directory Traversal": 0}}
10.8.2.6 - - [19/Dec/2019 05:23:43] "POST / HTTP/1.1" 200 -
10.8.2.6 - - [19/Dec/2019 05:23:43] "POST / HTTP/1.1" 200 -

Nice! We see the service is still getting requests with the Network Policy in place. We can even see that test SQLi request we sent earlier when we bypassed the WAF, but no SQLi attacks are seen since the Network Policy was applied.

Conclusion

Whew – that was fun! As you can see, it is possible to lock down an application with just a few microservices that need to communicate with each other using the Security Services Layer Pattern, but for anything more than a few services things can get complicated quickly. It does have the advantage, however, of allowing you to independently scale the security layers and the application layers.

Stay tuned for the next post where we’ll go over the Security Sidecar Pattern and we’ll see the advantages and disadvantages of that approach.

Next in the series: Part 4

Featured

Microservice Security Design Patterns for Kubernetes (Part 2)

Setting Up the Insecure Deployment

In Part 1 of this series on microservices security patterns for Kubernetes we went over three design patterns that enable micro-segmentation and deep inspection of the application and API traffic between microservices:

  1. Security Service Layer Pattern
  2. Security Sidecar Pattern
  3. Service Mesh Security Plugin Pattern

In this post we will set the groundwork to deep dive into the Security Service Layer Pattern with a live insecure deployment on Google Kubernetes Engine (GKE). By the end of this post you will be able to bring up an insecure deployment and demonstrate layer 7 attacks and unrestricted access between internal services. In the next post we will layer on a Security Service Layer Pattern to secure the application.

The Base Deployment

Let’s first get our cluster up and running with a simple deployment with no security and show what is possible in a nearly default state.  We’ll use this simple.yaml deployment I have created using my microsim app. microsim is a microservice simulator that can send simulated JSON/HTTP and application attack traffic between services. It has some logging and statistics reporting functionality that will allow us to see attacks being sent by the client and received or blocked by the server.

Here is a diagram of the deployment.

Figure 1: Simple Deployment

insecure deployment

In this microservice architecture we see three simulated services:

  1. Public Web interface service
  2. Internal Authentication service
  3. Internal Database service

In the default state, all services are able to communicate with one another and there are no protections from application layer attacks. Let’s take a quick look at the Pod Deployments and Services in this application.

www Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: www
spec:
  replicas: 3
  selector:
    matchLabels:
      app: www
  template:
    metadata:
      labels:
        app: www
    spec:
      containers:
      - name: microsimserver
        image: kellybrazil/microsimserver
        env:
        - name: STATS_PORT
          value: "5000"
        ports:
        - containerPort: 8080
      - name: microsimclient
        image: kellybrazil/microsimclient
        env:
        - name: REQUEST_URLS
          value: "http://auth.default.svc.cluster.local:8080,http://db.default.svc.cluster.local:8080"
        - name: SEND_SQLI
          value: "True"
        - name: STATS_PORT
          value: "5001"

In the www deployment above we see three Pod replicas, each running two containers. (microsimserver and microsimclient)

The microsimserver container is configured to expose port 8080, which is the default port the service listens on. By default, the server will respond with 16KB of data and some diagnostic information in either plain HTTP or JSON/HTTP, depending on whether the request is an HTTP GET or POST.

The microsimclient container is configured to send a single 1KB JSON/HTTP POST request every second to http://auth.default.svc.cluster.local:8080 or http://db.default.svc.cluster.local:8080 which will resolve to the internal auth and db Services using the default Kubernetes DNS resolver.

We also see that microsimclient is configured to occasionally send SQLi attack traffic to the auth and db Services. There are many other behaviors that can be configured, but we’ll keep things simple.

The stats server for microsimserver is configured to run on port 5000 and the stats server for microsimclient is configured to run on port 5001. These ports are not exposed to the cluster, so we will need to get shell access to the containers to see the stats.

Now, let’s look at the www service.

www Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: www
  name: www
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: www
  sessionAffinity: None
  type: LoadBalancer

The service is configured to publicly expose the www service via port 80 with a LoadBalancer type. The externalTrafficPolicy: Local option allows the originating IP address to be preserved within the cluster.

Now let’s take a look at the db deployment and service. The auth service is exactly the same as the db service so we’ll skip going over that one.

db Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
spec:
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: microsimserver
        image: kellybrazil/microsimserver
        env:
        - name: STATS_PORT
          value: "5000"
        ports:
        - containerPort: 8080

Just like the www service, there are three Pod replicas, but only one container (microsimserver) runs in each Pod. The default microsimserver listening port of 8080 is exposed and the stats server listens on port 5000, though it is not exposed, so we’ll need to shell into it to view the stats.

And here is the db Service:

db Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: db
  name: db
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: db
  sessionAffinity: None

Since this is an internal service, we are not using the LoadBalancer type, which will cause the Service to be created as a ClusterIP type, nor do we need to define externalTrafficPolicy.

Firing up the Cluster

Let’s bring up the cluster from within the GKE console. Create a standard cluster using the n1-standard-2 machine type with the Enable network policy option checked under the advanced Network security options:

Figure 2: Enable network policy in GKE

enable network policy

Note: you can also create a cluster with network policy enabled at the command line with the --enable-network-policy argument:

$ gcloud container clusters create test --machine-type=n1-standard-2 --enable-network-policy

Once the cluster is up and running, we can spin up the deployment using kubectl locally after configuring it with the gcloud command, or you can use the Google Cloud Shell terminal. For simplicity, let’s use the Cloud Shell and connect to the cluster:

Figure 3: Connect to the Cluster via Cloud Shell

run in Cloud Shell

Within Cloud Shell, copy paste the deployment text into a new file called simple.yaml with vi.

Then create the deployment:

$ kubectl create -f simple.yaml
deployment.apps/www created
deployment.apps/auth created
deployment.apps/db created
service/www created
service/auth created
service/db created

You will see the deployments and services start up. You can verify the application is running successfully with the following commands:

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
auth-5f964774bd-mvtcl   1/1     Running   0          67s
auth-5f964774bd-sn4cw   1/1     Running   0          66s
auth-5f964774bd-xtt54   1/1     Running   0          66s
db-578757bf68-dzjdq     1/1     Running   0          66s
db-578757bf68-kkwzr     1/1     Running   0          66s
db-578757bf68-mlf5t     1/1     Running   0          66s
www-5d89bcb54f-bcjm9    2/2     Running   0          67s
www-5d89bcb54f-bzpwl    2/2     Running   0          67s
www-5d89bcb54f-vbdf6    2/2     Running   0          67s
$ kubectl get deploy
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
auth   3/3     3            3           92s
db     3/3     3            3           92s
www    3/3     3            3           92s
$ kubectl get service
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)        AGE
auth         ClusterIP      10.0.13.227   <none>          8080/TCP       2m1s
db           ClusterIP      10.0.3.1      <none>          8080/TCP       2m1s
kubernetes   ClusterIP      10.0.0.1      <none>          443/TCP        10m
www          LoadBalancer   10.0.6.39     35.188.221.11   80:32596/TCP   2m1s

Find the external address assigned to the www service and send an HTTP GET request to it to verify the service is responding. You can do this from Cloud Shell or your laptop:

$ curl http://35.188.221.11
FPGpqiVZivddHQvkvDHFErFiW2WK8Kl3ky9cEeI7TA6vH8PYmA1obaZGd1AR3avz3SqPZlcrbXFOn3hVlFQdFm9S07ca
<snip>
jYbD5jNA62JEQbUSqk9V0JGgYLATbYe2rv3XeFQIEayJD4qeGnPp7UbEESPBmxrw
Wed Dec 11 20:07:08 2019   hostname: www-5d89bcb54f-vbdf6   ip: 10.56.0.4   remote: 35.197.46.124   hostheader: 35.188.221.11   path: /

You should see a long block of random text and some client and server information on the last line. Notice if you send the request as an HTTP POST the response comes back as JSON. Here I have run the response through jq to pretty-print the response:

$ curl -X POST http://35.188.221.11 | jq .
{
  "data": "hhV9jogGrM7FMxsQCUAcjdsLQRgjgpCoO...",
  "time": "Wed Dec 11 20:14:20 2019",
  "hostname": "www-5d89bcb54f-vbdf6",
  "ip": "10.56.0.4",
  "remote": "46.18.117.38",
  "hostheader": "35.188.221.11",
  "path": "/"
}

Testing the Deployment

Now, let’s prove that any Pod can communicate with any other Pod and that the SQLi attacks are being received by the internal services. We can do this by opening a shell to one of the www pods and one of the db pods.

Open two new tabs in Cloud Shell and find the Pod names from the kubectl get pods command output above.

In one tab, run the following to get a shell on the microsimclient container in the www Pod:

$ kubectl exec www-5d89bcb54f-bcjm9 -c microsimclient -it sh
/app #

In the other tab, run the following to get a shell on the microsimserver container in the db Pod:

$ kubectl exec db-578757bf68-dzjdq -c microsimserver -it sh
/app #

From the microsimclient shell, run the following curl command to see the application stats. This will show us how many normal and attack requests have been sent:

/app # curl http://localhost:5001
{
  "time": "Wed Dec 11 20:21:30 2019",
  "runtime": 1031,
  "hostname": "www-5d89bcb54f-bcjm9",
  "ip": "10.56.1.3",
  "stats": {
    "Requests": 1026,
    "Sent Bytes": 1062936,
    "Received Bytes": 17006053,
    "Internet Requests": 0,
    "Attacks": 9,
    "SQLi": 9,
    "XSS": 0,
    "Directory Traversal": 0,
    "DGA": 0,
    "Malware": 0,
    "Error": 1
  },
  "config": {
    "STATS_PORT": 5001,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "REQUEST_URLS": "http://auth.default.svc.cluster.local:8080,http://db.default.svc.cluster.local:8080",
    "REQUEST_INTERNET": false,
    "REQUEST_MALWARE": false,
    "SEND_SQLI": true,
    "SEND_DIR_TRAVERSAL": false,
    "SEND_XSS": false,
    "SEND_DGA": false,
    "REQUEST_WAIT_SECONDS": 1.0,
    "REQUEST_BYTES": 1024,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0,
    "REQUEST_PROBABILITY": 1.0,
    "EGRESS_PROBABILITY": 0.1,
    "ATTACK_PROBABILITY": 0.01
  }
}

Run the command a few times until you see a number of SQLi attacks have been sent. Here we see that this microsimclient instance has sent 9 SQLi attacks in the last 1031 seconds of runtime.

From the microsimserver shell, curl the server stats to see if any SQLi attacks have been detected:

/app # curl http://localhost:5000
{
  "time": "Wed Dec 11 20:23:52 2019",
  "runtime": 1177,
  "hostname": "db-578757bf68-dzjdq",
  "ip": "10.56.2.11",
  "stats": {
    "Requests": 610,
    "Sent Bytes": 10110236,
    "Received Bytes": 629888,
    "Attacks": 2,
    "SQLi": 2,
    "XSS": 0,
    "Directory Traversal": 0
  },
  "config": {
    "LISTEN_PORT": 8080,
    "STATS_PORT": 5000,
    "STATSD_HOST": null,
    "STATSD_PORT": 8125,
    "RESPOND_BYTES": 16384,
    "STOP_SECONDS": 0,
    "STOP_PADDING": false,
    "TOTAL_STOP_SECONDS": 0
  }
}

Here we see that this particular server has detected two SQLi attacks coming from the clients within the cluster. (East/West traffic) Remember, there are also five other db and auth Pods that are receiving attacks so you will see the attack load shared amongst them.

Let’s also demonstrate that the db server can directly communicate with the auth service:

/app # curl http://auth:8080
firOXAY4hktZLjHvbs41JhReCWHqs... <snip>
Wed Dec 11 20:26:38 2019   hostname: auth-5f964774bd-mvtcl   ip: 10.56.1.4   remote: 10.56.2.11   hostheader: auth:8080   path: /

Since we get a response it is clear that there is no micro-segmentation in place between the db and auth Services and Pods.

Microservice logging

As with most services in Kubernetes, both microsimclient and microsimserver regularly send logs for each request and response to stdout, which means they can be found with the kubectl logs command. Every 30 seconds a JSON summary will also be logged:

microsimclient logs

$ kubectl logs www-5d89bcb54f-bcjm9 microsimclient
2019-12-11T20:04:19   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:20   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:21   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:22   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:23   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:23   SQLi sent: http://auth.default.svc.cluster.local:8080/?username=joe%40example.com&password=%3BUNION+SELECT+1%2C+version%28%29+limit+1%2C1--
2019-12-11T20:04:24   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16574
2019-12-11T20:04:25   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:26   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:27   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:28   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:29   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:30   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:31   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:32   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:33   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:34   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:35   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:36   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:37   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:38   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:39   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:40   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:41   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:42   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:43   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:44   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:45   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
2019-12-11T20:04:46   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:47   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
2019-12-11T20:04:48   Request to http://auth.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16577
{"Total": {"Requests": 30, "Sent Bytes": 31080, "Received Bytes": 497267, "Internet Requests": 0, "Attacks": 1, "SQLi": 1, "XSS": 0, "Directory Traversal": 0, "DGA": 0, "Malware": 0, "Error": 0}, "Last 30 Seconds": {"Requests": 30, "Sent Bytes": 31080, "Received Bytes": 497267, "Internet Requests": 0, "Attacks": 1, "SQLi": 1, "XSS": 0, "Directory Traversal": 0, "DGA": 0, "Malware": 0, "Error": 0}}
2019-12-11T20:04:49   Request to http://db.default.svc.cluster.local:8080/   Request size: 1036   Response size: 16573
...

microsimserver logs

$ kubectl logs db-578757bf68-dzjdq microsimserver
10.56.1.5 - - [11/Dec/2019 20:04:22] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:22] "POST / HTTP/1.1" 200 -
10.56.1.3 - - [11/Dec/2019 20:04:24] "POST / HTTP/1.1" 200 -
10.56.1.5 - - [11/Dec/2019 20:04:25] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:26] "POST / HTTP/1.1" 200 -
10.56.1.5 - - [11/Dec/2019 20:04:27] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:33] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:35] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:41] "POST / HTTP/1.1" 200 -
10.56.0.4 - - [11/Dec/2019 20:04:43] "POST / HTTP/1.1" 200 -
{"Total": {"Requests": 10, "Sent Bytes": 165740, "Received Bytes": 10360, "Attacks": 0, "SQLi": 0, "XSS": 0, "Directory Traversal": 0}, "Last 30 Seconds": {"Requests": 10, "Sent Bytes": 165740, "Received Bytes": 10360, "Attacks": 0, "SQLi": 0, "XSS": 0, "Directory Traversal": 0}}
10.56.1.5 - - [11/Dec/2019 20:04:47] "POST / HTTP/1.1" 200 -
...

You can see how the traffic is automatically being load balanced by the Kubernetes cluster by inspecting the request sources in the microsimserver logs.

Adding Micro-segmentation and Application Layer Protection

Stay tuned for the next post where we will take this simple, insecure deployment, and implement a Security Services Layer pattern. Then we’ll show how the internal application layer attacks are blocked with this approach. Finally, we will demonstrate micro-segmentation which will restrict access between microservices, for example, traffic between the auth and db services.

Note: Depending on your Google Cloud account status you may incur charges for the cluster, so remember to delete it from the GKE console when you are done. You may also need to delete any load balancer objects that were created by the deployment within GCP to avoid residual charges to your account.

Next in the series: Part 3

Featured

Microservice Security Design Patterns for Kubernetes (Part 1)

In this multi-part blog series, I will describe some microservice security design patterns to implement micro-segmentation and deep inspection in the interior of your Kubernetes cluster to further secure your microservice applications, not just the cluster. I will also demonstrate the design patterns with working Proof of Concept deployments that you can use as a starting point.

Follow up posts:
Part 2 Setting up the Insecure Deployment
Part 3 The Security Service Layer Pattern
Part 4 The Security Sidecar Pattern
Part 5 The Service Mesh Sidecar-on-Sidecar Pattern

There are many tutorials on microservice security and how to secure your Kubernetes clusters and they include many of the following topics:

These are worthy topics and they encompass many of the issues that are relevant to securing modern microservice architectures. But there are a couple of important items that I’d like to emphasize:

  • Controlling East/West traffic (layers 3 and 4) between Pods within the Kubernetes cluster (aka micro-segmentation)
  • Deep inspection of the application traffic (layers 5, 6, and 7) between Pods within the Kubernetes cluster (aka IPS or WAF)

These are concepts that have been around for a while in the traditional on-premises and virtualized data center world. Long ago it was recognized that it was no longer adequate to create a hard, crusty edge and leave a soft, gooey interior for attackers to exploit. The attack surface area can include vulnerabilities buried deep inside the application architecture that can be exploited. These include well-known OWASP top 10 web application attacks such as Cross-site Scripting (XSS), SQL Injection, Remote Code Execution (RCE), API attacks, and more.

Let’s discuss some microservice security patterns that can help.

Kubernetes Application Security Patterns

There are three fairly intuitive design patterns that I will be describing:

  1. Security Service Layer Pattern
  2. Security Sidecar Pattern
  3. Service Mesh Security Plugin Pattern

I’ll be using the following simple Kubernetes deployment to show how we can layer micro-segmentation and application inspection within the cluster to provide better microservice security.

Figure 1: Simple Simulated Microservice Deployment

Simple simulated microservice deployment

In this microservice architecture we see three simulated services:

  1. Public Web interface service
  2. Internal Authentication service
  3. Internal Database service

I’m using my microservice traffic and attack generation simulator called microsim to provide a realistic environment with a majority of ‘normal’ JSON/HTTP traffic between services with occasional SQL Injection attack traffic from the WWW service to the internal Auth and DB services.

Now let’s get into the different design patterns.

Security Service Layer Pattern

The Security Service Layer Pattern is probably the simplest to understand, since it is analogous to how micro-segmentation and deep inspection are deployed in traditional environments.

Figure 2: Security Service Layer Pattern

In this design pattern we see the insertion of a security layer in front of each microservice. In this case we are using the official OWASP modsecurity-crs container on Docker Hub. This container provides WAF functionality with the OWASP Core Rule Set and will detect attacks over HTTP, including the simulated SQL Injection attack traffic between microservices. Layer 3 and 4 micro-segmentation is implemented via a network provider that supports Network Policy.

Some of the pros and cons of this design include:

Pros:

  • Simple to understand
  • Allows scaling of the security tiers independent of the microservices they are protecting
  • Treats application security as a microservice
  • No need to change microservice ports

Cons:

  • Creates additional services in the cluster
  • Adds traffic flow complexity
  • Requires more micro-segmentation rules

Security Sidecar Pattern

The Security Sidecar Pattern takes the concept of the Security Service Layer Pattern and collapses the additional services into the microservice Pods.  Sidecar proxy containers, such as modsecurity, can be explicitly configured as part of the Deployment spec or can be injected into the Pods via MutatingWebhook.

Figure 3: Security Sidecar Pattern

In this design pattern we see the insertion of a security proxy container within each Pod, so both the security proxy and application containers are running in the same Pod. In this case we are also using the official OWASP modsecurity-crs container on Docker Hub. Layer 3 and 4 micro-segmentation is implemented via a network provider that supports Network Policy.

Some of the pros and cons of this design include:

Pros:

  • Simple to understand
  • Unifies the scaling of the security and application microservices
  • The security proxy can be automatically injected into the Pod
  • Works with an existing Service Mesh using the Sidecar on Sidecar pattern
  • Requires fewer micro-segmentation rules

Cons:

  • Requires the Security container and Application container to run on different TCP ports within the Pod
  • May result in over-provisioning of the security layer resources

Service Mesh Security Plugin Pattern

The Service Mesh Security Plugin Pattern takes the concept of the Security Sidecar Pattern but implements the security functionality as a plugin to the Service Mesh’s data plane sidecar container (e.g. Envoy in an Istio Service Mesh). 

Figure 4: Service Mesh Security Plugin Pattern

Service mesh security plugin pattern

In this design pattern we see the insertion of a service mesh data plane container (e.g. Envoy) within each Pod, so both the Service Mesh proxy and application containers are running in the same Pod. In this case the application layer inspection is handled through a modsecurity plugin for Envoy. Layer 3 and 4 micro-segmentation is implemented via the Service Mesh policy.

Some of the pros and cons of this design include:

Pros:

  • More cleanly extends security into an existing Service Mesh
  • Unifies the scaling of the security and application microservices
  • The Service Mesh proxy can be automatically injected into the Pod
  • Micro-segmentation rules can be implemented via Service Mesh policy
  • Service Mesh enables many advanced application delivery features

Cons:

  • Service Mesh deployments are more complex
  • May result in over-provisioning of the security layer resources

Secure Microservice POC Deployments

In my opinion, the Security Sidecar Pattern is the most convenient for small projects, but using a Service Mesh is probably a better idea for larger, more complex architectures. In some cases a hybrid approach will make more sense.

Leave a reply if you know of any other designs you’ve seen in the field! Stay tuned for future posts where I will demonstrate simple proof of concept implementations of these security design patterns.

Next in the series: Part 2

Featured

Bringing the Unix Philosophy to the 21st Century

Try the jc web demo!

Do One Thing Well

The Unix philosophy of using compact expert tools that do one thing well and pipelining them together to manipulate data is a great idea and has worked well for the past few decades. This philosophy was outlined in the 1978 Foreward to the Bell System Technical Journal describing the UNIX Time-Sharing System:

Foreward to the Bell System Technical Journal

Items i and ii are oft repeated, and for good reason. But it is time to take this philosophy to the 21st century by further defining a standard output format for non-interactive use.

Unfortunately, this is the state of things today if you want to grab the IP address of one of the ethernet interfaces on your linux system:

$ ifconfig ens33 | grep inet | awk '{print $2}' | cut -d/ -f1 | head -n 1

This is not beautiful.

Up until about 2013 it made just as much sense as anything to assume unstructured text was a good way to output data at the command line. Unix/linux has many text parsing tools like sed, awk, grep, tr, cut, rev, etc. that can be pipelined together to reformat the desired data before sending it to the next program. Of course, this has always been a pain and is the source of many questions all over the web about how to parse the output of so-and-so program. The requirement to parse unstructured (in some cases only human readable) data manually has made life much more difficult than it needs to be for the average linux administrator.

But in 2013 a certain data format called JSON was standardized as ECMA-404 and later in 2017 as RFC 8259 and ISO/IEC 21778:2017. JSON is ubiquitous these days in REST APIs and is used to serialize everything from data between web applications, to Indicators of Compromise in the STIX2 specification, to configuration files. There are JSON parsing libraries in all modern programming languages and even JSON parsing tools for the command line, like jq. JSON is everywhere, it’s easy to use, and it’s a standard.

Had JSON been around when I was born in the 1970’s Ken Thompson and Dennis Ritchie may very well have embraced it as a recommended output format to help programs “do one thing well” in a pipeline.

To that end, I argue that linux and all of its supporting GNU and non-GNU utilities should offer JSON output options. We already see some limited support of this in systemctl and the iproute2 utilities like ip where you can output in JSON format with the -j option. The problem is that many linux distros do not include a version that offers JSON output (e.g. centos, currently). And even then, not all functions support JSON output as shown below:

Here is ip addr with JSON output:

$ ip -j addr show dev ens33
 [{
         "addr_info": [{},{}]
     },{
         "ifindex": 2,
         "ifname": "ens33",
         "flags": ["BROADCAST","MULTICAST","UP","LOWER_UP"],
         "mtu": 1500,
         "qdisc": "fq_codel",
         "operstate": "UP",
         "group": "default",
         "txqlen": 1000,
         "link_type": "ether",
         "address": "00:0c:29:99:45:17",
         "broadcast": "ff:ff:ff:ff:ff:ff",
         "addr_info": [{
                 "family": "inet",
                 "local": "192.168.71.131",
                 "prefixlen": 24,
                 "broadcast": "192.168.71.255",
                 "scope": "global",
                 "dynamic": true,
                 "label": "ens33",
                 "valid_life_time": 1732,
                 "preferred_life_time": 1732
             },{
                 "family": "inet6",
                 "local": "fe80::20c:29ff:fe99:4517",
                 "prefixlen": 64,
                 "scope": "link",
                 "valid_life_time": 4294967295,
                 "preferred_life_time": 4294967295
             }]
     }
 ]

And here is ip route not outputting JSON, even with the -j flag:

$ ip -j route
 default via 192.168.71.2 dev ens33 proto dhcp src 192.168.71.131 metric 100 
 192.168.71.0/24 dev ens33 proto kernel scope link src 192.168.71.131 
 192.168.71.2 dev ens33 proto dhcp scope link src 192.168.71.131 metric 100

Some other more modern tools like, kubectl and the aws-cli tool offer more consistent JSON output options which allow much easier parsing and pipelining of the output. But there are many older tools that still output nearly unparsable text. (e.g. netstat, lsblk, ifconfig, iptables, etc.) Interestingly Windows PowerShell has embraced using structured data, and that’s a good thing that the linux community can learn from.

How do we move forward?

The solution is to start an effort to go back to all of these legacy GNU and non-GNU command line utilities that output text data and add a JSON output option to them. All operating system APIs, like the /proc and /sys filesystems should serialize their files in JSON or provide the data in an alternative API that outputs JSON.

https://github.com/kellyjonbrazil/jc

In the meantime, I have created a tool called jc (https://github.com/kellyjonbrazil/jc) that converts the output of dozens of GNU and non-GNU commands and configuration files to JSON. Instead of everyone needing to create their own custom parsers for these common utilities and files, jc acts as a central clearinghouse of parsing libraries that just need to be written once and can be used by everyone.

Try the jc web demo!

jc is now available as an Ansible filter plugin!

JC In Action

Here’s how jc can be used to make your life easier today and until GNU/linux brings the Unix philosophy into the 21st century. Let’s take that same example of grabbing an ethernet IP address from above:

$ ifconfig ens33 | grep inet | awk '{print $2}' | cut -d/ -f1 | head -n 1
192.168.71.138

And here’s how you do the same thing with jc and a CLI JSON parsing tool like jq:

$ ifconfig ens33 | jc --ifconfig | jq -r '.[].ipv4_addr'
192.168.71.138

or

$ jc ifconfig ens33 | jq -r '.[].ipv4_addr'
192.168.71.138

Here’s another example of listing the listening TCP ports on the system:

$ netstat -tln | tr -s ' ' | cut -d ' ' -f 4 | rev | cut -d : -f 1 | rev | tail -n +3
25
22

That’s a lot of text manipulation just to get a simple list of port numbers! Here’s the same thing using jc and jq:

$ netstat -tln | jc --netstat | jq '.[].local_port_num'
25
22

or

$ jc netstat -tln | jq '.[].local_port_num'
25
22

Notice how much more intuitive it is to search and compare semantically enhanced structured data vs. awkwardly parsing low-level text? Also, the JSON output can be preserved to be used by any higher-level programming language like Python or JavaScript without line parsing. This is the future, my friends!

jc currently supports the following parsers: arp, df, dig, env, free, /etc/fstab, history, /etc/hosts, ifconfig, iptables, jobs, ls, lsblk, lsmod, lsof, mount, netstat, ps, route, ss, stat, systemctl, systemctl list-jobs, systemctl list-sockets, systemctl list-unit-files, uname -a, uptime, and w.

If you have a recommendation for a command or file type that is not currently supported by jc, add it to the comments and I’ll see if I can figure out how to parse and serialize it. If you would like to contribute a parser, please feel free!

With jc, we can make the linux world a better place until the OS and GNU tools join us in the 21’st century!

Featured

Hi, fellow cybersecurity and computing nerds!

I’m Kelly Brazil and I’ve been in the cybersecurity industry for a couple decades now. I taught myself how to program when I was in third grade and I love to dabble in lots of techie topics including network security, cloud computing, microservice architectures and security, software defined networking and SD-WAN, linux, APIs, guitar, and Oxford commas.

Welcome to my blog – I’ll get the urge to write on one of these topics from time to time and I encourage open dialogue.

Some of my prior blog posts can be found here:

Projects:

Media, Mentions, and Events:

RSS
Follow by Email
LinkedIn
LinkedIn
Share