Installation

Install the Kestrel runtime, Kestrel Jupyter front-end, and STIX-shifter connector modules.

Operating Systems

Currently, Kestrel is supported on Linux and macOS.

Requirements

Python

This project builds on Python 3. Refer to the Python installation guide if you do not have Python 3.

SQLite

If you are using following Linux distributions or newer, the SQLite requirement is already met:

  • Archlinux

  • Debian 10

  • Fedora 33

  • Gentoo

  • openSUSE Leap 15.2

  • RedHat 8

  • Ubuntu 20.04 LTS

Otherwise, check the SQLite version in a terminal and upgrade sqlite3 >= 3.24 as needed, which is required by firepit, a Kestrel dependency, in its default configuration:

$ sqlite3 --version

Runtime Installation

You can install Kestrel runtime from stable release or nightly built version (source code). Either way installs all packages in the kestrel-lang repository, and dependent packages, such as firepit and stix-shifter.

It is a good practice to install Kestrel in a Python virtual environment so all dependencies will be the latest. You can easily setup, activate, and update a Python virtual environment named huntingspace:

$ python -m venv huntingspace
$ . huntingspace/bin/activate
$ pip install --upgrade pip setuptools wheel

Stable Release

Run this command in your terminal (huntingspace activated):

$ pip install kestrel-lang

Nightly Built Version (Source Code)

Run this command in your terminal (huntingspace activated):

$ git clone git://github.com/opencybersecurityalliance/kestrel-lang
$ cd kestrel-lang && pip install .

Front-Ends Installation

Kestrel runtime currently supports three front-ends (Kestrel in a Nutshell):

  1. Command-line execution utility kestrel: Installed with the package kestrel.

$ kestrel [-h] [-v] [--debug] hunt101.hf
  1. Kestrel Jupyter Notebook kernel: Must install and set up the kestrel-jupyter package (Jupyter Notebook dependencies will be automatically installed if they do not exist):

$ pip install kestrel-jupyter
$ python -m kestrel_jupyter_kernel.setup
  1. Python API:

STIX-shifter Connector Installation

Among Data Source And Analytics Interfaces, STIX-shifter is the main data source interface currently implemented by the Kestrel runtime. STIX-shifter provides a federated search interface against more than a dozen EDRs, NDRs, and SIEM systems for data retrieval.

Because of the federated nature of STIX-shifter, the project releases a string of Python packages (called connectors of STIX-shifter) for each data source. Depending on the data source you are connecting to, e.g., Sysmon data stored in Elasticsearch, you need to install the corresponding connector such as stix-shifter-modules-elastic-ecs:

$ pip install stix-shifter-modules-elastic-ecs

STIX-shifter Data Source Config

After installing the STIX-shifter connector, you need to tell a Kestrel front-end, e.g., Jupyter, details of the data source you are connecting to. This is done by exporting three environment variables for each data source, e.g.:

$ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
$ export STIXSHIFTER_HOST101_CONNECTION='{"host":"elastic.securitylog.company.com", "port":9200, "indices":"host101"}'
$ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"VuaCfGcBCdbkQm-e5aOx", "api_key":"ui2lp2axTNmsyakw9tvNnw"}}'

Multiple STIX-shifter connections can be specified together in a single YAML file for convenience, as such:

stixshifter:
- name: host101
  connector: elastic_ecs
  connection:
    host: elastic.securitylog.company.com
    port: 9200
    indices: host101
  config:
    auth:
    id: VuaCfGcBCdbkQm-e5aOx
    api_key: ui2lp2axTNmsyakw9tvNnw

You can specify as many profiles as you like. Configurations are loaded in the following order, where subsequent setting override previous ones.

  1. ~/.kestrel/config.yml Default location for the Kestrel configuration YAML file

  2. KESTREL_CONFIG Environment variable which points to a YAML file that overrides options already loaded via 1.

  3. STIXSHIFTER_<profile>_{CONNECTOR, CONNECTION, CONFIG} environment variables as above. Overrides setting from 1 and 2.

Note that it will intelligently override individual settings. E.g.,

$ export STIXSHIFTER_HOST101_CONNECTION='{"port" : 8080}'

Will use the host101 configuration from the ~/.kestrel/config.yml file but changing only the port to 8080.

(Optional) Kestrel Analytics

Want to have some Kestrel analytics ready at your fingertip? Threat intelligence enrichments like SANS API? Domain name lookup for IP addresses? Finding IP geolocations and pin them on an interactive map? Invoking machine learning inference function? Clone the community-contributed Kestrel analytics repo to start:

$ git clone https://github.com/opencybersecurityalliance/kestrel-analytics.git

Go to the analytics directory and build the analytics docker containers to APPLY in your hunt.

Kestrel in Action

Now the Kestrel runtime is set up and you can run a Kestrel huntflow with the command-line utility or launch a Jupyter service for developing a huntbook interactively (huntingspace activated):

$ jupyter notebook