Kestrel Interfaces

As a threat hunting language, Kestrel is designed to access a variety of data sources and execute encapsulated analytics in every possible way, besides assembling retrieval, transformational, enriching, and inspective hunt steps into huntflows. In another word, Kestrel deals with different data sources in its retrieval hunt steps as well as different analytics in its enriching hunt steps. It is important to have an abstraction to be adaptable and extensible for data sources and analytics—the design of Kestrel interfaces.

Illustrated in the figure above, Kestrel uses a two-level abstraction for both data source and analytics: (i) a data source or analytics interface defines how a data source or analytics executes, i.e., input, output, plus execution mechanism, and (ii) each data source or analytics is developed to be executed under one or more interfaces.

Each interface has one or multiple schema strings, for example, stixshifter:// for the STIX-shifter Data Source Interface and docker:// for the Docker Analytics Interface. To use a specific data source or analytics, a user specifies an identifier of the data source or analytics as schema://name where name is the data source name or analytics name.

Data Source Interfaces

Kestrel currently implements two data source interfaces: STIX-shifter Data Source Interface and STIX bundle Data Source Interface. The former employees STIX-shifter as a federated search layer to reach to more than 30 different data sources via STIX-shifter connectors. The latter points to canned STIX bundle data for demo or development purposes.

Find how to setup/use data sources in Kestrel at Connect to Data Sources.

In real-world hunts, it is preferred to use a data source through STIX-shifter Data Source Interface to avoid re-implementing data pipelines that exist. As a hunter or hunting platform developers, you may identify the STIX-shifter connectors to be used in your orgainzation and customize them, e.g., update the translation mapping according to your specific data schema. If no STIX-shifter connector exists for your data source, you can follow the STIX-shifter connector development guide to create one from a template by providing the API to the data source as well as the mappings to/from STIX for translation.

You are not required to use STIX-shifter or the STIX-shifter Data Source Interface. If you know how to get data in STIX observations from your data sources, you can add new data sources to STIX bundle Data Source Interface to connect to your data sources. If you don’t like STIX and want direct connection to Kestrel Data Representation, you can create a new data source interface to directly ingest data into firepit, the Kestrel data store. This can be achieved by creating a new Python class inheriting the AbstractDataSourceInterface class. More instructions are in Kestrel Data Source Interface and firepit documentation.

Analytics Interfaces

Kestrel currently implements two analytics interfaces: Python Analytics Interface and Docker Analytics Interface. The former defines/runs a Kestrel analytics as a Python function, while the latter defines/runs a Kestrel analytics as a Docker container.

Check out community-contributed Kestrel analytics at the kestrel-analytics repo to get an idea of what analytics are possible to do in Kestrel. All analytics in the repo can be invoked by either the Python or Docker analytics interface. To use them via the Python Analytics Interface, one needs to tell the interface where the analytic functions are by creating a pythonanalytics.yaml config file (sample provided at kestrel-analytics repo). To use them via the Docker Analytics Interface, one needs to do docker build with the Dockerfile provided in each analytics folder. Visit Setup Kestrel Analytics to learn more about how to setup analytics.

Bring Your Own Analytics

Of course these will not cover all analytics one needs in hunts. One can quickly wrap a Python function into a Kestrel Python analytics as described in Develop a Python Analytics. One can also build a Kestrel analytics as a Docker container by following the guide in Docker Analytics Interface and the blog Building Your Own Kestrel Analytics. It is obvious that a Kestrel analytics under the Docker interface can execute any code in any language, even binary without source code, or an API to other services. In fact, a Kestrel analytics under the Python interface can do the same by using the Python function as a proxy to invoke complex analytic logic in any languages or binaries.

If both analytics interfaces are still not enough, for example, one already have a collection of analytic functions as AWS Lambda functions, one can easily develops a new Kestrel analytics interface to run them. Similar to developing a new Kestrel data source interface, one needs to create a new Python class inheriting the AbstractAnalyticsInterface class. More instructions are in Kestrel Analytics Interface.