STIX-shifter Data Source Interface

The STIX-shifter data source package provides access to data sources via stix-shifter.

The STIX-shifter interface connects to multiple data sources. Users need to provide one profile per data source. The profile name (case insensitive) will be used in the FROM clause of the Kestrel GET command, e.g., newvar = GET entity-type FROM stixshifter://profilename WHERE .... Kestrel runtime will load profiles from 3 places (the later will override the former):

  1. STIX-shifter interface config file (only when a Kestrel session starts):

    Create the STIX-shifter interface config file (YAML):

    • Default path: ~/.config/kestrel/stixshifter.yaml.

    • A customized path specified in the environment variable KESTREL_STIXSHIFTER_CONFIG.

    Example of STIX-shifter interface config file containing profiles (note that the options section is not required):

            connector: elastic_ecs
                port: 9200
                indices: host101
                pagination: false # disable pagination (only <10k results) to have better performance; Kestrel default: true
                options:  # use any of this section when needed
                    verify_cert: false  # allow invalid/expired/self-signed certificate
                    retrieval_batch_size: 10000  # set to 10000 to match default Elasticsearch page size; Kestrel default across connectors: 2000
                    single_batch_timeout: 120  # increase it if hit 60 seconds (Kestrel default) timeout error for each batch of retrieval
                    cool_down_after_transmission: 2  # seconds to cool down between data source API calls, required by some API such as sentinelone; Kestrel default: 0
                    subquery_time_window: 3600 # split each query into multiple subqueries with smaller time windows specified here in seconds; Kestrel default: 0 (not split query)
                    allow_dev_connector: True  # do not check version of a connector to allow custom/testing connector installed with any version; Kestrel default: False
                    dialects:  # more info:
                      - beats  # need it if the index is created by Filebeat/Winlogbeat/*beat
                    id: VuaCfGcBCdbkQm-e5aOx
                    api_key: ui2lp2axTNmsyakw9tvNnw
            connector: qradar
                port: 443
                    SEC: 123e4567-e89b-12d3-a456-426614174000
            connector: cbcloud
                port: 443
                    org-key: D5DQRHQP
                    token: HT8EMI32DSIMAQ7DJM
    options:  # this section is not required
        fast_translate:  # use firepit-native translation (Dataframe as vessel) instead of stix-shifter result translation (JSON as vessel) for the following connectors
            - qradar
            - elastic_ecs
        translation_workers_count: 8  # default: 2

    Full specifications for data source profile sections/fields:

    The stix-shifter YAML config supports expansion of environment variables, e.g., $HOST101_ID and $HOST101_KEY will be replaced by values from the environment variables when the following section of the config loads by Kestrel:

                    id: $HOST101_ID
                    api_key: $HOST101_KEY
  2. environment variables (only when a Kestrel session starts):

    Three environment variables are required for each profile:

    • STIXSHIFTER_PROFILENAME_CONNECTOR: the STIX-shifter connector name, e.g., elastic_ecs.

    • STIXSHIFTER_PROFILENAME_CONNECTION: the STIX-shifter connection object in JSON string.

    • STIXSHIFTER_PROFILENAME_CONFIG: the STIX-shifter configuration object in JSON string.

    Example of environment variables for a profile:

    $ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
    $ export STIXSHIFTER_HOST101_CONNECTION='{"host":"", "port":9200, "indices":"host101"}'
    $ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"VuaCfGcBCdbkQm-e5aOx", "api_key":"ui2lp2axTNmsyakw9tvNnw"}}'
  3. any in-session edit through the CONFIG command.

After added data source profiles into stixshifter.yaml, you can test the data source:

$ stix-shifter-diag data_source_name

where data_source_name is any profile named in the stixshifter.yaml config file, usually used in FROM stixshifter://data_source_name in the GET command.

The diagnosis utility will check config, test query translation, try connect to the data source to execute a small and a large query, and retrieve data back. Details of all steps will be printed for diagnosis purpose.

If you launch Kestrel in debug mode, STIX-shifter debug mode is still not enabled by default. To record debug level logs of STIX-shifter, create environment variable KESTREL_STIXSHIFTER_DEBUG with any value.

class kestrel_datasource_stixshifter.interface.StixShifterInterface[source]

Bases: AbstractDataSourceInterface

static schemes()[source]

STIX-shifter data source interface only supports stixshifter:// scheme.

static list_data_sources(config)[source]

Get configured data sources from environment variable profiles.

static query(uri, pattern, session_id, config, store, limit=None)[source]

Query a stixshifter data source.