STIX-shifter Data Source Interface

The STIX-shifter data source package provides access to data sources via stix-shifter.

The STIX-shifter interface connects to multiple data sources. Users need to provide one profile per data source. The profile name (case insensitive) will be used in the FROM clause of the Kestrel GET command, e.g., newvar = GET entity-type FROM stixshifter://profilename WHERE .... Kestrel runtime will load profiles from 3 places (the later will override the former):

1. STIX-shifter interface config file (only when a Kestrel session starts):

   Create the STIX-shifter interface config file (YAML):

   • Default path: ~/.config/kestrel/stixshifter.yaml.

   • A customized path specified in the environment variable KESTREL_STIXSHIFTER_CONFIG.

   Example of STIX-shifter interface config file containing profiles (note that the options section is not required):

   profiles:
       host101:
           connector: elastic_ecs
           connection:
               host: elastic.securitylog.company.com
               port: 9200
               selfSignedCert: false # this means do NOT check cert
               indices: host101
               options:  # use any of this section when needed
                   retrieval_batch_size: 10000  # set to 10000 to match default Elasticsearch page size; Kestrel default across connectors: 2000
                   single_batch_timeout: 120  # increase it if hit 60 seconds (Kestrel default) timeout error for each batch of retrieval
                   dialects:  # more info: https://github.com/opencybersecurityalliance/stix-shifter/tree/develop/stix_shifter_modules/elastic_ecs#dialects
                     - beats  # need it if the index is created by Filebeat/Winlogbeat/*beat
           config:
               auth:
                   id: VuaCfGcBCdbkQm-e5aOx
                   api_key: ui2lp2axTNmsyakw9tvNnw
       host102:
           connector: qradar
           connection:
               host: qradar.securitylog.company.com
               port: 443
           config:
               auth:
                   SEC: 123e4567-e89b-12d3-a456-426614174000
       host103:
           connector: cbcloud
           connection:
               host: cbcloud.securitylog.company.com
               port: 443
           config:
               auth:
                   org-key: D5DQRHQP
                   token: HT8EMI32DSIMAQ7DJM
   options:  # this section is not required
       fast_translate:  # use firepit-native translation (Dataframe as vessel) instead of stix-shifter result translation (JSON as vessel) for the following connectors
           - qradar
           - elastic_ecs
   

   Full specifications for data source profile sections/fields:

   • Connector-specific fields: in stix-shifter, go to stix_shifter_modules/connector_name/configuration like elastic_ecs config.

   • General fields shared across connectors: in stix-shifter, go to stix_shifter_modules/lang_en.json.

2. environment variables (only when a Kestrel session starts):

   Three environment variables are required for each profile:

   • STIXSHIFTER_PROFILENAME_CONNECTOR: the STIX-shifter connector name, e.g., elastic_ecs.

   • STIXSHIFTER_PROFILENAME_CONNECTION: the STIX-shifter connection object in JSON string.

   • STIXSHIFTER_PROFILENAME_CONFIG: the STIX-shifter configuration object in JSON string.

   Example of environment variables for a profile:

   $ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
   $ export STIXSHIFTER_HOST101_CONNECTION='{"host":"elastic.securitylog.company.com", "port":9200, "indices":"host101"}'
   $ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"VuaCfGcBCdbkQm-e5aOx", "api_key":"ui2lp2axTNmsyakw9tvNnw"}}'
   

3. any in-session edit through the CONFIG command.

If you launch Kestrel in debug mode, STIX-shifter debug mode is still not enabled by default. To record debug level logs of STIX-shifter, create environment variable KESTREL_STIXSHIFTER_DEBUG with any value.

class kestrel_datasource_stixshifter.interface.StixShifterInterface

Bases: AbstractDataSourceInterface

static schemes()

STIX-shifter data source interface only supports stixshifter:// scheme.

static list_data_sources(config)

Get configured data sources from environment variable profiles.

async static query(uri, pattern, session_id, config, store=None)

Query a stixshifter data source.