USE_KIBANA/ELASTICSEARCH=0 ignored, still tries to connect

[brsolomon-deloitte]

Using local docker image scirius:test-1 built from scirius at commit 0b8fd6d fails in trying to connect to Elasticsearch, even though local_settings.py defaults to USE_ELASTICSEARCH=False and USE_KIBANA=False.

It appears that docker/scirius/bin/reset_dashboards.sh doesn't respect these variables whatsoever and calls python manage.py kibana_reset indiscriminately:

scirius/docker/scirius/bin/reset_dashboards.sh

Line 25 in 0b8fd6d

reset_dashboards() {

$ docker container run -e SECRET_KEY=$(openssl rand -hex 16) --expose 8000 scirius:test-1
Migrations for 'accounts':
  accounts/migrations/0007_auto_20220421_1743.py
    - Alter field timezone on sciriususer
Operations to perform:
  Apply all migrations: accounts, auth, authtoken, contenttypes, rules, sessions, suricata
Running migrations:
  Applying rules.0001_initial... OK
  Applying rules.0002_auto_20141207_1824... OK
  Applying rules.0003_auto_20141210_1421... OK
  Applying rules.0004_auto_20141210_1525... OK
  Applying rules.0005_auto_20141210_1734... OK
  Applying rules.0006_auto_20141210_1846... OK
  Applying rules.0007_auto_20141210_2037... OK
  Applying rules.0008_auto_20141210_2057... OK
  Applying rules.0009_auto_20141214_1203... OK
  Applying rules.0010_auto_20141222_1209... OK
  Applying rules.0011_auto_20141222_1304... OK
  Applying rules.0012_auto_20141222_1306... OK
  Applying rules.0013_auto_20141229_1527... OK
  Applying rules.0014_auto_20141229_1528... OK
  Applying rules.0015_auto_20141229_1610... OK
  Applying rules.0016_auto_20141229_1629... OK
  Applying rules.0017_auto_20141229_1716... OK
  Applying rules.0018_auto_20141229_1716... OK
  Applying rules.0019_auto_20141229_1719... OK
  Applying rules.0020_auto_20141229_1852... OK
  Applying rules.0021_auto_20141229_1853... OK
  Applying rules.0022_auto_20141229_1858... OK
  Applying rules.0023_auto_20141229_1903... OK
  Applying rules.0024_auto_20141229_2204... OK
  Applying rules.0025_auto_20141230_0812... OK
  Applying rules.0026_auto_20141231_0948... OK
  Applying rules.0027_auto_20141231_0953... OK
  Applying rules.0028_auto_20150101_2305... OK
  Applying rules.0029_auto_20150102_1212... OK
  Applying rules.0030_auto_20150103_1136... OK
  Applying rules.0031_auto_20150103_1138... OK
  Applying rules.0032_auto_20150103_1255... OK
  Applying rules.0033_auto_20150109_2319... OK
  Applying rules.0034_auto_20150111_2200... OK
  Applying rules.0035_auto_20150202_0937... OK
  Applying rules.0036_auto_20150203_1421... OK
  Applying rules.0037_auto_20150407_2040... OK
  Applying rules.0038_auto_20150516_0912... OK
  Applying rules.0039_auto_20150805_1737... OK
  Applying rules.0040_ruleset_rules_count... OK
  Applying rules.0041_source_authkey... OK
  Applying rules.0042_rule_state_in_source... OK
  Applying rules.0043_threshold... OK
  Applying rules.0044_flowbit_type... OK
  Applying rules.0045_auto_20160405_1300... OK
  Applying rules.0046_source_cert_verif... OK
  Applying rules.0047_proxy_validation... OK
  Applying rules.0048_custom_es... OK
  Applying rules.0049_auto_20161121_2342... OK
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying rules.0050_auto_20161128_2110... OK
  Applying rules.0051_auto_20161207_0758... OK
  Applying auth.0001_initial... OK
  Applying rules.0052_useraction_user... OK
  Applying rules.0053_unique_none_rules... OK
  Applying rules.0054_login_action... OK
  Applying rules.0055_auto_20180213_1723... OK
  Applying rules.0056_auto_20180223_0823... OK
  Applying rules.0057_auto_20180302_1312... OK
  Applying rules.0058_source_public_source... OK
  Applying rules.0059_auto_20180309_2012... OK
  Applying rules.0060_auto_20180403_0921... OK
  Applying rules.0061_auto_20180507_1410... OK
  Applying rules.0062_useraction_username... OK
  Applying rules.0063_ruleprocessingfilter_ruleprocessingfilterdef... OK
  Applying rules.0064_ruleprocessingfilter_rulesets... OK
  Applying rules.0061_auto_20180503_2200... OK
  Applying rules.0063_merge_20180718_0118... OK
  Applying rules.0065_merge_20180719_1505... OK
  Applying rules.0066_auto_20180807_1428... OK
  Applying rules.0067_source_use_iprep... OK
  Applying rules.0068_auto_20180818_2204... OK
  Applying rules.0069_auto_20190220_1500... OK
  Applying rules.0070_ruleprocessingfilterdef_full_string... OK
  Applying rules.0071_filterset... OK
  Applying rules.0072_send_mail... OK
  Applying rules.0073_filterset_description... OK
  Applying rules.0074_redlights_useraction... OK
  Applying rules.0075_suppress_validator... OK
  Applying rules.0075_custom_es_no_empty... OK
  Applying rules.0076_merge_20190926_1233... OK
  Applying rules.0077_auto_20191002_0820... OK
  Applying rules.0078_auto_20200206_1648... OK
  Applying rules.0079_source_remove_choice... OK
  Applying rules.0080_source_version... OK
  Applying rules.0081_django-2... OK
  Applying rules.0082_source_use_sys_proxy... OK
  Applying rules.0083_multi_es_validation... OK
  Applying rules.0084_fakepermissionmodel... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying accounts.0001_initial... OK
  Applying accounts.0002_auto_20151110_1657... OK
  Applying accounts.0003_timezone... OK
  Applying accounts.0004_group... OK
  Applying rules.0085_roles_migrations... OK
  Applying accounts.0005_remove_user_flags... OK
  Applying accounts.0006_fix_role_default_priority... OK
  Applying accounts.0007_auto_20220421_1743... OK
  Applying authtoken.0001_initial... OK
  Applying authtoken.0002_auto_20160226_1747... OK
  Applying authtoken.0003_tokenproxy... OK
  Applying rules.0086_ruleset_suppressed_sids... OK
  Applying rules.0087_systemsettings_use_proxy_for_es... OK
  Applying rules.0088_ruleprocessingfilter_import_member... OK
  Applying rules.0089_ruleprocessingfilter_event_type... OK
  Applying rules.0090_useraction_ip... OK
  Applying rules.0091_useraction_missing_ruleset... OK
  Applying sessions.0001_initial... OK
  Applying suricata.0001_initial... OK
  Applying suricata.0002_auto_20151110_1657... OK
  Applying suricata.0003_suricata_yaml_file... OK
  Applying suricata.0004_auto_20160316_0844... OK
  Applying suricata.0005_django-2... OK
from django.contrib.auth.models import User; User.objects.create_superuser(***)
Successfully created source "ETOpen Ruleset"
Successfully updated source "ETOpen Ruleset"
Successfully created source "SSLBL abuse.ch"
Successfully updated source "SSLBL abuse.ch"
Successfully created source "PT Research Ruleset"
Successfully updated source "PT Research Ruleset"
Successfully created default ruleset "Default ruleset"
Successfully removed "stream-events" from ruleset "Default ruleset"
Successfully created suricata "suricata"
Successfully pushed ruleset to suricata "suricata"

236 static files copied to '/static'.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.

[brsolomon-deloitte]

Same thing with https://github.com/StamusNetworks/scirius/blob/master/docker/scirius/bin/create_ILM_policy.sh which tries to contact Elasticsearch indiscriminately.

[brsolomon-deloitte]

It would be easy for kibana_reset in rules/es_data.py to actually check for USE_KIBANA before trying to make a bunch of API calls to it.