Friday, August 3, 2018

Splunk 7.1.0 and DMA acceleration backfill.


Splunk version 7.1.0 Datamodel backfill casue heavy load on indexer peers.
SPL-155560, SPL-155219

DMA accelerating too much data when acceleration.backfill_time unset, resulting in heavy indexer load.
Workaround:
acceleration.backfill_time needs to be set for all DMA searches in datamodels.conf. Otherwise the DMA will attempt to accelerate for "all time"


# list all apps using acceleration datamodel on your SH or deployment server (/etc/deployment-apps) run this:
grep --include="datamodels.conf" -R /opt/splunk/etc/apps/ -e "acceleration = true" -e "acceleration = 1"

Then create your changes in /local to prevent that it will be overwritten by upgrades. Also consider to lower the max concurrent from default 3 if your indexer is very busy. For every stanza with acceleration =1 create your config:
/local/datamodels.conf

[stanza]
acceleration.backfill_time = -5d
acceleration.max_concurrent = 2

How many days back you chose to backfill depends on your environments performance, but it can never be more than  “acceleration.earliest_time”

# verify datamodel

At the SH under Settings – Data Model
List all apps and expand the the Data model that is active and check the status.

 

No comments:

Post a Comment