MyObservability

Advanced Thresholding

Options

• Time-based Thresholds: Time-based thresholds are user-defined threshold values to be used at different times of the day or week to account for changing KPI workloads.
• Adaptive Thresholds: Calculated by machine learning algorithms that dynamically adapt and change based on the KPI’s observed behavior.

Time-based Thresholds

Time-based static thresholds let you define specific threshold values to be used at different times to account for changing workloads over time.

Available KPI threshold templates

ITSI provides default thresholding templates that you can use to build your time policies.

Thresholding templates are either static or adaptive.

• Use static templates to create time policies that do not change after you configure them.
• Use adaptive templates to create time polices that generate thresholds dynamically and update daily based on changes in your data.

Note:

• You can use adaptive thresholds with aggregate thresholds but not per-entity thresholds.
• You can only have one active time policy at any given time. When you create a new time policy, the previous time policy is overwritten and cannot be recovered.

Templet Types

• Quantile: Threshold bounds at various percentiles based on historic data.
  • Ex: A 95th percentile quantile threshold for a KPI means that 95% of the values should fall below that threshold, while the remaining 5% can be above it.
• Standard Deviation: Maximum deviation from the mean value that is considered acceptable for a given KPI.
  • Ex: Threshold of 2 means that any values that fall more than 2 standard deviations away from the mean value will trigger an alert.
• Range: Focuses on the minimum and maximum data points from your historic data and the span between those values (max – min).
• Percentage: Calculates threshold values based on the mean value of your historic data plus a specified percentage from the mean as represented by your threshold settings

Time zones with threshold templates

Time blocks in threshold templates, including custom templates you create, are stored in the backend in UTC time but presented in the UI in your own time zone.

• Through KPI Thresholding templet
  • Pre-defined templet
  • New KPI Threshold templete (User-defined)
  • Clone Oout-of-the-box threshold templet.
• Select KPI and set custom thresholds.

Apply threshold template in a different timezone than the user’s default timezone, change the timezone in your profile settings under preferences, and then clone the threshold template.

Create adaptive KPI thresholds in ITSI

Adaptive thresholding in IT Service Intelligence (ITSI) uses machine learning techniques to analyze historic data and determine what KPI behavior should be considered normal in your IT environment. By dynamically calculating time-dependent thresholds, adaptive thresholding allows operations to more closely match alerts to the expected workload on an hour-by-hour basis.

Performance considerations

Adaptive thresholds are automatically recalculated on a nightly basis so that gradual changes in behavior don’t trigger false alerts. Each time a threshold is recalculated, the service must be re-saved into the its_services KV store collection. This process can moderately impact performance. The performance impact increases if a lot of your services contain KPIs with Adaptive Thresholding enabled.

Configure alerts for abnormal thresholds

After you threshold your KPIs with adaptive thresholds, you need to consider what type of alert configurations make sense to transform abnormal KPI results into actionable alerts.

The following are two common alert strategies:

• Alert when a KPI is exhibiting extremely abnormal behavior.
• Alert when multiple KPIs are simultaneously exhibiting abnormal behavior.

Custom Thresholds Windows (Beta)

Create custom KPI threshold windows to account for time periods when your KPI threshold values may vary due to seasonal events, like holidays, in order to avoid false alerts.

Custom threshold window statuses

Each custom threshold window has one of the following statuses:

• Active: the custom threshold window is running.
• Completed: the custom threshold window completed its schedule run.
• Invalid: the custom threshold window is not linked to any KPIs and therefore not running, or has already completed its scheduled run time. The original KPI(s) it was linked to may have also been deleted. You can delete these custom threshold windows.
• Restoring: the custom threshold window is being restored after a backup.
• Scheduled: the custom threshold window will run at a future time.
• Skipped: something caused the window not to run, or the window time already passed.
• Stopped: the custom threshold window was stopped and no longer running.

---

Next Chapter: Deep Dives

---