Alerts in Splunk

Hello everyone, in this blog am going to tell about types of alerts and alert actions and how to create those alerts in Splunk 6.3.3.

If you’ve been using Splunk’s Search app for a while, you know how you can use Splunk’s powerful search capabilities to learn all kinds of things about the machine data in your system. But this doesn’t help you with the thousands of recurring situations that everyone in IT faces on a regular basis. You can’t run searches every time to find these events.

To deal with this kind of situation a variety of alerts can be configured for both real-time and historical searches. Alerts can be set-up on historical searches to run automatically on regular schedules or when the search results meet specific circumstances. Alerts can be based on a wide range of threshold and trend-based scenarios, including empty shopping carts, brute force firewall attacks, and server system errors.

Splunk supports three types of alerts. The three alert categories are

  1. Per-result alert

This alert type is used if one need to know the moment a matching result comes in. This type is also useful if it need to design an alert for machine consumption (such as a workflow-oriented application). Alerts can be throttled to ensure that they don’t trigger too frequently. This can be used to

  • Trigger an alert for every failed login attempt, but alert at most once an hour for any given username.
  • Trigger an alert when a “file system full” error occurs on any host, but only send notifications for any given host once per 30 minutes.
  1. Scheduled alert

This alert type triggers whenever a scheduled run of a historical search returns results that meet a particular condition that has been configured in the alert definition. Best for cases where immediate reaction to an alert is not a priority. This alert can be throttled to reduce the frequency of redundant alerts. This can be used to

  • Trigger an alert whenever the number of items sold in the previous day is less than 500.
  • Trigger an alert when the number of 404 errors in any 1 hour interval exceeds 100.
  1. Rolling-window alert

This alert type is used to monitor events in real time within a rolling time window of a width that you define, such as a minute, 10 minutes, or an hour. The alert triggers when its conditions are met by events as they pass through this window in real time. These alerts can be throttled to ensure that they don’t trigger too frequently. This can be used to

  • Trigger an alert whenever there are three consecutive failed logins for a user between now and 10 minutes ago, but trigger for any given user only once in an hour.
  • Trigger an alert when a host is unable to complete an hourly file transfer to another host within the last hour, but don’t alert more than once an hour for any particular host.


Creating alerts in Splunk web

Let’s see an example of creating an alert. Below given are the steps to create a per-result alert.

  1. From the Search Page, enter the following search:

index=_internal (log_level=ERROR OR log_level=WARN* OR log_level=FATAL OR log_level=CRITICAL) | stats count as log_events

  1. Select Save As > Alert
  2. In the Save as Alert dialog box, enter a Title for the alert.
  3. For Alert Type, select Real Time.
    A per-result alert is always a real-time alert type.
  4. For trigger condition, select Per-Result.
  5. Select the actions you want to enable.
    For this example, select List in Triggered Alert.
  6. Click Save.


Alert actions in Splunk

The below actions can be set-up for an alert in Splunk.

1.Add to triggered alerts

An alert can be configured to add it to the list of triggered alerts. Select the List in Activity > Triggered Alerts action to display triggered alerts in the Alert manager. The Alert manager lists the details of triggered alerts for 24 hours or a specified duration. Alerts can be listed based on its severity level specified during its creation. The default severity level is medium.


2.Email notification.

Alerts can be configure to send an email notification to specified recipients when the alert triggers. The email notification can include information related to the alert. The email notification is a multipart MIME message that includes both HTML and text parts. Email notification for an alert can be configured when an alert is saved from the search page. Alerts can also configure email notification from the Alerts Page and directly from a search command.


3. Webhook action

Webhook allows to define custom callbacks on a particular web resource. For instance, a webhook can set-up to make an alert message pop up in a chat room or post a notification on a web page. It is possible to create a webhook action for instant alert notifications at a particular URL. When an alert is triggered, the webhook will make an HTTP POST request on the URL. The webhook passes JSON formatted information about the alert in the body of the POST request. A webhook starts with an alert.


4. Run a script for an alert action

A script can be made to run when an alert triggers. Select Run a script under Enable actions. Enter the file name of the script that has to be run. For example, an alert can be configured to run a script that generates a Simple Network Management Protocol (SNMP) trap notification. The script sends the notification to another system such as a Network Systems Management console. A different alert can be configured to run a script that calls an API, which in turn sends the triggering event to another system.


So that’s all about creating alerts and alert actions, hope this blog is useful. To know more about alerts in Splunk refer the Alerts manual.



Happy Splunking!!!!!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s