There are three levels of action in any IT system.
For proactive maintenance you need to have alerts where certain thresholds are approached. The user needs to be alerted in advance of any situation where they may lose their work. For example, in Diagnostics there is a datastore growth monitor, which will allow you to prevent uncompromising situations
Next are anomalies. These are unpredictable conditions that arise. By having automated alerts in place, these anomalies can be processed quickly. If you are unaware of anomalies you cannot act. In this case, the alert will initiate actions by your sysadmin staff in an attempt to resolve this before being informed by your users
Finally, there is acting on breakdowns which come in the form of errors or tracebacks. This usually happens when the end-user raises tickets for the errors or tracebacks, something which may have been prevented by being aware of the potential challenges and anomalies which could occur.
All of these levels can use notification tools, but it is more likely that the first two action levels receive the most benefit from automated notification.
The easiest implementation of notifications is to send an email to the maintainers to act upon. For example, your end users are thrown out of the database due to certain unforeseen conditions in your infrastructure. They will receive a “datastore_not_initialized” fatal condition on their workstations or Citrix sessions.
Diagnostics can notify you of this condition as soon as it happens (in near real-time). You can configure an alert to send out an email as soon as users start getting this condition.
But is email the best notification tool?
I do not know how your email box fills up, but mine is unsuitable for notifications. When I am in a meeting, I ignore my mail because I want to concentrate on the content of the meeting. People take time to meet with me, so I feel obligated to pay attention.
An email can be in waiting for more than one hour and if unlucky: multiple hours!
Conclusion: Email notification would fail for me and my user.
Other Notifications
What other kinds or notifications are available? You can use SMS texting. But there are also specialized tools for this purpose in the market: pushover, prowl etc. You can also think of messengers like telegram or even MS Teams.
Too Technical
Diagnostics leverages the functionality of Splunk under the hood. Spunk has a huge app store with apps available to connect to third party notification tools. It is also possible to write an app yourself…
Too Simple
Some further investigation on how to circumvent this app solution and avoid the “under the hood solutions” brought me this insight: Most services allow you to send an email!
Prowl for example makes it possible to send messages to <token>@api.prowlapp.com. MS Teams also allows you to send messages to channels in MS Teams.
Conclusion
Get ahead of the tickets and issues of your end-users! Configure your Diagnostics alerting to notify you on your favourite notification app by sending the message to that service! Most importantly, we recommend selecting a mechanism that causes you to initiate action.