Self State Monitor¶
Self State Monitor is a built-in mechanism designed to protect
end user from false
NODATA notifications and notify administrator
about issues in Moira and/or Graphite systems.
Why Self State Monitor¶
A situation is possible when Graphite Relay, Redis DB or Moira-Filter service breaks down.
This leads to the fact that Moira doesn’t receive any metrics from Graphite.
In this case, Moira has no metrics on which it could check state of the triggers.
According to the Moira logic, it should switch triggers to
and send alert messages to users.
To handle this situation properly, we recommend turning on the Self State Monitor. In this case, Moira will prevent itself from sending alert messages to end users but notify administrators of the existing problem.
When Self State Monitor detects a problem, it disables any notifications to end users and does not turn it back on without manual intervention.
Please, read this manual before using Self State Monitor in production.
For a better understanding, look at the architecture of the Moira microservices.
When Self State Monitor helps¶
Self state monitor checks these situations:
- If there is no connection between Moira and Redis for longer than
- If Moira-Filter receive no metrics for longer than
- If Moira-Checker checks no triggers for longer than
All the above configuration parametres can be found in the Moira-Notifier section on configuration page.
How Self State Monitor works¶
When you turn Self State Monitor on, it works this way:
Self State Monitor checks Moira state every 10 seconds.
Something breaks down. It can be Graphite-Relay, connection to Redis DB or crashed Moira-Filter docker container.
Self State send alarm message to administrator with issue discription.
Here is an example of message:
Self State Monitor turns Moira-Notifier service off, switching it in
When Moira-Notifier switches to
ERRORstate, it mutes all messages to end users and only alerts administrators about Moira health issues. You need to fix existing problems and then manually switch Moira-Notifier back to
When Moira-Notifier not in
OKstate, Moira will show you an error in Web UI:
Turn Moira-Notifier ON and OFF¶
You can reveal current Moira-Notifier state or change it on a hidden
Please, note this toggle changes Moira-Notifier state, not user notifications preferences.
When you disable notifications with this toggle, Moira-Notifier stops sending messages to all users.