Incidents
An incident represents a period of downtime for a monitored service. Sentro creates incidents automatically when monitors detect failures and resolves them automatically when the service recovers.
Incident lifecycle
TRIGGERED --> ACKNOWLEDGED --> RESOLVED
| ^
+----------------------------------+
(auto-resolved when service recovers)
TRIGGERED
Created automatically when a monitor check fails (or when a heartbeat ping is missed beyond the grace period). The incident records:
- Which monitor detected the failure.
- When the failure started.
- The cause (e.g., "Expected status 200, got 503" or "Request timed out after 30000ms").
Alerts are dispatched to all matching alert rules at this point.
ACKNOWLEDGED
A team member has seen the incident and is investigating. Acknowledgment is optional but useful for coordination -- it tells the rest of the team that someone is on it. The dashboard shows who acknowledged the incident and when.
RESOLVED
The incident is closed. This happens in two ways:
- Automatically -- When the monitor's checks start succeeding again, Sentro resolves the incident and records the duration.
- Manually -- A team member can resolve an incident from the dashboard.
Recovery alerts are sent to channels with a UP trigger rule.
Incident timeline
Every incident has a timeline of updates that documents what happened:
| Update type | Description |
|---|---|
| TRIGGERED | System-generated when the incident is created. Includes the failure cause. |
| INVESTIGATING | Team is looking into the issue. |
| IDENTIFIED | The root cause has been found. |
| MONITORING | A fix has been applied and the team is watching for recovery. |
| RESOLVED | The incident is closed, either automatically or manually. |
Each update includes a message, who posted it (user or system), and a timestamp. This timeline becomes the record of how the incident was handled.
Severity levels
When reviewing an incident, you can assign a severity level for post-mortem analysis:
| Severity | Use for |
|---|---|
| CRITICAL | Complete outage affecting all users. |
| MAJOR | Significant degradation or partial outage. |
| MINOR | Limited impact, affecting a subset of users or features. |
| INFO | No user impact, but worth recording (e.g., brief blip that self-resolved). |
Severity is informational -- it does not affect alerting behavior.
Post-mortems
After resolving an incident, you can add a post-mortem note (Markdown supported). This is useful for documenting:
- What went wrong.
- How it was detected and resolved.
- What will be done to prevent recurrence.
Post-mortems are visible to all team members in the incident detail view.
Viewing incidents
The incidents page in the dashboard shows all incidents with filters for:
- All -- Every incident, past and present.
- Active -- Currently ongoing incidents.
- Resolved -- Past incidents with resolution details.
Each incident shows the affected monitor, start time, duration (or "ongoing"), cause, and acknowledgment status.
Incident duration
Duration is calculated as the time between when the incident was triggered and when it was resolved. For active incidents, the dashboard shows a live counter of how long the outage has been going on.