Skip to main content

Incidents

An incident represents a period of downtime for a monitored service. Sentro creates incidents automatically when monitors detect failures and resolves them automatically when the service recovers.

Incident lifecycle​

TRIGGERED  -->  ACKNOWLEDGED  -->  RESOLVED
| ^
+----------------------------------+
(auto-resolved when service recovers)

TRIGGERED​

Created automatically when a monitor check fails (or when a heartbeat ping is missed beyond the grace period). The incident records:

  • Which monitor detected the failure.
  • When the failure started.
  • The cause (e.g., "Expected status 200, got 503" or "Request timed out after 30000ms").

Alerts are dispatched to all matching alert rules at this point.

ACKNOWLEDGED​

A team member has seen the incident and is investigating. Acknowledgment is optional but useful for coordination -- it tells the rest of the team that someone is on it. The dashboard shows who acknowledged the incident and when.

RESOLVED​

The incident is closed. This happens in two ways:

  • Automatically -- When the monitor's checks start succeeding again, Sentro resolves the incident and records the duration.
  • Manually -- A team member can resolve an incident from the dashboard.

Recovery alerts are sent to channels with a UP trigger rule.

Incident timeline​

Every incident has a timeline of updates that documents what happened:

Update typeDescription
TRIGGEREDSystem-generated when the incident is created. Includes the failure cause.
INVESTIGATINGTeam is looking into the issue.
IDENTIFIEDThe root cause has been found.
MONITORINGA fix has been applied and the team is watching for recovery.
RESOLVEDThe incident is closed, either automatically or manually.

Each update includes a message, who posted it (user or system), and a timestamp. This timeline becomes the record of how the incident was handled.

Severity levels​

When reviewing an incident, you can assign a severity level for post-mortem analysis:

SeverityUse for
CRITICALComplete outage affecting all users.
MAJORSignificant degradation or partial outage.
MINORLimited impact, affecting a subset of users or features.
INFONo user impact, but worth recording (e.g., brief blip that self-resolved).

Severity is informational -- it does not affect alerting behavior.

Post-mortems​

After resolving an incident, you can add a post-mortem note (Markdown supported). This is useful for documenting:

  • What went wrong.
  • How it was detected and resolved.
  • What will be done to prevent recurrence.

Post-mortems are visible to all team members in the incident detail view.

Viewing incidents​

The incidents page in the dashboard shows all incidents with filters for:

  • All -- Every incident, past and present.
  • Active -- Currently ongoing incidents.
  • Resolved -- Past incidents with resolution details.

Each incident shows the affected monitor, start time, duration (or "ongoing"), cause, and acknowledgment status.

Incident duration​

Duration is calculated as the time between when the incident was triggered and when it was resolved. For active incidents, the dashboard shows a live counter of how long the outage has been going on.