alerts

alerts

Alerts and backend incidents are an experimental feature. Their behavior and configuration, including alert states, incident lifecycle, and pipeline variables, may change in a later release. Every alerts block, including those inside backends and servers, enables the feature; configuration checks report it, and the server logs an Experimental feature enabled message. About experimental features.

Defines alerts against counter rates and gauge values. Each instance evaluates its own metrics. Alerts and their group summaries are available in Radiator UI.

An alerts block enables the feature. Without one, Radiator Server evaluates no alerts, tracks no backend incidents, and logs backend status changes as it did before alerts existed. Radiator UI then shows no Alerts page, and counter tiles keep their warning and error icons. Any alerts block, including one in a backend or server, enables metric alerts, automatic log alerts, and backend incidents with their defaults.

alerts {
    scan_interval 1m;
    priorities [critical, high, medium, low, none];

    group "system" {
        description "System resources";
    }

    defaults {
        group "system";
        priority none;
        activate {
            window 5m;
            min_samples 5;
            required 100%;
        }
        clear {
            window 2m;
            min_samples 2;
            required 80%;
        }
        max_sample_age 150s;
        no_data no_data;
        logging {
            active warn;
            cleared info;
            no_data warn;
        }
    }

    alert "MEMORY_PRESSURE" {
        namespace "system::os::MemoryFullPressurePercent";
        threshold > 10;
        no_data disable;
    }
}

scan_interval defaults to 1m and must be positive. Scans continue without metric updates so stale data can be detected. A completed operating-system metric collection also requests a scan. Missed intervals are not replayed.

Defaults merge field by field: built-in settings, global defaults, component defaults, and alert settings. Global defaults apply across configuration files regardless of declaration order.

Automatic log alerts

Documented direct WARN/ERROR messages and messages promoted to WARN/ERROR automatically become alerts when their counters exist. Each member evaluates a separate alert for each full metric namespace and message. Counters created for direct WARN/ERROR events are internal to alert evaluation and excluded from public counter APIs, Prometheus and OpenMetrics. Existing promotion counters remain public metrics. Alert-transition diagnostics are excluded to prevent recursive alerts and actions. Other handcrafted counters do not become log alerts merely because their category is warning or error.

The default window is 15m, the threshold is > 0, and the group is Logs. The value is the latest cumulative counter total minus its total at the window boundary, without a per-second conversion. The trailing interval excludes its start and includes its end: (now - window, now]. The first occurrences count immediately using the counter's initial zero baseline. Existing statistics history supplies available readings when a rule is created or its window changes. An older counter without a retained boundary reading reports no data until a complete baseline is available. Window precision follows the available counter readings and scan cadence. Counters and their history reset when Radiator Server restarts.

alerts {
    builtin_logs {
        window 15m;
        group "Operations";

        override "Configuration warning" {
            threshold == 10..;
            priority low;
        }

        override "Management-requested shutdown failed" {
            silenced true;
        }
    }
}

For threshold == 10.., counts from 1 to 9 remain visible and OK; counts of 10 or more are active. A zero count clears the alert and removes its current row, regardless of its threshold. An unavailable source remains distinguishable from a quiet counter through no-data or stale member status. Sparklines contain the same window deltas as the displayed value, rather than cumulative totals or rates.

Settings in builtin_logs apply to all matching messages. Global and component alert defaults also apply; matching built-in policies override their specified fields. An override "MESSAGE" block selects the exact documented message text. At top level, namespace "PREFIX" restricts a policy to that namespace and its descendants, matching complete path components. Inside a backend or server's alerts block, builtin_logs and its overrides inherit that component's namespace automatically. More specific namespace policies override broader ones; message overrides then override defaults at the same namespace depth. Equally specific policies apply in configuration order, with later values taking precedence.

Supported settings are enabled, window, threshold, group, priority, silenced, no_data, and logging. The @execute pipeline uses the same actions as explicit alerts. window must be positive and uses the standard duration units. Built-in log alerts compare their window delta directly; they do not use separate activate or clear qualification windows. enabled false excludes matching alerts from actions and group status; their counters continue collecting. silenced true keeps measured state while suppressing actions, transition logs, and group participation. Transition logging and hooks are off unless explicitly configured, including through global defaults.

Severity is the declared or promoted log level and does not change when priority is overridden. ERROR events default to the highest configured priority and WARN events to the next priority, or the last available name. Ordinary metric alerts map their configured priority rank to ERROR, WARN, INFO, DEBUG, and TRACE in descending order. Additional priority ranks also map to TRACE.

Backend incidents

Backend incidents are tracked only when the configuration has an alerts block. Disable incident management independently of metric alerts with the top-level incidents setting:

alerts {
    incidents false;
}

The default is true. A successful configuration reload applies this setting without restarting. When disabled, backend incident correlation, incident results, lifecycle logs and incident actions stop. Existing incident state and completed history are discarded without reporting recovery. Backend failure messages continue through ordinary logging and promotion. Metric alert evaluation, transition logging and actions remain configured independently. Re-enabling starts fresh incident tracking; discarded incidents are not restored.

Annotated backend events drive one active incident per backend on each member. Failure causes and individual connections do not create separate incident identities. The stable alert name is BackendIncident.

The first failed attempt opens the incident. Further failures and recovery attempts update the same incident and retain its original cause. Selected root warnings appear once per incident; subsequent attempts remain detailed diagnostics with the same incident ID. Retained root log rows include the current incident summary. File output records the summary available when each row was written. The alert remains active during quiet periods, regardless of the event-count window or missing-data policy. Only an explicit recovery event from the backend resolves it. An ordinary success log does not imply recovery.

Incident results expose the original cause, latest error, cumulative failure and retry counts, incident ID, and resolution timestamp. The latest 64 annotated events retain their timestamps, source namespaces, messages, errors, and lifecycle roles. The original cause and totals remain available when older events leave that recent history.

Verified recovery does not close the incident at once. The incident enters the clearing state and stays there for its clearing period, 15 minutes by default. If the backend reports a new failure during that period, the same incident becomes active again, keeping its incident ID. Only the backend's own failure events reopen an incident; other warnings and errors do not. When the clearing period ends without a new failure, the incident closes and leaves the current alerts. Each member retains its latest ten closed episodes separately. Closed incident history is available in Radiator UI. A failure after an incident has closed opens a new incident with a new ID. These histories remain in memory and reset on restart.

Set the clearing period with clearing_period, for example in defaults or in override "BackendIncident". The default matches the event-count window of automatic log alerts, so the counts that an outage left in grouped alerts have expired by the time its incident closes.

Enabled, unsilenced condition transitions produce Incident opened, Incident clearing, Incident reopened and Incident closed application records under alerts::incidents. The closure includes started_at, cleared_at, duration_seconds, severity, priority, priority_rank, original_message, original_error and cumulative failure/retry counts. The duration is the outage itself, from the first failure to the verified recovery; the clearing period is not included. Duration uses a monotonic clock, so a wall-clock correction does not change the measured lifetime.

While an incident is active or clearing, warnings and errors that request handling reports while calling its backend are grouped beneath the incident instead of raising automatic log alerts of their own. The incident lists them as evidence sources. Once the incident closes, the same messages alert on their own again.

Incident values and sparklines show cumulative failures for the current episode. Lifecycle changes request scans, coalesced to one per second unless the configured scan interval is shorter.

Backend dependencies have separate correlation keys within the same incident. A successful operation only recovers its matching dependency. Every failed dependency must recover before the incident resolves. Authentication denials and lookup misses are healthy dependency responses. Recent events include the dependency key when one applies.

Background connection-pool maintenance also reports failures and confirms recovery when its configured minimum is satisfied. Other messages retain event-count alerting unless explicitly classified as lifecycle evidence.

Removing or replacing the owning backend removes its incident from current results immediately, without running a recovery action. Draining requests from an old configuration cannot keep that incident active in the new configuration. Unchanged runtimes preserve incidents across unrelated configuration changes. Changing a backend's settings closes its incident once the new configuration is in effect, because the failing servers or operations may no longer exist; a reload that fails keeps the incident open. A late result from a retired runtime cannot reopen or resolve a replacement runtime's incident. Condition state remains in memory and resets on restart.

Use builtin_logs defaults and overrides for group, priority, enablement, silencing, transition logging, and pipelines. Select the incident by BackendIncident, independently of its initiating message. Window and numeric threshold settings apply only to event-count alerts.

alerts {
    builtin_logs {
        override "BackendIncident" {
            priority critical;
            group "Backend health";
            clearing_period 10m;
        }
    }
}

Metric targets and units

Each alert needs a unique name within its owner scope, a group, and a numeric threshold condition. A namespace that names a backend or server must refer to one declared in the configuration, and every namespace component must be valid; otherwise the configuration does not load. Supported comparisons are >, >=, <, <=, ==, and !=. Use == or != for exact numeric state codes, including enum-style gauges. Metrics currently expose numeric values; string enum labels are not metric values.

Use .. for an inclusive range. Omit either endpoint for an open end. Combine ranges and exact values in a list. == matches any member of the set; != matches values outside the entire set.

# Alert when the value is outside both permitted bands.
threshold != [10..20, 30..40];

# Alert within either range, or at the exact value 50.
threshold == [10..20, 30..40, 50];

# Alert at or below 5, or at or above 40.
threshold == [..5, 40..];

# Match any of these numeric state codes.
threshold == [2, 4, 7];

# A single range does not require a list.
threshold == 10..20;

# Byte-sized thresholds use native RadConf unit literals.
threshold != [80MiB..160MiB, 256MiB..512MiB];

An exact value is equivalent to a range with identical endpoints. Empty sets, reversed endpoints, and ranges with both endpoints absent are invalid. Use == or != for sets; scalar comparisons such as threshold > 20 still work. For byte-valued metrics, byte-size literals convert to bytes before comparison.

namespace identifies an absolute metric path. Inside a backend or server, use metric for a relative target:

backends {
    radius "UPSTREAM" {
        # Existing backend settings belong here.
        alerts {
            alert "TIMEOUT_RATE" {
                group "backends";
                metric "Timeouts";
                threshold > 5;
            }
        }
    }
}

The target inherits the component's statistics namespace, including the server transport when applicable. Specify exactly one of namespace and metric.

Gauges use their native units. Counters use the difference between consecutive readings divided by elapsed seconds. For a byte counter, the threshold is bytes per second. Counter rates require two raw readings. min_samples counts evaluated values, so 5 rate samples require 6 raw readings. A decreasing counter starts a new baseline without generating a negative rate.

Custom dynamic metrics use explicit custom::... namespace paths. They can be referenced before they exist. Unknown built-in metric names disable the affected alert. Unsupported operating-system metrics remain absent rather than reporting zero.

On Linux, system RAM and swap amounts come only from /proc/meminfo. system::os::SystemMemoryTotalBytes uses MemTotal, and system::os::SystemMemoryAvailableBytes uses MemAvailable from the same snapshot. Unreadable files and missing or invalid fields produce no new samples. Existing samples expire according to max_sample_age, after which the alert applies its no_data policy.

Activation, clearing, and missing data

Activation requires a complete rolling window and at least the configured number of valid samples. The required percentage must be satisfied by both elapsed time and sample count. The built-in activation defaults are window 5m, min_samples 5, and required 100%.

Clearing uses an independent window with defaults window 2m, min_samples 2, and required 80%. The default clearing condition is the inverse of the activation comparison. An optional clearing threshold provides hysteresis and accepts the same scalar or set syntax. Without an explicit clearing threshold, clearing uses the complement of the entire activation condition.

Most gauges keep their value until something changes it, so each scan interval samples the gauge's current value, whether or not it was written recently. The system::os process and memory gauges are different: Radiator Server rewrites them on every sample, so a write older than max_sample_age is stale. A gauge that has never been written has no data. Counters are sampled once per scan interval as rates. max_sample_age defaults to 150s; a counter sample is stale only after its age exceeds this duration.

Use one of these no_data policies when the metric is missing, stale, or has insufficient samples or window coverage:

• ok: report healthy.
• fail: report active without requiring threshold qualification.
• no_data: report unknown. This is the default.
• disable: disable the alert until sufficient fresh data becomes available.

Disabled alerts do not affect group status and do not emit transition effects. Their count remains available for optional inspection.

After startup, a reload that changes evaluation settings, or re-enabling an alert, the first results set a baseline instead of emitting transition effects. Missing data while the first window fills is not reported, and a healthy first result has nothing to clear. An alert that is already active emits its activation immediately. Afterwards, each change from the last reported state emits its logs and actions, so clearing always follows a reported problem.

Groups, silencing, and actions

A group is active if any enabled, unsilenced member is active; otherwise it is unknown if any member is unknown. It is healthy when all participating members are healthy. A group without participating members is unknown.

silenced true preserves evaluation and measured state while suppressing logs, pipelines, and group participation. The default is false.

Logging is optional. Configure active, cleared, and no_data with trace, debug, info, warn, error, or off. Use logging false; to disable inherited transition logs.

An optional @execute block runs every time the alert's state changes, in any direction. Unlike a handler's @execute, it does not run on every scan, only on state changes. It contains ordinary pipeline actions, including if, log, and invoke. The pipeline tells changes apart with alert.state and alert.previous_state, which are ok, active, no_data, or clearing for a backend incident that recovered and is waiting out its clearing period. alert.previous_state is empty for an alert's first delivered change, such as a new incident. Knowing the previous state matters most for no_data: after active, a problem may continue unseen; after ok, monitoring has a gap.

The pipeline runs without a request, so only the alert variables describe the change. Read them in RadConf by path, such as alert.name, or in Lua scripts through context.vars:

alert.incident_id identifies a condition episode; alert.incident is its structured cause and progress, or JSON null for a metric alert. alert.counter_window_seconds is the event-count window for a built-in log alert and none for ordinary metric alerts. alert.threshold_ranges is a JSON array of all inclusive lower/upper bounds; open endpoints are JSON null. alert.condition is the complete compact threshold text for metric alerts and the condition name for backend incidents. For a single interval, alert.threshold and alert.threshold_upper provide its bounds. These two values are none for a union; scalar conditions expose alert.threshold only. Actions execute on the instance that owns the alert. Reading an API snapshot never invokes actions. Actions of one alert run in transition order: a new action starts after the previous one finishes, so a slow activation action cannot be overtaken by its clearing action. A started action completes even when its alert is removed by a reload or its incident closes.

This example writes one line per change to an ALERTS AAA logger defined in the logging configuration. It reports a lost metric differently depending on whether the alert was active before:

alerts {
    alert "MEMORY_PRESSURE" {
        namespace "system::os::MemoryFullPressurePercent";
        threshold > 10;
        @execute {
            if all {
                alert.state == "no_data";
                alert.previous_state == "active";
            } then {
                log "ALERTS" {
                    format "%{alert.name} stopped reporting while active; the problem may continue";
                }
            } else if all {
                alert.state == "active";
            } then {
                log "ALERTS" {
                    format "%{alert.name} is active: %{alert.value} %{alert.condition}";
                }
            } else {
                log "ALERTS" {
                    format "%{alert.name} changed from '%{alert.previous_state}' to %{alert.state}";
                }
            }
        }
    }
}

Reading alert snapshots requires the monitor privilege and never triggers evaluation. Each snapshot reflects the last completed scan. Unavailable cluster members remain separate from disabled alerts.

Priorities

priorities defines the allowed names in descending priority order. Its default is [critical, high, medium, low, none]. Set priority high; in an individual alert, a component's defaults, or the global defaults. With no explicit priority, alerts use the last configured name (none with the default list). Custom lists can use any non-empty, unique names, for example priorities [urgent, routine, unclassified];.

The API publishes the member's ordered priorities and each alert's priority and zero-based priorityRank (zero is highest). Ranks are relative to that member's configuration. A priority that is not in priorities is a configuration error, so the configuration does not load and the running alerts keep their current settings. Transition logs and pipelines receive the priority as priority and alert.priority, respectively. Priority changes preserve measured progress. The alert table places priority beside state and sorts by state, priority, age, and alert name. Priority colors follow the configured order: red, orange, amber, blue, then gray. Silenced, disabled, and stale alerts use muted priority badges.

Active durations display the two largest consecutive whole units, including zero in the second unit: 1d 2h, 1h 0min, 2min 5s. Values below a minute use seconds. Sample counts and window coverage are available by hovering over the Value cell. Unavailable measurements use gray status boxes and sparkline borders.

Alert views and navigation

The default list sorts active alerts first, then no-data results, then OK results. Within each state, the configured priority order comes first, followed by the oldest activation and alert name. Active ages use the elapsed duration reported by each member. Stale and disabled results follow current results.

Click an alert name to open its individual result on the evaluating member. The view stays live and shows its measurement, threshold, qualification progress, last scan, and transition history. The URL includes both source and alert identity, so rules with the same name on different members remain distinct. Missing rules and unavailable members are identified rather than replaced with another result.

Click a namespace to see alerts for that path and its descendants across members. Source links select the evaluating member. The smaller scope label identifies where the rule was declared in RadConf: Top level means the outer alerts block, while component scopes identify the owning backend or server.

Filters combine source, group, priority, state, rule scope, silencing, namespace, and alert-name or description search. They remain in the URL across reloads and can be bookmarked or shared. Clear filters returns to the full list. Disabled rules remain hidden until the small disabled-alert count is selected.

The main toolbar shows a red bell and active-alert count only when current, unsilenced alerts are active. It counts across available cluster members and omits silenced, disabled, and stale results. It disappears when there are no participating active alerts, including configurations without alert rules. Clicking it opens the alert list filtered to active, unsilenced results. The count shares the alert view's live snapshots and is visible to users with Monitor privilege or higher.