Backend Load Balancing

Backend Load Balancing

Radiator can use multiple backend servers. This gives you load balancing and high availability.

With multiple servers, Radiator can:

• send all traffic to one preferred server and keep others as backup,
• spread traffic across many servers,
• prefer the least busy server,
• skip unhealthy servers, and
• return a recovered server back to use,
• keep warm connections ready for faster authentication and faster failover.

These behaviors are controlled by server-selection, server health settings, and connection pool settings.

Quick Decision Guide

How Radiator Chooses a Server

Radiator supports four server selection methods. Configure them with server-selection. The default is fallback.

fallback

Radiator tries servers in priority order.

• The server with the lowest priority value is tried first.
• Lower-priority servers are only used when a higher-priority server is not usable.
• This is the default behavior.

Use fallback when you want a primary server and backup servers.

backends {
    radius "BACKEND_CLUSTER" {
        server-selection fallback;  # or omit, fallback is default

        server "PRIMARY" {
            priority 0;  # Highest priority, tried first
            # ...
        }

        server "SECONDARY" {
            priority 1;  # Lower priority, backup
            # ...
        }
    }
}

round-robin

Radiator rotates requests across all usable servers.

• Request 1 goes to one server.
• Request 2 goes to the next server.
• If one server fails, Radiator tries the next usable server.

Use round-robin when all servers can handle the same work.

backends {
    radius "BACKEND_CLUSTER" {
        server-selection round-robin;

        server "SERVER1" {
            # ...
        }

        server "SERVER2" {
            # ...
        }

        server "SERVER3" {
            # ...
        }
    }
}

least-connections

Radiator sends the request to the usable server with the fewest busy connections.

If more than one server is tied:

• the lower priority value wins first,
• then server name order is used.

This method is available for SQL backends (PostgreSQL and MySQL), LDAP backends, and RADIUS backends.

Use least-connections when backend work time varies and you want to avoid sending too much work to one busy server.

backends {
    postgres "USER_DB" {
        server-selection least-connections;

        server "replica1" {
            host "pg-replica1.example.com";
            connections 15;
            min-connections 2;
            idle-timeout 5m;
        }

        server "replica2" {
            host "pg-replica2.example.com";
            connections 15;
            min-connections 2;
            idle-timeout 5m;
        }

        query "FIND_USER" { ... }
    }
}

no-fallback

Radiator only tries the first usable server.

If that server fails, the request fails immediately. Radiator does not try another server.

Use no-fallback when you want backend problems to be visible immediately.

backends {
    radius "BACKEND_TEST" {
        server-selection no-fallback;

        server "SINGLE_SERVER" {
            # ...
        }
    }
}

Server Priority

priority controls server order for fallback and no-fallback.

• Allowed values: 0 to 255
• Default: 0
• Lower number means higher priority

Example:

backends {
    radius "TIERED_BACKENDS" {
        server-selection fallback;

        server "PRIMARY_DC" {
            priority 0;  # Highest priority, tried first
            # ...
        }

        server "SECONDARY_DC" {
            priority 1;  # Tried if PRIMARY fails
            # ...
        }

        server "DR_SITE" {
            priority 2;  # Last resort
            # ...
        }
    }
}

If two servers have the same priority, Radiator uses alphabetical server name order.

How Radiator Decides a Server Is Unhealthy

Radiator can stop using a server in two different ways:

1. service-level-objective - Service Level Objective based health control for repeated request failures.
2. min-connections based fast skip when a server has zero live connections.

These two mechanisms are different and can work together. See Backend Differences At A Glance for backend-specific support.

Server Health and Status

Each backend server tracks its own health state. Radiator can then:

• mark servers as unavailable after connection failures
• skip unavailable servers during selection
• mark servers as degraded when the service-level-objective threshold is reached
• exclude degraded servers from normal routing until probe requests succeed
• keep warm connections with min-connections, where configured (see Fast Skip With min-connections)
• attempt to reconnect to failed servers periodically
• return servers to service when connections succeed

Service Level Objective

Each backend server supports a service-level-objective block.

Use this when you want Radiator to stop using a server after repeated request failures and to return that server to use after successful probe requests.

When the configured limit is reached:

• Radiator marks it as degraded,
• stops sending normal traffic to it, and
• sends only occasional probe requests until it recovers.

See Backend Differences At A Glance for backend-specific support.

When you configure server-selection, Radiator automatically applies the default service-level-objective values to each server that does not already have its own block:

• failure-rate 3/5
• initial-backoff-period 3s
• max-backoff-period 30s
• recovery-probe-count 2

Add an explicit service-level-objective block on a server if you want different values. For a backend with only one server, add the block explicitly if you want this health check. For parameter details and examples, see service-level-objective.

Fast Skip With min-connections

min-connections tells Radiator to keep a minimum number of live connections open.

This setting is supported only for SQL and RADIUS.

When min-connections > 0, Radiator gets three behaviors at the same time:

1. Warm connections are kept ready.
2. A server with zero live connections is skipped immediately.
3. The Unavailable counter increases when that skip happens.

Set min-connections to at least 1 when fast failover matters.

With the default 0:

• the pool is lazy,
• Radiator opens connections only when needed, and
• Radiator learns that a server is bad only after a real request fails.

How a Backend Request Is Handled

The request flow below applies to SQL, RADIUS, and LDAP backends.

In the normal case, an idle connection or socket is already available and the request is sent immediately. The diagrams below show the more important failover and pool decisions.

Before the pool-side checks below, server selection already filters out servers that are currently degraded by the service-level-objective. If min-connections is in use, the request path can also fast-skip a server that has zero live connections. See Fast Skip With min-connections.

Connection acquisition

Multi-server routing

Connection Pool Parameters

SQL (PostgreSQL, MySQL), RADIUS, and LDAP backends all maintain a connection pool per server block. The parameters below tune that pool. Most apply to all three backend families; protocol-specific parameters and behavior are called out under Backend-specific notes.

connections - pool size per server

Default: 10

This is the maximum number of open connections or sockets to one backend server.

For SQL, each connection serves one query at a time.

You can estimate pool size with:

connections = target_tps × (round_trip_ms + query_ms) / 1000

Example: with 10 ms network round-trip time and 1 ms query time at 1000 requests per second, you need about 11 connections.

For RADIUS the same limit applies, but each backend socket can carry up to 256 requests in flight at the same time. Because of that, the same number of sockets can support much higher request rates. See Backend-specific notes.

For LDAP the limit applies to the shared connection pool used for server-bound search operations. Per-user bind operations use a separate exclusive-connections pool (also default 10).

min-connections - warm connection level

Default: 0 (disabled). Supported by: SQL and RADIUS only.

This is the number of connections Radiator tries to keep open all the time.

When you set min-connections above zero, Radiator gets three behaviors:

1. Warm connections. A background task keeps the pool at this level and reopens a replacement immediately when idle-timeout closes a connection that would take the pool below this level. The first request does not wait for a new TCP or TLS connection setup, which improves authentication performance.
2. Unreachable-server detection. When the pool has zero established connections, Radiator skips that server immediately during server selection and fails over to the next server in the list. See Multi-server routing.
3. Unavailable counter. Each fast-skip increments a dedicated Unavailable counter.

Set min-connections to at least 1 on every server where fast failover matters.

With the default 0, load balancing still works, but fast skip is disabled. Radiator only learns that a server is unreachable after a real request fails.

Single-server backends: min-connections is an HA feature. On a backend with only one server, min-connections >= 1 makes requests fast-fail while the server is unreachable. The request path does not retry the dead server. Only the background maintainer does.

Error: min-connections must not exceed connections.

idle-timeout - idle connection lifetime

This is how long a connection above min-connections can stay idle before being closed. Accepts duration units. Default 5m for SQL; for RADIUS and LDAP idle connections are kept open indefinitely unless this is set.

timeout - request time budget

This limits the total time for connection acquisition plus the backend operation.

Where you configure it depends on the backend type:

• SQL: on the backend action inside an AAA handler, not on the backend or server blocks. This is set per query because SQL queries can take different amounts of time.

  @execute {
          backend {
                  name "USERS";
                  query "FIND_USER";
                  timeout 5s;
          }
  }
  

• RADIUS, LDAP: on the server block, as the per-request response deadline.

Backend-specific notes

SQL uses one connection per outstanding query. The pool grows on demand up to connections and shrinks back toward min-connections when idle. Both PostgreSQL and MySQL behave identically with respect to load balancing.

RADIUS can keep up to 256 requests in flight at the same time on one backend socket. This applies to UDP and TCP/TLS. Radiator keeps reusing existing live sockets until they can no longer accept more requests, only then opening another up to connections. For UDP, Radiator still maintains a pool of backend sockets, but opening a UDP socket does not prove that the remote server is reachable. RADIUS servers additionally support:

• retries (default 2): how many retransmissions are attempted on the same server before server selection tries another server or returns an error.
• status (default false): enables periodic Status-Server health checks.

When status false, failed servers do not recover through Status-Server probes. If min-connections > 0, Radiator can still reconnect them through the background connection maintainer. Enable status true if you want periodic active health checks in addition to min-connections recovery.

Known limitation (RADIUS): Do not set min-connections equal or very close to connections. During recovery, reconnect attempts and new requests can run at the same time and push the open connection count above connections.

LDAP maintains two pools per server:

• A shared pool (connections, default 10) of connections bound with the credentials in the authentication block, used for search operations. shared-connections is accepted as an alias for connections. When a shared connection is reused from the pool, Radiator validates it with an LDAP WhoAmI round-trip.
• An exclusive pool (exclusive-connections, default 10) of connections that are rebound with each user's credentials for bind operations and returned to the pool afterwards. These connections only get a structural liveness check before reuse because they are rebound anyway.

LDAP does not implement min-connections or the unreachable-server fast-skip. Both pools are fully lazy; an unreachable LDAP server is only detected after a request fails. LDAP also does not have a configurable periodic heartbeat like RADIUS status. Consequently the Unavailable counter is not emitted for LDAP.

For the full list of server options see the per-backend reference pages linked from Related Documentation.

Backend Differences At A Glance

Pool Monitoring

Pool metrics are available through the Management API.

Counter Names

Every backend server exposes the same counter structure under its own prefix:

• PostgreSQL: backend/Postgres/{NAME}/{SERVER}/...
• MySQL: backend/MySQL/{NAME}/{SERVER}/...
• LDAP: backend/LDAP/{NAME}/{SERVER}/...
• RADIUS: backend/RADIUS/{NAME}/{SERVER}/...

Common Counter Meanings

Backend-Specific Differences

• SQL: Requests and Replies count queries. PostgreSQL and MySQL use the same counter layout.
• LDAP: Requests and Replies count LDAP operations. LDAP does not emit Unavailable because it does not implement min-connections fast-skip.
• RADIUS: PoolExhausted means all RADIUS identifiers were in use across the current connections, not only that all connection slots were full.

Pool Size Log Fields

Pool size metrics are included as fields in structured log messages:

In-use connections = pool_total - pool_idle. If this stays near the maximum under normal load, increase the pool size or add more servers.

Common Patterns

SQL Primary/Replica (Active-Passive)

Use fallback when you want one preferred database server and one backup:

backends {
    postgres "USERS" {
        server-selection fallback;

        server "primary" {
            host "pg-primary.example.com";
            database "radiator";
            username "radiator";
            password env.DB_PASSWORD;
            connections 10;
            min-connections 2;
            idle-timeout 5m;
            priority 0;
        }

        server "replica" {
            host "pg-replica.example.com";
            database "radiator";
            username "radiator";
            password env.DB_PASSWORD;
            connections 10;
            min-connections 2;
            idle-timeout 5m;
            priority 1;
        }

        query "FIND_USER" {
            statement "SELECT username, password FROM users WHERE username = $1";
            bindings { aaa.identity; }
            mapping { user.password = password; }
        }
    }
}

SQL Read Scaling (Active-Active)

Use round-robin to spread read queries across many replicas:

backends {
    postgres "USER_DB_READ" {
        server-selection round-robin;

        server "REPLICA1" { host "pg-replica1.example.com"; }
        server "REPLICA2" { host "pg-replica2.example.com"; }
        server "REPLICA3" { host "pg-replica3.example.com"; }

        query "FIND_USER" { ... }
    }
}

LDAP Round-Robin

Use round-robin when both LDAP servers can handle the same searches:

backends {
    ldap "LDAP_CLUSTER" {
        server-selection round-robin;

        server "LDAP1" {
            url "ldap://ldap1.example.com:389/";
            timeout 3s;
            authentication {
                dn "cn=radiator,dc=example,dc=com";
                password "ldap_password";
            }
        }

        server "LDAP2" {
            url "ldap://ldap2.example.com:389/";
            timeout 3s;
            authentication {
                dn "cn=radiator,dc=example,dc=com";
                password "ldap_password";
            }
        }

        search "AUTHENTICATE" {
            base "ou=users,dc=example,dc=com";
            scope sub;
            filter "(&(uid=%{aaa.identity})(objectClass=inetOrgPerson))";
            mapping {
                user.username = uid;
                vars.dn = entry::dn;
            }
        }
    }
}

RADIUS Active-Passive Failover

Use fallback when one RADIUS server should be preferred and another should be backup:

backends {
    radius "ACTIVE_PASSIVE" {
        server-selection fallback;

        server "PRIMARY" {
            priority 0;
            secret "mysecret";
            timeout 3s;
            retries 2;
            min-connections 1;
            connect { protocol udp; host "192.168.1.10"; port 1812; }
        }

        server "BACKUP" {
            priority 1;
            secret "mysecret";
            timeout 3s;
            retries 2;
            min-connections 1;
            connect { protocol udp; host "192.168.1.11"; port 1812; }
        }
    }
}

Geo-Distributed Backends

Use fallback with priority when you want to prefer a local data center and only use a remote one when needed:

backends {
    radius "GEO_DISTRIBUTED" {
        server-selection fallback;

        server "LOCAL_DC" {
            priority 0;
            min-connections 1;
            timeout 2s;
            # ...
        }

        server "REMOTE_DC" {
            priority 1;
            min-connections 1;
            timeout 5s;  # Higher timeout for WAN
            # ...
        }
    }
}

Related Documentation

• Service Level Objective - how Radiator marks a server degraded and how it recovers
• High Availability and Load Balancing - wider high availability design patterns
• PostgreSQL server-selection - PostgreSQL selection reference
• MySQL server-selection - MySQL selection reference
• LDAP server-selection - LDAP selection reference
• RADIUS server-selection - RADIUS selection reference
• Prometheus Scraping - metrics collection and monitoring