RadiatorDB Maintenance

RadiatorDB Maintenance

This article describes the states that RadiatorDB nodes and collections go through, and how node removal and collection changes move them between these states. For the initial deployment, see RadiatorDB installation.

The RadiatorDB maintenance view of Radiator UI shows the current state of every node and collection, removes and reprovisions nodes, and destroys collections.

Node Lifecycle

A node is one PostgreSQL database defined by a RadConf server block in the RadiatorDB backend. Radiator Server sets up every configured node in the background, publishes it to request routing, and then monitors it with health checks.

Each Radiator Server process tracks the state of the nodes separately. When Radiator Server starts, every configured node starts from Waiting for connection, including nodes that were removed before the restart.

Remove Node accepts a node in any state. The diagram shows only the most common starting states.

Node States

The delay between automatic retries grows after each consecutive failure, up to retry_max_delay. Each setup attempt is limited by setup_timeout.

Inactive, Waiting for recovery, Failed setup, and Intervention required also show what the node requires, for example Failed setup: repair required:

• not checked: Radiator has not inspected the node yet. Only Inactive shows this.
• setup required: the node has no completed setup.
• configuration apply required: the stored state of the node does not match the current configuration or schema, or the node configuration is invalid, for example its database name.
• repair required: the stored state of the node needs repair.
• temporarily unavailable: PostgreSQL is temporarily unavailable.

Node Health

Each node also has a health that is separate from its state. Request routing uses only serving nodes whose latest health is Available or Degraded.

Repair a Node

A node in Failed setup or Intervention required stays there until you act. Fix the cause that the latest error reports, and then do one of the following:

• Run Remove Node, and then Reprovision Node.
• Restart Radiator Server.

If the error says that Remove Node did not complete, resolve the reported problem and run Remove Node again before you reprovision the node.

Remove a Node

Remove Node removes the node from request routing, deletes its subscriptions, replication slots, and RadiatorDB storage, and moves it to Deprovisioned. PostgreSQL objects that RadiatorDB does not own are kept.

The node stays configured. To remove the node permanently, also remove its RadConf server block from the RadiatorDB backend in every Radiator configuration. Otherwise Radiator Server provisions the node again on the next restart.

Remove Node waits until every remaining node has applied the removed node's writes, because no other node can supply those writes after removal. See Effect on Traffic. If any configured node is unreachable, or a remaining node cannot apply the removed node's writes within RADIATORDB_NODE_REMOVAL_TIMEOUT, Remove Node fails and keeps the node's storage. If the failure happens after Remove Node took the node out of request routing, the node moves to Intervention required. Run Remove Node again after you resolve the reported problem.

Reprovision a Node

Reprovision Node is available only for a deprovisioned node. It moves the node to Waiting for connection, and setup provisions the node again. The node creates the configured collections and copies the documents from a node that is already serving.

A deprovisioned node also returns to setup when Radiator Server restarts, as long as its RadConf server block is still configured.

Force Remove a Node

Forced removal does not wait for the remaining nodes to apply the removed node's writes. It skips every configured node, including the removed node, that does not answer one connection attempt within health_check_timeout. Errors on reachable nodes still fail the removal. Radiator logs RadiatorDB node force removal completed, which names the skipped nodes and the remaining nodes that may hold different data.

Forced removal has these consequences:

• Writes from the removed node that were not yet replicated to a remaining node are lost. The remaining nodes may permanently hold different data.
• If the removed node was skipped, its storage stays in its database. Reprovision Node in the same Radiator Server process deletes that storage before the initial copy. After a Radiator Server restart, setup of the node fails instead. Run Remove Node again while the node is reachable, and then reprovision it.
• A skipped remaining node keeps its subscription from the removed node and its replication slot for the removed node. The slot retains PostgreSQL write-ahead log (WAL) on that node. Delete the removed node from the configuration and restart Radiator Server to delete these resources. To reprovision the removed node instead, run Remove Node again while the skipped node is reachable, and then reprovision it.

Collection Lifecycle

A collection is defined by a RadConf collection block in the RadiatorDB backend. Each node stores its own copy of every collection. A collection has one cluster-wide state, which Radiator determines from the configuration and from every node that it can inspect. Unreachable and deprovisioned nodes do not affect the state.

Collection States

The storage of a collection is either normal storage or time-series storage with its timeseries partition policy.

For each collection, Radiator UI also shows its presence on each node:

• Stored: the node stores the collection, with the shown storage.
• Not stored: the node was inspected and does not store the collection.
• Unknown: the node could not be inspected.
• Deprovisioned: the node is deprovisioned and was not inspected.

Add a Collection

When you add a RadConf collection block to the RadiatorDB backend and restart Radiator Server or reload the configuration, the collection is Pending. Setup creates the collection on every configured node at once. It does so only when every configured node that is not deprovisioned is set up and serving. While the collection is Pending, the management API returns 503 for it, and pipelines cannot use it. Collections that every node already stores are served as before. After the creation, the collection is Configured and Radiator logs RadiatorDB collection created.

The creation also waits while any Radiator Server process sets up a node, and while the PostgreSQL server of the first configured node in name order is unreachable. Radiator retries the creation when a node finishes setup, and otherwise at most once every retry_max_delay. A Pending collection shows what the creation waits for.

A new or reprovisioned node does not delay the creation. It creates the collection during its own setup and copies the documents from a node that is already serving. Until its setup completes, the collection is Partially stored.

If some nodes store documents in the collection while other set-up nodes do not store it, the collection is Leftover. Setup refuses to create it, because the nodes would hold different documents permanently. This happens after a destroy that did not complete on every node. The collection stays unavailable and Radiator logs RadiatorDB collection creation refused. Destroy the collection again, and setup creates it empty on every node. To keep the documents, see Keep leftover documents.

Remove a Collection

Removing a RadConf collection block does not delete the collection's data. After the next restart or configuration reload, the collection is Not configured. Every node keeps the documents, revisions, and storage of the collection as a retained collection. Radiator Server does not serve a retained collection: the management API returns 404 for it, pipelines cannot use it, and garbage collection and time-series partition maintenance skip it. An unfiltered backup export or import does not include retained collections.

If you add the RadConf collection block again with the same storage, the collection is Configured again and Radiator Server serves the retained data. If the storage differs, the collection has a storage conflict, as described in Change the storage of a collection.

A new or reprovisioned node cannot complete its initial copy while it stores a retained collection. The node shows Waiting to retry setup until you destroy the retained collections.

To delete a collection permanently:

1. Remove its RadConf collection block from the RadiatorDB backend in every Radiator configuration.
2. Restart Radiator Server or reload the configuration. The collection is Not configured.
3. Destroy the retained collection.

Change the Storage of a Collection

When the configured storage of a collection differs from the storage that a node stores, the collection is Storage conflict. Setup of every node that stores the collection fails, and the node moves to Failed setup. The collection stays in this state until you restore the previous storage configuration or destroy the collection.

To change the storage of a collection, for example from normal storage to time-series storage or to another timeseries partition policy:

1. Change its RadConf collection block in every Radiator configuration.
2. Restart Radiator Server or reload the configuration. The collection is Storage conflict.
3. Destroy the collection.
4. Restart Radiator Server or reload the configuration again. Setup creates the collection empty with the new storage.

If some nodes store the collection with a different storage while other nodes store documents in it with the configured storage, the collection is Reprovision required. Destroy Collection refuses it, because the destroyed nodes would never receive those documents. Remove each node with the different storage, and then reprovision it to copy the collection from the other nodes.

Destroy a Collection

Destroy Collection deletes the documents, revisions, partitions, and registry entry of a collection. It destroys only collections that are Not configured, Storage conflict, or Leftover.

A Storage conflict collection is destroyed only on the nodes whose storage differs from the configuration.

Without force, every configured node that is not deprovisioned must be reachable. Destroy Collection waits until every node has applied the writes of the other nodes before it deletes the collection. See Effect on Traffic. RADIATORDB_COLLECTION_DESTROY_TIMEOUT bounds the connection, replication catch-up, and lock waits of one destroy. If it passes before the first commit, nothing is destroyed. If a commit fails after other nodes have committed, Radiator logs RadiatorDB collection destroy failed. Destroy the collection again to finish. Otherwise Radiator logs RadiatorDB collection destroy completed.

Destroy Collection reports one result for each configured node:

• Destroyed: the collection was destroyed on the node.
• Not stored: the node did not store the collection.
• Kept, stores the configured storage: the node stores the Storage conflict collection with the configured storage, so it was not destroyed there.
• Skipped, still stores the collection: a forced destroy skipped the unreachable node.
• Failed, still stores the collection: the destroy failed on the node.
• Deprovisioned, not changed: the node is deprovisioned.

Force Destroy a Collection

Forced destroy skips every configured node that does not answer one connection attempt within health_check_timeout. Unlike forced removal, it still waits until the reachable nodes have applied each other's writes. Radiator logs RadiatorDB collection force destroy completed, which names the skipped nodes.

Forced destroy has these consequences:

• A skipped node keeps the collection and its data. Destroy the collection again when the node is reachable. If you add the RadConf collection block again before that, the collection is Leftover, as described in Add a Collection.
• If a skipped node holds unreplicated writes to the collection, replication from that node to the other nodes stops at those writes when the node is reachable again. Remove the node, and then reprovision it to restore replication. Its unreplicated writes are lost.

Keep Leftover Documents

Destroying a Leftover collection deletes its documents. To keep them, back up the collection before you destroy it, and import the backup after setup has created the collection again:

1. Remove the RadConf collection block from the RadiatorDB backend, and restart Radiator Server or reload the configuration. The collection is Not configured.
2. Export a backup that includes the collection by name. The export reads one node, so check that the backup contains the documents.
3. Destroy the collection.
4. Add the RadConf collection block again, and restart Radiator Server or reload the configuration. Setup creates the collection empty on every node and logs RadiatorDB collection created.
5. Import the backup with the collection included by name.

Effect on Traffic

To avoid losing writes or leaving the nodes with different data, Remove Node and Destroy Collection stop RadiatorDB writes on every reachable node while they run. With writes stopped, the nodes can apply each other's remaining writes before the operation changes anything. Radiator Server retries writes that arrive during this pause until it ends.

The pause lasts about as long as the nodes need to apply the writes that they have not replicated yet. Check the replication lag in the RadiatorDB maintenance view of Radiator UI before you start. Forced removal does not wait for replication, so it stops writes only briefly.

Destroy Collection also briefly blocks reads of every collection on each node while it deletes the collection. A long-running read on a node delays the destroy and the other reads on that node.