Apache Module mod_proxy_beacon

Usage example ¶

The following pair of configurations sets up a self-registering balancer. The backends require no knowledge of each other and the proxy needs no pre-declared BalancerMember entries — only an empty balancer with room to grow.

On the reverse proxy:

# Receive backend announcements on the cluster network interface (UDP).
ProxyBeaconListen 0.0.0.0:5555
ProxyBeaconSecret    "hex:00112233445566778899aabbccddeeff"
ProxyBeaconBalancer  cluster

# A backend is dropped from rotation if it does not announce for 30 seconds.
ProxyBeaconTimeout   30

# An initially-empty balancer with spare slots for the dynamic members.
<Proxy balancer://cluster>
  ProxySet growth=16
</Proxy>
ProxyPass        "/" "balancer://cluster/"
ProxyPassReverse "/" "balancer://cluster/"

On each backend server:

# Announce this backend's routable origin to the proxy every 10 seconds (UDP).
ProxyBeaconAddress   proxy.example.com:5555
ProxyBeaconAdvertise http://10.0.0.5:8080
ProxyBeaconSecret    "hex:00112233445566778899aabbccddeeff"
ProxyBeaconInterval  10

When a backend starts it begins sending announcements. The proxy verifies each announcement against the shared secret, adds http://10.0.0.5:8080 as a member of balancer://cluster, and enables it. If that backend later stops announcing for longer than ProxyBeaconTimeout, the proxy disables the member (taking it out of rotation); a subsequent announcement re-enables it.

Member identity ¶

An announced URL identifies a backend origin: scheme://host[:port], and nothing more. The proxy uses the same string as the member's identity everywhere: as the key under which it tracks the backend's lease and replay state, and as the worker name it creates in the balancer. To keep those identities equal, the proxy:

• canonicalizes every announced URL: the scheme and host are lower-cased, the scheme's default port is dropped (HTTP://Backend.Example.com:80 and http://backend.example.com are the same member), and a single trailing / is ignored;
• drops any announcement that is not an origin: one carrying a path (other than /), a query string, a fragment, user information, an invalid port (such as 0), or the unix: socket form. The drop is logged (rate-limited) as unacceptable url and the backend is not added.

Paths are refused because the balancer finds a member by longest prefix match. Announcements for http://10.0.0.5:8080/one and http://10.0.0.5:8080/one/two would be tracked as two members with separate leases but resolve to a single worker, so the lapse of one lease could disable a worker whose other lease was still current. Choose request paths on the proxy with ProxyPass, not through the advertised URL.

A backend that is announced successfully is confirmed to exist in the balancer before it is reported as added, enabled or evicted; an announcement that the proxy cannot turn into a member is logged and retried after a back-off rather than reported as a success.

Ownership and watchdog hand-off ¶

Every worker that mod_proxy_beacon creates is marked with the Bcn status (flag B), which is stored in the worker's shared state and shown by the balancer-manager. The module only enables or evicts workers that carry this mark. A worker of the same name that does not — a BalancerMember from the configuration, or one added by an administrator — is left alone; an announcement for it is dropped and logged as member not managed by beacon.

The announcements are received and the leases kept by a single mod_watchdog child process. If that process exits or is killed, the watchdog singleton moves to another child, which starts with no record of the members it did not add itself. When a child becomes the owner it therefore takes over every worker in the balancer carrying the Bcn mark (logged as took over N existing member(s)):

• each taken-over member gets a fresh lease starting at that moment, so a member that has since fallen silent is evicted ProxyBeaconTimeout after the hand-off instead of staying enabled indefinitely;
• a member that is already disabled is treated as evicted, and its next announcement re-enables it.

A new owner can only start once the watchdog's callback mutex has been released by the process that held it. Whether the default mutex mechanism is released when its holder is killed depends on the platform; if prompt hand-off after a crash matters, select a mechanism that is released on process death for the watchdog-callback mutex with the Mutex directive (for example Mutex sysvsem watchdog-callback).

Administrator control of members ¶

Beacon uses the Disabled status flag to take a member out of rotation when its lease expires, and clears it when the member announces again; it owns that flag for the members it manages. An administrator who disables such a member in the balancer-manager is therefore overridden by the next announcement after an eviction.

To hold a member out of rotation regardless of what it announces, set it to Stopped or Drain instead. Beacon never changes those flags, nor the flag set by an active health check (mod_proxy_hcheck); they combine with the lease state, so a stopped member stays stopped through any number of evictions and re-enablements.

Message validation and limits ¶

The receiver validates every datagram before acting on it:

• the datagram must fit in 2048 bytes and must not contain a NUL byte; a longer one is dropped rather than truncated;
• the MAC is verified, whether or not ProxyBeaconBalancer is set, so even the log-only mode logs only authenticated announcements;
• the authenticated message must have the exact form BEACON key=value key=value ... mac=... (single spaces, known fields not repeated), and ts= must be a plain decimal number of at most 18 digits;
• the url must be a printable-ASCII origin (see Member identity).

The proxy tracks at most 256 announced URLs. A record is released once it belongs to an evicted member, or to an add that failed, and has been silent for more than twice ProxyBeaconMaxSkew (by then a replay of its last message would be stale anyway, so no replay protection is lost). Until then a flood of distinct URLs from a holder of the secret can fill the table and keep new backends out; the members themselves remain in the balancer, and a backend that announces again after its record was released is re-adopted. A balancer's own limit (growth, at most 100) caps how many members can ever be added.

If the UDP socket cannot be opened (an unresolvable address, a port in use), opening is retried with a delay that starts at 30 seconds and doubles up to 10 minutes. Name lookups are synchronous and run on the single mod_watchdog thread that serves every beacon configuration of the server, so a name that fails to resolve can stall them for as long as the resolver takes to give up. Prefer an IP address in ProxyBeaconAddress and ProxyBeaconListen.