Clustering — variable replica count without fixed addresses¶

Goal: N GeoServer replicas (≥ 3, any number: kubectl scale, k9s, Rancher…) that share the same configuration in the database, discover each other without fixed IP addresses, propagate their configuration changes, keep the administration UI usable and serve asynchronous WPS from any node.

flowchart TB
    subgraph K8s["Namespace"]
        ING[Ingress<br/>GSROUTE cookie affinity] --> SVC[geoserver Service<br/>ClusterIP :8080]
        SVC --> P1[GeoServer pod 1]
        SVC --> P2[GeoServer pod 2]
        SVC --> P3[GeoServer pod N]
        HZ[(Headless Service geoserver-hz<br/>clusterIP: None<br/>publishNotReadyAddresses: true)]
        P1 <-. "Hazelcast :5701<br/>DNS discovery" .-> P2
        P2 <-. "Hazelcast :5701" .-> P3
        P1 <-. "Hazelcast :5701" .-> P3
        P1 & P2 & P3 --- HZ
        PVC[(RWX PVC geoserver-wps)]
        P1 & P2 & P3 --- PVC
        PDB[PodDisruptionBudget minAvailable: 1]
    end
    DB[(Shared TimescaleDB<br/>geoserver schema)]
    P1 & P2 & P3 --> DB

Hold "Alt" / "Option" to enable pan & zoom

Decisions: ADR-0008. Artifacts: poc/03-clustering/ (manifests, deploy.sh, tests/) plus evolutions of poc/02-geoserver-image/ (bundle, bootstrap, plugin).

Prerequisites¶

Sub-projects 1 and 2 (database, image, Secrets, geoserver-env ConfigMap).
An Ingress controller (ingress-nginx on the test bench; Traefik documented).
RWX storage for asynchronous WPS results (Longhorn on the test bench). Without RWX: disable result storage, or accept that a result is only readable from the node that produced it.

How it works¶

What actually happens — membership, and what is (not) synchronized¶

There is no separate clustering server: Hazelcast runs embedded inside each GeoServer JVM, and the replicas form the cluster themselves, peer-to-peer. The headless Service involved is not a process — it is a DNS entry that Kubernetes keeps up to date with the pod IPs.

How the cluster forms:

At startup, a pod resolves the headless Service name. The DNS answer is only a list of candidates: the IPs of the pods behind the Service at that moment.
The pod opens a TCP connection to each candidate on port 5701. During the handshake both sides compare their cluster name — a member of another deployment is rejected.
The oldest member acts as coordinator: it owns the authoritative member list and distributes it to everyone (that list is the Hazelcast cluster … size=N, members=[…] line in the logs). If the coordinator dies, the oldest surviving member takes over — nothing to administer.
From then on, permanent heartbeats between all members: a pod that stops answering is evicted after a few seconds; a new pod joining is added. Membership therefore tracks the Deployment's replica count with no configuration.

What travels over the cluster — deliberately almost nothing:

Invalidation events (sync_method=event): when a node changes the catalog or the configuration, it first writes to the database (the source of truth), then publishes a small "object X changed" message. The other nodes drop their in-memory copy of that object and re-read it from the database — that is the measured ~3 s propagation. The message never carries the object itself. Where the ~3 s comes from: event delivery is milliseconds; 2 s of it is HZ_SYNC_DELAY=2, a deliberate debounce so that a burst of changes (one catalog operation often touches several objects) is coalesced into one re-read instead of many — tunable down if faster propagation matters more. This only concerns rare administrative configuration changes: new data needs no propagation at all (the mosaic queries the granule index at request time).
UI sessions (optional, HZ_SESSION_SHARING): a replicated map of admin UI sessions, so a user survives the loss of the node that held their session.

Failure analysis: if the Hazelcast layer were entirely down, replicas would keep serving correct maps (everything is in the database); at worst a node would serve a few seconds of stale in-memory configuration until its next reload. The cluster is a propagation accelerator, not a hard dependency.

Propagation: the `gs-hz-cluster` module (Hazelcast)¶

Each pod is a member of a Hazelcast cluster (port 5701). Any change to the catalog or the configuration on one node is published to the others, which refresh their in-memory copy (sync_method=event).
Module prerequisites: JDBCConfig + JDBCStore (in place since sub-project 2).
The bundle (module + Hazelcast 5.3.8 + dependencies) is resolved by Maven from the release POM on the OSGeo Nexus and filtered against the JARs already present in the image (bundles/hz-cluster/pom.xml, ARG WITH_HZ_CLUSTER) — the method from ADR-0007.

Discovery through Kubernetes DNS (no IPs, no RBAC)¶

The headless Service geoserver-hz (clusterIP: None) publishes one DNS entry per pod; Hazelcast queries it (<kubernetes><service-dns>). Two non-negotiable points, both hit at the prototype stage:

publishNotReadyAddresses: true: pods that are still starting (not yet Ready) must be visible, otherwise on a cold boot each pod only sees itself and forms its own cluster.
Check before Tomcat: the bootstrap resolves HZ_SERVICE_DNS; if it does not resolve, the container stops with an explicit message (exit code 4) within a few seconds — no startup stuck in a loop.

The configuration (cluster/cluster.properties, cluster/hazelcast.xml) is regenerated at every startup from the HZ_* variables; the cluster directory is added to JDBCStore's ignoreDirs, otherwise the module reads an (empty) configuration from the database and falls back to its defaults.

Variable	Default	Role
`HZ_ENABLED`	`true`	enables the module
`HZ_DISCOVERY`	`kubernetes`	`kubernetes` (DNS) or `tcp-ip` (loopback, Docker tests)
`HZ_SERVICE_DNS`	—	`geoserver-hz.<namespace>.svc.cluster.local`
`HZ_CLUSTER_NAME`	`geoserver`	cluster name
`HZ_SYNC_METHOD` / `HZ_SYNC_DELAY`	`event` / `2`	propagation
`HZ_SESSION_SHARING` / `HZ_SESSION_STICKY`	`false` / `true`	Wicket session sharing (see below)
`WPS_STORAGE_DIR`	—	shared WPS directory (declared by the plugin)

Administration UI sessions¶

Without any mechanism, behind a round-robin Service, the UI logs out on every request (measured: 0/10 authenticated requests). Two validated solutions:

Solution	Result	When
Session affinity at the Ingress (`GSROUTE` cookie, ingress-nginx; Traefik: `traefik.ingress.kubernetes.io/service.sticky.cookie`)	10/10	Default: no code, sessions not replicated, standard
`HZ_SESSION_SHARING=true` (Wicket sessions replicated by Hazelcast, `HzSessionShareFilter` filter injected into `web.xml` by the bootstrap)	10/10 under true round-robin	Without a sticky-capable Ingress, or to survive the loss of the node holding the session

Shared WPS¶

Execution status in the database (wps-jdbc, sub-project 2) + results on the RWX PVC mounted by all pods (/opt/geoserver_wps, declared in the WPS configuration by the geoserver-init plugin): an execution submitted on one node can be tracked and read from any other one.

Node identification¶

GEOSERVER_NODE_OPTS=id:${HOSTNAME} (production guide "make cluster nodes identifiable"): the pod name appears in the UI header.

Steps¶

cd poc/02-geoserver-image && IMAGE=<registry>/meteo-gis/geoserver:3.0.1-poc3 ./build.sh --push
cd ../03-clustering && IMAGE=<registry>/meteo-gis/geoserver:3.0.1-poc3 ./deploy.sh
kubectl -n meteo-gis scale deployment/geoserver --replicas=5      # or k9s, Rancher…

deploy.sh applies: headless Service, WPS PVC, PodDisruptionBudget, Ingress, geoserver-cluster-env ConfigMap, Deployment with 3 replicas (preferred anti-affinity, port 5701, WPS volume, terminationGracePeriodSeconds: 60).

Verification (tests in `poc/03-clustering/tests/`)¶

Cluster formation at deployment time (DNS discovery, size 1 → 2 → 3):

01 Propagation: workspace created through REST on pod A, visible on B within 3 s, deleted on B, absent on A:

02 Scale 3 → 5 → 2 → 3 with continuous GetMap requests: 0 errors out of 95:

03 Cold boot 0 → 3 simultaneously: 3 pods ready in ~55 s, a single cluster:

04 Discovery failure (wrong DNS): explicit stop within 3 s:

05 Sessions: Ingress affinity 10/10; round-robin without sharing 0/10; with HZ_SESSION_SHARING=true 10/10:

06 Cross-node WPS: submitted on A, ProcessSucceeded and the result (939 bytes) read on B, files on the shared volume:

07 Admin password rotation: effective on the other node, old password rejected:

Node identifier in the UI:

Deviations and pitfalls observed¶

Very first deployment on an empty configuration database: all replicas run the first-boot initialization (schema creation + import) concurrently; one wins, the others can exceed the startup probe and get restarted once before converging. Harmless but noisy — for the initial deployment, start with replicas: 1 and scale up once the first pod is Ready (observed on a fresh environment; every later cold boot is fast since the database is already initialized).

A concurrent first boot can also leave duplicate catalog entries (e.g. the same coverage/layer created twice by two racing initializers). Symptom: the layer works through the workspace virtual service (/geoserver/<ws>/wms) but the global endpoint (/geoserver/ows, used by Layer Preview) returns LayerNotDefined, and GetCapabilities lists the layer twice. Cleanup: DELETE /rest/workspaces/<ws>/coveragestores/<store>?recurse=true for the affected stores, then restart one pod — its initializer recreates them once, cleanly. Restart any remaining pod that still serves a stale reader (Failed to create reader in the WMS exception).
cluster/ must be in JDBCStore's ignoreDirs (otherwise gsEventCluster / reload defaults apply).
Hazelcast on JDK 21: add the recommended --add-opens/--add-modules java.se (done in EXTRA_JAVA_OPTS).
HzSessionShareListener does not exist in 3.0.1: only the filter needs to be declared (initialization is handled by a Spring bean).
The Failed to publish resource notification, cluster not initialized (yet) warnings at startup are normal (before the cluster forms).
Tests: kubectl port-forward svc/… pins one pod (it does not exercise round-robin); kubectl run --rm -i sometimes loses the output — the tests go through kubectl exec in a GeoServer pod.