Skip to content

Clustering — variable replica count without fixed addresses

Goal: N GeoServer replicas (≥ 3, any number: kubectl scale, k9s, Rancher…) that share the same configuration in the database, discover each other without fixed IP addresses, propagate their configuration changes, keep the administration UI usable and serve asynchronous WPS from any node.

flowchart TB
    subgraph K8s["Namespace"]
        ING[Ingress<br/>GSROUTE cookie affinity] --> SVC[geoserver Service<br/>ClusterIP :8080]
        SVC --> P1[GeoServer pod 1]
        SVC --> P2[GeoServer pod 2]
        SVC --> P3[GeoServer pod N]
        HZ[(Headless Service geoserver-hz<br/>clusterIP: None<br/>publishNotReadyAddresses: true)]
        P1 <-. "Hazelcast :5701<br/>DNS discovery" .-> P2
        P2 <-. "Hazelcast :5701" .-> P3
        P1 <-. "Hazelcast :5701" .-> P3
        P1 & P2 & P3 --- HZ
        PVC[(RWX PVC geoserver-wps)]
        P1 & P2 & P3 --- PVC
        PDB[PodDisruptionBudget minAvailable: 1]
    end
    DB[(Shared TimescaleDB<br/>geoserver schema)]
    P1 & P2 & P3 --> DB
Hold "Alt" / "Option" to enable pan & zoom

Decisions: ADR-0008. Artifacts: poc/03-clustering/ (manifests, deploy.sh, tests/) plus evolutions of poc/02-geoserver-image/ (bundle, bootstrap, plugin).

Prerequisites

  • Sub-projects 1 and 2 (database, image, Secrets, geoserver-env ConfigMap).
  • An Ingress controller (ingress-nginx on the test bench; Traefik documented).
  • RWX storage for asynchronous WPS results (Longhorn on the test bench). Without RWX: disable result storage, or accept that a result is only readable from the node that produced it.

How it works

Propagation: the gs-hz-cluster module (Hazelcast)

  • Each pod is a member of a Hazelcast cluster (port 5701). Any change to the catalog or the configuration on one node is published to the others, which refresh their in-memory copy (sync_method=event).
  • Module prerequisites: JDBCConfig + JDBCStore (in place since sub-project 2).
  • The bundle (module + Hazelcast 5.3.8 + dependencies) is resolved by Maven from the release POM on the OSGeo Nexus and filtered against the JARs already present in the image (bundles/hz-cluster/pom.xml, ARG WITH_HZ_CLUSTER) — the method from ADR-0007.

Discovery through Kubernetes DNS (no IPs, no RBAC)

The headless Service geoserver-hz (clusterIP: None) publishes one DNS entry per pod; Hazelcast queries it (<kubernetes><service-dns>). Two non-negotiable points, both hit at the prototype stage:

  • publishNotReadyAddresses: true: pods that are still starting (not yet Ready) must be visible, otherwise on a cold boot each pod only sees itself and forms its own cluster.
  • Check before Tomcat: the bootstrap resolves HZ_SERVICE_DNS; if it does not resolve, the container stops with an explicit message (exit code 4) within a few seconds — no startup stuck in a loop.

The configuration (cluster/cluster.properties, cluster/hazelcast.xml) is regenerated at every startup from the HZ_* variables; the cluster directory is added to JDBCStore's ignoreDirs, otherwise the module reads an (empty) configuration from the database and falls back to its defaults.

Variable Default Role
HZ_ENABLED true enables the module
HZ_DISCOVERY kubernetes kubernetes (DNS) or tcp-ip (loopback, Docker tests)
HZ_SERVICE_DNS — geoserver-hz.<namespace>.svc.cluster.local
HZ_CLUSTER_NAME geoserver cluster name
HZ_SYNC_METHOD / HZ_SYNC_DELAY event / 2 propagation
HZ_SESSION_SHARING / HZ_SESSION_STICKY false / true Wicket session sharing (see below)
WPS_STORAGE_DIR — shared WPS directory (declared by the plugin)

Administration UI sessions

Without any mechanism, behind a round-robin Service, the UI logs out on every request (measured: 0/10 authenticated requests). Two validated solutions:

Solution Result When
Session affinity at the Ingress (GSROUTE cookie, ingress-nginx; Traefik: traefik.ingress.kubernetes.io/service.sticky.cookie) 10/10 Default: no code, sessions not replicated, standard
HZ_SESSION_SHARING=true (Wicket sessions replicated by Hazelcast, HzSessionShareFilter filter injected into web.xml by the bootstrap) 10/10 under true round-robin Without a sticky-capable Ingress, or to survive the loss of the node holding the session

Shared WPS

Execution status in the database (wps-jdbc, sub-project 2) + results on the RWX PVC mounted by all pods (/opt/geoserver_wps, declared in the WPS configuration by the geoserver-init plugin): an execution submitted on one node can be tracked and read from any other one.

Node identification

GEOSERVER_NODE_OPTS=id:${HOSTNAME} (production guide "make cluster nodes identifiable"): the pod name appears in the UI header.

Steps

cd poc/02-geoserver-image && IMAGE=<registry>/meteo-gis/geoserver:3.0.1-poc3 ./build.sh --push
cd ../03-clustering && IMAGE=<registry>/meteo-gis/geoserver:3.0.1-poc3 ./deploy.sh
kubectl -n meteo-gis scale deployment/geoserver --replicas=5      # or k9s, Rancher…

deploy.sh applies: headless Service, WPS PVC, PodDisruptionBudget, Ingress, geoserver-cluster-env ConfigMap, Deployment with 3 replicas (preferred anti-affinity, port 5701, WPS volume, terminationGracePeriodSeconds: 60).

Verification (tests in poc/03-clustering/tests/)

Cluster formation at deployment time (DNS discovery, size 1 → 2 → 3):

Hazelcast members

01 Propagation: workspace created through REST on pod A, visible on B within 3 s, deleted on B, absent on A:

Propagation

02 Scale 3 → 5 → 2 → 3 with continuous GetMap requests: 0 errors out of 95:

Scale

03 Cold boot 0 → 3 simultaneously: 3 pods ready in ~55 s, a single cluster:

Cold boot

04 Discovery failure (wrong DNS): explicit stop within 3 s:

Discovery failure

05 Sessions: Ingress affinity 10/10; round-robin without sharing 0/10; with HZ_SESSION_SHARING=true 10/10:

Sessions

Sessions with Hazelcast sharing

06 Cross-node WPS: submitted on A, ProcessSucceeded and the result (939 bytes) read on B, files on the shared volume:

Cross-node WPS

07 Admin password rotation: effective on the other node, old password rejected:

Admin rotation

Node identifier in the UI:

Node identifier

Deviations and pitfalls observed

  • Very first deployment on an empty configuration database: all replicas run the first-boot initialization (schema creation + import) concurrently; one wins, the others can exceed the startup probe and get restarted once before converging. Harmless but noisy — for the initial deployment, start with replicas: 1 and scale up once the first pod is Ready (observed on a fresh environment; every later cold boot is fast since the database is already initialized).

    A concurrent first boot can also leave duplicate catalog entries (e.g. the same coverage/layer created twice by two racing initializers). Symptom: the layer works through the workspace virtual service (/geoserver/<ws>/wms) but the global endpoint (/geoserver/ows, used by Layer Preview) returns LayerNotDefined, and GetCapabilities lists the layer twice. Cleanup: DELETE /rest/workspaces/<ws>/coveragestores/<store>?recurse=true for the affected stores, then restart one pod — its initializer recreates them once, cleanly. Restart any remaining pod that still serves a stale reader (Failed to create reader in the WMS exception).

  • cluster/ must be in JDBCStore's ignoreDirs (otherwise gsEventCluster / reload defaults apply).

  • Hazelcast on JDK 21: add the recommended --add-opens/--add-modules java.se (done in EXTRA_JAVA_OPTS).
  • HzSessionShareListener does not exist in 3.0.1: only the filter needs to be declared (initialization is handled by a Spring bean).
  • The Failed to publish resource notification, cluster not initialized (yet) warnings at startup are normal (before the cluster forms).
  • Tests: kubectl port-forward svc/… pins one pod (it does not exercise round-robin); kubectl run --rm -i sometimes loses the output — the tests go through kubectl exec in a GeoServer pod.