ImageMosaic ingestion — time series and multi-dimensional models¶

Goal: continuously publish weather rasters (models: model run × forecast validity time × vertical level; satellites: time series) as GeoServer ImageMosaic layers, COGs on S3 indexed in PostGIS, visible to all replicas as soon as they are ingested, with the quantity's unit known and displayed.

flowchart LR
    SRC[Sources<br/>models, satellites] --> ING[Ingestion service<br/>Python + GDAL]
    ING -- "COG (canonical unit)" --> S3[(rasters bucket)]
    ING -- "INSERT index + one view per mosaic" --> DB[(mosaic schema<br/>parameter · model_granule · sat_granule)]
    GS[GeoServer ×N] -- "jdbc/meteo (read)" --> DB
    GS -- "range reads" --> S3
    PLUG[geoserver-init plugin<br/>MOSAICS ConfigMap] -- "indexer/datastore.properties (Resource API)<br/>coverage + dimensions + title (unit)" --> GS
    C[WMS clients] -- "TIME · ELEVATION · DIM_REFERENCE_TIME" --> GS

Hold "Alt" / "Option" to enable pan & zoom

Decisions: ADR-0003 (accepted), ADR-0010. Artifacts: poc/05-ingestion/ (schema, service, Jobs, tests).

Prerequisites¶

Sub-projects 1 to 4 (database, image, cluster, S3). Secrets: ingestion-db (ingestion service account), s3-credentials, meteo-reader.

Data model (`mosaic` schema, owned by the ingestion service)¶

Table	Role
`parameter`	One physical quantity = one row: `param_key`, `cf_standard_name`, `canonical_unit`, label
`parameter_code`	Known codes per scheme (`grib2`, `eccodes`, `bufr`, `metar`, `cf`) with the unit in that scheme
`level_type`	Level types (GRIB2 table 4.5)
`model_granule`	One row per model COG: `the_geom`, `location` (`s3://…`), `model`, `param_key`, `reference_time` (model run), `time` (validity), `elevation`, `level_type`, `level_label`, `source_unit`, `ingested_at`
`sat_granule`	One row per satellite image: `satellite`, `instrument`, `product`, `param_key`, `time`, `source_unit`
views `arpege_t`, `meteosat12_fci_ir105`…	One view per mosaic (filter on model+quantity or satellite+instrument+product); `gt_pk_metadata` declares `fid` as the key for GeoTools

The schema is created by a Job (admin account), GeoServer's reader receives SELECT through grant-reader (sub-project 1); datastore.properties specifies schema=mosaic.

Ingestion service¶

ghcr.io/osgeo/gdal container + Python (boto3, psycopg, numpy), CLI ingest.py model | sat. For each granule:

field (synthetic at the prototype stage — actual GRIB/NetCDF decoding is out of scope);
conversion to the canonical unit of the reference table (demonstrated: source Cel → K), source_unit preserved;
writing of a COG (GDAL, DEFLATE, 256 tiles) with metadata param_key, units, time, reference_time, level;
S3 put_object following the ADR-0003 convention (models/arpege/t/arpege_t_isobaric85000_20260901T00Z_20260901T06Z.tif, sat/meteosat-12/fci/meteosat-12_fci_ir105_20260901T1200Z.tif);
INSERT … ON CONFLICT DO UPDATE into the index and CREATE OR REPLACE VIEW of the mosaic.

Run as a Job (k8s/job-ingest-demo.yaml, ingest.sh); CronJob or orchestrator in production. A quantity absent from the reference table = explicit error ("add it to mosaic.parameter").

Real-data example: GOES-19 from NOAA Open Data¶

fetch_goes.py (+ k8s/job-ingest-goes.yaml) ingests real satellite imagery through the exact same conventions: GOES-19 ABI channel 13 (infrared 10.3 µm brightness temperature, kelvin) from the public NOAA bucket https://noaa-goes19.s3.amazonaws.com — anonymous HTTPS, one full-disk granule every 10 minutes, reproducible by anyone. Each granule is converted from the geostationary NetCDF to an EPSG:4326 COG (GDAL Warp), uploaded as sat/goes-19/abi/goes-19_abi_ir103_<time>.tif and indexed with its real acquisition time and footprint. Publication is then the usual one-line MOSAICS entry (goes19_abi_ir103|…) — run the ingestion Job before declaring the mosaic. Two pitfalls: the ubuntu-small GDAL image has no netCDF driver (the ingestion image uses ubuntu-full), and the "Unhandled X/Y axis unit rad" warning is harmless (the warped bounds are correct).

Publication by the plugin (declarative)¶

geoserver-mosaics ConfigMap: MOSAICS="name|view|kind|param_key|title;…". For each entry, at each pod's startup:

mosaics/<name>/indexer.properties (Cog=true, CogRangeReader=…S3RangeReader, UseExistingSchema=true, CanBeEmpty=true, TypeName=<view>, TimeAttribute=time, ElevationAttribute=elevation, AdditionalDomainAttributes=reference_time for models) and datastore.properties (PostgisNGJNDIDataStoreFactory, jndiReferenceName=java:comp/env/jdbc/meteo, schema=mosaic) — written through the Resource API, hence stored in the database by JDBCStore and shared;
ImageMosaic store (file:mosaics/<name>), coverage and layer;
dimensions enabled: time (LIST, default MAXIMUM), elevation (Pa), custom dimension reference_time;
title = label + canonical unit read from mosaic.parameter through the JNDI resource, abstract and keywords unit:K, param:air_temperature.

The S3 credentials of the RangeReader come from the environment (IIO_S3_AWS_*, sub-project 4): no secret in the mosaic files.

No notification needed

ImageMosaic queries the database index on every request, dimension domains included: a granule inserted by the ingestion is visible in the GetCapabilities and served by all pods immediately, without any restart or message (test 03). File-based mosaics would have required a per-node harvest — that is the reason behind the "index in the database" choice.

Verification (tests in `poc/05-ingestion/tests/`)¶

Schema and reference table, demonstration ingestion (6 model granules, 3 satellite ones):

01 Capabilities: titles carrying the unit, dimensions and their values:

02 GetMap by TIME × ELEVATION: different images (md5), defaults when no dimension is given:

03 New forecast validity time ingested → 4 time values on both pods, GetMap served, pods unchanged:

04 Satellite mosaic per instant:

05 Pod replaced: mosaics still published:

Deviations and pitfalls observed¶

The reader role now serves several schemas: grant-reader.sql no longer redefines an existing search_path; the stores specify schema=.
Views as the index: virtual key fid declared in gt_pk_metadata.
The mosaic configuration files go through the Resource API (JDBCStore): shared between pods, regenerated at every startup.