Rework docker compose for swarm support

.gitignore

@@ -8,9 +8,11 @@
 # ignore the .env file so we can specify environment variables that are
 # not pushed to github
 .env
-.envs-*
+envs/.envs-*
+secrets/slapd_password.txt
 # ignore the certs files and Caddy data directory
 resources/ssl/*
 resources/caddy/data/*
 # ignore possible caddy binary downloaded
 /caddy
+volumes/

README.md

@@ -1,7 +1,18 @@
 # geOrchestra on Docker
 
+## Preconisation
+
 This repository is meant to offer a convenient way to start geOrchestra for **development** or **demo** purposes. Production use is not recommended without hardening measures.
 
+If you want to use this for production you might need to:
+1. modify the way to use certificates
+2. remove databases (database and postgis deployment and related volumes) from docker-compose
+3. update [.envs](envs/) files
+4. remove unwanted open ports
+5. modify volumes management (don't let docker service do it..), you might want to store them in specific path
+6. refit resources allocation with your use (Xmx Xms)
+
+
 ## Quick Start
 
 **1. Prerequisite**
@@ -25,14 +36,16 @@ git clone --recurse-submodules https://github.com/georchestra/docker.git
 
 Choose which branch to run, eg for latest stable:
 ```
-git checkout 24.0 && git submodule update
+git checkout 25.0 && git submodule update
 ```
 
 **3. Run**
 
+**3.1 Docker compose**
+
 The default docker-compose file contains all geOrchestra modules.
 
-It's recommended to double-check the `docker-compose.yml` and `docker-compose.override.yml` files if you need to comment useless modules (e.g extractor, mapstore,... ).
+It's recommended to double-check the `docker-compose.yml` file if you need to comment modules you don't need (e.g ogc-api-records, mapstore,... ).
 
 You need to use the new Compose plugin V2, `docker-compose` (V1) is not supported by default: [https://docs.docker.com/compose/install/linux/](https://docs.docker.com/compose/install/linux/).   
 If you still want to use the old `docker-compose` (V1), you need to remove all the parameters `depends_on` from the files `docker-compose.yml` and `docker-compose.override.yml`.
@@ -50,6 +63,50 @@ To stop geOrchestra:
 docker compose down
 ```
 
+**3.2 Docker swarm**
+
+[docker-compose.swarm.yml](docker-compose.swarm.yml) contains spécific services needed for deploying it in swarm
+
+On the first deploy using swarm, some services may fail (and restart), it is normal because there is no possible dependancies, in the case you should wait longer and check when mandatory service are up (eg. ldap, database), everything should get running and up.
+
+In order to run you will need to run those few commands:
+
+To initialize your cluster
+```
+docker swarm init
+```
+To deploy/redeploy (after modification of the docker-compose) georchestra:
+```
+docker stack deploy -c docker-compose.yml -c docker-compose.swarm.yml georchestra
+```
+verify the stack is present
+```
+docker stack ls
+```
+Verify that services are running
+```
+docker stack services georchestra
+```
+To access the log of the gateway for instance you can use:
+```
+docker service logs georchestra_gateway
+```
+To restart a service :
+```
+docker service update --force georchestra_gateway
+```
+To stop/delete the deployment:
+```
+docker stack rm georchestra
+```
+If you need to clean all volumes from you test you ca use it command:
+```
+STACK_NAME=georchestra
+for vol in `docker volume ls --filter "Name=${STACK_NAME}_.*" --format json | jq -r '.Name'`; do docker volume rm $vol ; done
+```
+
+
+
 **4. Play**
 
 Open [https://georchestra-127-0-0-1.nip.io/](https://georchestra-127-0-0-1.nip.io/) in your browser. Then:
@@ -61,17 +118,11 @@ To login, use these credentials:
  * `testuser` / `testuser`
  * `testadmin` / `testadmin`
 
-To upload data into the GeoServer data volume (`geoserver_geodata`), use `rsync`:
-```
-rsync -arv -e 'ssh -p 2222' /path/to/geodata/ geoserver@georchestra-127-0-0-1.nip.io:/mnt/geoserver_geodata/
-```
-(password is: `geoserver`)
+To upload data into the GeoServer data volume (`geoserver_geodata`), use `docker volume inspect docker_geoserver_geodata `  to get the path where it is stored and use cp to copy where you want
 
 Files uploaded into this volume will also be available to the geoserver instance in `/mnt/geoserver_geodata/`.
 
-Emails sent by the SDI (eg when users request a new password) will not be relayed on the internet but trapped by a local SMTP service.  
-These emails can be read on https://georchestra-127-0-0-1.nip.io/webmail/ (with login `smtp` and password `smtp`).
-
+Emails sent by the SDI (eg when users request a new password) will relayed by the service "smtp" on the internet, you can configure it in the file [.envs-smtprelay](envs/.envs-smtprelay).
 
 ## Locally trust the TLS certificate for geOrchestra
 
@@ -117,17 +168,12 @@ These docker-compose files describe:
  * how they are linked together,
  * where the configuration and data volumes are
 
-The `docker-compose.override.yml` file adds services to interact with your geOrchestra instance (they are not part of geOrchestra "core"):
- * reverse proxy / load balancer
- * ssh / rsync services,
- * smtp, webmail.
-
 **Feel free to comment out the apps you do not need**.
 
 The base docker composition does not include any standalone geowebcache instance, nor the atlas module.
 If you need them, you have to include the corresponding complementary docker-compose file at run-time:
 ```
-docker compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.gwc.yml -f docker-compose.atlas.yml up
+docker compose -f docker-compose.yml up
 ```
 
 ## Upgrading
@@ -144,11 +190,11 @@ To upgrade, we recommend you to:
 This docker composition supports environment variables, if you need to customize something it might be in the different environment variables files.
 
 Here is the list of these files:
-- [.envs-common](.envs-common) 
-- [.envs-database-datafeeder](.envs-database-datafeeder)
-- [.envs-database-georchestra](.envs-database-georchestra)
-- [.envs-hosts](.envs-hosts)
-- [.envs-ldap](.envs-ldap)
+- [.envs-common](envs/.envs-common) 
+- [.envs-database-datafeeder](envs/.envs-database-datafeeder)
+- [.envs-database-georchestra](envs/.envs-database-georchestra)
+- [.envs-hosts](envs/.envs-hosts)
+- [.envs-ldap](envs/.envs-ldap)
 
 If you add variables, be careful because it might be added into the wrong/unwanted container.
 
@@ -167,11 +213,21 @@ Most changes will require a service restart, except maybe updating viewer contex
 
 ### Kibana
 
+
+The optional `kibana` service is used for dashboarding purposes and is integrated to the GeoNetwork admin UI. See in the `Statistics & status / Content statistics` admin menu to access it.
+
+A specific configuration is provided in the `kibana/` subdirectory.
+
+Please note that it will require to load by hand the following file from the kibana admin ui:
+
+https://raw.githubusercontent.com/georchestra/geonetwork/georchestra-gn4-4.0.6/es/es-dashboards/data/export.ndjson#
+
+
 In order to have Kibana up and running, you will need to:
 1. After Elasticsearch up and healthy, launch the command `docker compose exec -it elasticsearch bin/elasticsearch-reset-password -u kibana_system`. It will ask to fill a password for the `kibana_system` user.
-2. Uncomment and fill this password into the `.envs-elastic` file.
+2. Uncomment and fill this password into the `envs/.envs-elastic` file.
 3. Enable kibana server with `scale: 1` in `docker-compose.yml`.
-4. Start Kibana with `docker compose up -d kibana`.
+4. Start Kibana with `docker compose up -d kibana` or ``.
 
 ## Building
 
@@ -206,17 +262,6 @@ to
 geofenceEntityManagerFactory.jpaPropertyMap[hibernate.hbm2ddl.auto]=update
 ```
 
-## Kibana
-
-The optional `kibana` service is used for dashboarding purposes and is integrated to the GeoNetwork admin UI. See in the `Statistics & status / Content statistics` admin menu to access it.
-
-A specific configuration is provided in the `kibana/` subdirectory.
-
-Please note that it will require to load by hand the following file from the kibana admin ui:
-
-https://raw.githubusercontent.com/georchestra/geonetwork/georchestra-gn4-4.0.6/es/es-dashboards/data/export.ndjson#
-
-
 
 ## Elasticsearch
 
@@ -234,11 +279,11 @@ https://techoverflow.net/2019/04/17/how-to-disable-elasticsearch-disk-quota-wate
 
 Beside georchestra/docker directory, you need to clone [georchestra/georchestra repo](https://github.com/georchestra/georchestra) first.
 
-Next, install maven to execute [main georchestra Makefile](https://github.com/georchestra/georchestra/blob/master/Makefile) on each modification (e.g console, security-proxy, whatever you change).
+Next, install maven to execute [main georchestra Makefile](https://github.com/georchestra/georchestra/blob/master/Makefile) on each modification (e.g console, gateway, whatever you change).
 
-For example, if you change some security-proxy code, use :
+For example, if you change some gateway code, use :
 
-`make docker-build-proxy`
+`make docker-build-gateway`
 
 ... to execute easily this maven command :
 
@@ -254,25 +299,30 @@ You can now test modifications locally with the current FQDN (by default `georch
 
 **3. Debug**
 
-Open `docker/docker-compose.yml` and identify `proxy` section.
+Open `docker/docker-compose.yml` and identify `gateway` section.
 
-Change `proxy` section to insert some JAVA options and ports `5005` to get :
+Change `gateway` section to insert some JAVA options and ports `5005` to get :
 
 ```
-  proxy:
-    image: georchestra/security-proxy:latest
+  gateway:
+    image: georchestra/gateway:latest-debug
     depends_on:
-      - ldap
-      - database
+    - database
     volumes:
-      - georchestra_datadir:/etc/georchestra
+    - ./config:/etc/georchestra
     environment:
-      - JAVA_OPTIONS=-Dorg.eclipse.jetty.annotations.AnnotationParser.LEVEL=OFF -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005
-      - XMS=256M
-      - XMX=1G
+    - JAVA_OPTIONS=-Dorg.eclipse.jetty.annotations.AnnotationParser.LEVEL=OFF -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005
+    - XMS=256M
+    - XMX=1G
+    env_file:
+    - ./envs/.envs-common
+    - ./envs/.envs-ldap
+    - ./envs/.envs-hosts
+    - ./envs/.envs-database-georchestra
     restart: always
     ports:
-      - "5005:5005"
+    - "5005:5005"
+    - "8080:8080"
 ```
 
 Apply Docker changes :