fix promote to master in streaming replication (#370)

* fix promote to master in streaming replication

* remove tests in example compose

* Fix grammar error

README.md

@@ -597,7 +597,7 @@ The two main replication methods allowed are
 
 
 ### Database permissions and password authentication
-Replication  uses a dedicated user `REPLICATION_USER`. The role ${REPLICATION_USER}
+Replication  uses a dedicated user `REPLICATION_USER`. The role `${REPLICATION_USER}`
 uses the default group role `pg_read_all_data`. You can read more about this
 from the [PostgreSQL documentation](https://www.postgresql.org/docs/14/predefined-roles.html)
 
@@ -616,7 +616,7 @@ layer are saved, they are automatically propagated to the replicant. Note also t
 replicant is read-only.
 
 ```shell
-docker run --name "streaming-replication" -e REPLICATION=true -e WAL_LEVEL='replica' -d -p 25432:5432  kartoza/postgis:13.0
+docker run --name "streaming-replication" -e REPLICATION=true -e WAL_LEVEL='replica' -d -p 25432:5432  kartoza/postgis:14.3.2
 ```
 
 **Note** If you do not pass the env variable `REPLICATION_PASS` a random password
@@ -635,20 +635,20 @@ we can't write new data to it. The whole database cluster will be replicated.
 
 #### Database permissions
 
-Since we are using a role ${REPLICATION_USER}, we need to ensure that it has access to all
+Since we are using a role `${REPLICATION_USER}`, we need to ensure that it has access to all
 the tables in a particular schema. So if a user adds another schema called `data`
 to the database `gis` he also has to update the permission for the user
-with the following SQL assuming the ${REPLICATION_USER} is called replicator
+with the following SQL assuming the `${REPLICATION_USER}` is called replicator
 
 ```sql
 ALTER DEFAULT PRIVILEGES IN SCHEMA data GRANT SELECT ON TABLES TO replicator;
 ```
 
-**NB** You need to set up a strong password for replication otherwise the
+**Note** You need to set up a strong password for replication otherwise the
-default password for ${REPLICATION_USER} will default to `replicator`
+default password for `${REPLICATION_USER}` will default to random generated string
 
-To experiment with the replication abilities, you can see a [docker-compose.yml](sample/replication/docker-compose.yml)
+To experiment with the streaming replication abilities, you can see a [docker-compose.yml](replication_examples/replication/docker-compose.yml).
-sample. There are several environment variables that you can set, such as:
+There are several environment variables that you can set, such as:
 
 Master settings:
 - **ALLOW_IP_RANGE**: A `pg_hba.conf` domain format which will allow specified host(s)
@@ -676,7 +676,7 @@ Slave settings:
 - **REPLICATION_USER** User to initiate streaming replication
 - **REPLICATION_PASS** Password for a user with streaming replication role
 
-To run the sample replication, follow these instructions:
+To run the example streaming_replication, follow these instructions:
 
 Do a manual image build by executing the `build.sh` script
 
@@ -684,7 +684,7 @@ Do a manual image build by executing the `build.sh` script
 ./build.sh
 ```
 
-Go into the `sample/replication` directory and experiment with the following Make
+Go into the `replication_examples/streaming_replication` directory and experiment with the following Make
 command to run both master and slave services.
 
 ```shell
@@ -701,7 +701,7 @@ To view logs for master and slave respectively, use the following command:
 
 ```shell
 make master-log
-make slave-log
+make node-log
 ```
 
 You can try experiment with several scenarios to see how replication works
@@ -709,15 +709,15 @@ You can try experiment with several scenarios to see how replication works
 #### Sync changes from master to replicant
 
 You can use any postgres database tools to create new tables in master, by
-connecting using POSTGRES_USER and POSTGRES_PASS credentials using exposed port.
+connecting using `POSTGRES_USER` and `POSTGRES_PASS` credentials using exposed port.
-In the sample, the master database was exposed on port 7777.
+In the streaming_replication example, the master database was exposed on port 7777.
 Or you can do it via command line, by entering the shell:
 
 ```shell
 make master-shell
 ```
 
-Then made any database changes using psql.
+Then make any database changes using psql.
 
 After that, you can see that the replicant follows the changes by inspecting the
 slave database. You can, again, use database management tools using connection
@@ -725,7 +725,7 @@ credentials, hostname, and ports for replicant. Or you can do it via command lin
 by entering the shell:
 
 ```shell
-make slave-shell
+make node-shell
 ```
 
 Then view your changes using psql.
@@ -738,14 +738,14 @@ into slave environment and set `DESTROY_DATABASE_ON_RESTART: 'False'`.
 
 After this, you can make changes to your replicant, but master and replicant will not
 be in sync anymore. This is useful if the replicant needs to take over a failover master.
-However it is recommended to take additional action, such as creating a backup from the
+However, it is recommended to take additional action, such as creating a backup from the
 slave so a dedicated master can be created again.
 
 #### Preventing replicant database destroy on restart
 
 You can optionally set `DESTROY_DATABASE_ON_RESTART: 'False'` after successful sync
 to prevent the database from being destroyed on restart. With this setting you can
-shut down your replicant and restart it later and it will continue to sync using the existing
+shut down your replicant and restart it later, and it will continue to sync using the existing
 database (as long as there are no consistencies conflicts).
 
 However, you should note that this option doesn't mean anything if you didn't
@@ -762,7 +762,7 @@ To activate the following you need to use the environment variable
 docker run --name "logical-replication" -e WAL_LEVEL=logical -d  kartoza/postgis:13.0
 ```
 
-For a detailed example see the docker-compose in the folder `sample/logical_replication`.
+For a detailed example see the docker-compose in the folder `replication_examples/logical_replication`.
 
 ### Docker image versions
 
@@ -786,4 +786,4 @@ please consider taking out a [Support Level Agreeement](https://kartoza.com/en/s
 - Rizky Maulana (rizky@kartoza.com)
 - Admire Nyakudya (admire@kartoza.com)
 
-March 2021
+April 2022

docker-compose.yml

@@ -1,5 +1,5 @@
 # docker-compose build
-version: '2.1'
+version: '3.9'
 volumes:
   dbbackups:
   postgis-data:
@@ -7,7 +7,7 @@ volumes:
 services:
 
   db:
-    image: kartoza/postgis:14-3.1
+    image: kartoza/postgis:14-3.2
     volumes:
       - postgis-data:/var/lib/postgresql
       - dbbackups:/backups
@@ -20,13 +20,13 @@ services:
       # Add extensions you need to be enabled by default in the DB. Default are the five specified below
       - POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting
     ports:
-      - 25432:5432
+      - "25432:5432"
     restart: on-failure
     healthcheck:
       test: "exit 0"
 
   dbbackups:
-    image: kartoza/pg-backup:14-3.1
+    image: kartoza/pg-backup:14-3.2
     hostname: pg-backups
     volumes:
       - dbbackups:/backups
@@ -36,7 +36,6 @@ services:
       - POSTGRES_PASS=docker
       - POSTGRES_PORT=5432
       - POSTGRES_HOST=db
-      - POSTGRES_DBNAME=gis
     restart: on-failure
     depends_on:
       db:

replication_examples/streaming_replication/Makefile

@@ -0,0 +1,27 @@
+up:
+	docker-compose up -d
+
+down:
+	docker-compose down
+
+scale:
+	docker-compose up -d --scale pg-node=3
+
+unscale:
+	docker-compose up -d --scale pg-node=1
+
+status:
+	docker-compose ps
+
+master-shell:
+	docker-compose exec pg-master /bin/bash
+
+node-shell:
+	docker-compose exec pg-node /bin/bash
+
+master-log:
+	docker-compose logs -f --tail=30 pg-master
+
+node-log:
+	docker-compose logs -f --tail=30 pg-node
+

replication_examples/streaming_replication/docker-compose.yml

@@ -1,20 +1,20 @@
 
-version: '2.1'
+version: '3.9'
 
 volumes:
   pg-master-data-dir:
-  pg-slave-data-dir:
+  pg-node-data-dir:
 
 
 services:
   pg-master:
-    image: kartoza/postgis:14-3.1
+    image: kartoza/postgis:14-3.2
     restart: 'always'
     # You can optionally mount to volume, to play with the persistence and
     # observe how the slave will behave after restarts.
     volumes:
       - pg-master-data-dir:/var/lib/postgresql
-      - ./tests:/tests
+      - ./scripts/setup-master.sql:/docker-entrypoint-initdb.d/setup-master.sql
     environment:
       # ALLOW_IP_RANGE option is used to specify additionals allowed domains
       # in pg_hba.
@@ -34,16 +34,15 @@ services:
     healthcheck:
       test: "exit 0"
 
-  pg-slave:
+  pg-node:
-    image: kartoza/postgis:14-3.1
+    image: kartoza/postgis:14-3.2
     restart: 'always'
     # You can optionally mount to volume, but we're not able to scale it
     # in that case.
     # The slave will always destroy its database and copy from master at
     # runtime
     volumes:
-      - pg-slave-data-dir:/var/lib/postgresql
+      - pg-node-data-dir:/var/lib/postgresql
-      - ./tests:/tests
 
     environment:
       # ALLOW_IP_RANGE option is used to specify additionals allowed domains
@@ -77,7 +76,7 @@ services:
       # If specified with any value, then it will convert current slave into
       # a writable state. Useful if master is down and the current slave needs
       # to be promoted until manual recovery.
-#      PROMOTE_MASTER: 'True'
+      #PROMOTE_MASTER: 'True'
 
       # For now we don't support different credentials for replication
       # so we use the same credentials as master's superuser, or anything that

replication_examples/streaming_replication/scripts/setup-master.sql

@@ -0,0 +1,11 @@
+-- Create a table
+CREATE TABLE IF NOT EXISTS sweets
+    (
+        id SERIAL,
+        name TEXT,
+        price DECIMAL,
+        CONSTRAINT sweets_pkey PRIMARY KEY (id)
+    );
+
+-- Inserts records into the table
+INSERT INTO sweets (name, price) VALUES ('strawberry', 4.50), ('Coffee', 6.20), ('lollipop', 3.80);

sample/replication/Makefile

@@ -1,50 +0,0 @@
-up:
-	docker-compose up -d
-
-down:
-	docker-compose down
-
-scale:
-	docker-compose up -d --scale pg-slave=3
-
-unscale:
-	docker-compose up -d --scale pg-slave=1
-
-status:
-	docker-compose ps
-
-master-shell:
-	docker-compose exec pg-master /bin/bash
-
-slave-shell:
-	docker-compose exec pg-slave /bin/bash
-
-master-log:
-	docker-compose logs -f --tail=30 pg-master
-
-slave-log:
-	docker-compose logs -f --tail=30 pg-slave
-
-
-# These commands were used for travis
-
-master-log-tail	:
-	docker-compose logs --tail=30 pg-master
-
-slave-log-tail:
-	docker-compose logs --tail=30 pg-slave
-
-check-master-running:
-	@echo "Check master is running"
-	@docker-compose logs --tail=1 pg-master | grep 'database system is ready to accept connections' &> /dev/null
-
-check-master-replication:
-	@docker-compose exec pg-master /bin/sh -c "su - postgres -c \"/tests/replication_test_master.sh\""
-
-check-slave-running:
-	@echo "Check slave is running"
-	@docker-compose logs pg-slave | grep 'database system is ready to accept read only connections' &> /dev/null
-	@docker-compose logs pg-slave | grep 'started streaming WAL from primary' &> /dev/null
-
-check-slave-replication:
-	@docker-compose exec pg-slave /bin/sh -c "su - postgres -c \"/tests/replication_test_slave.sh\""

sample/replication/tests/replication_test_master.sh

@@ -1,19 +0,0 @@
-#!/usr/bin/env bash
-
-. /scripts/env-data.sh
-
-set -e
-
-echo "Check master replication"
-
-# Create a new table
-
-echo "Create new table"
-psql -d ${POSTGRES_DBNAME} -c "CREATE TABLE test_replication_table ();"
-
-# Check table exists in master
-
-echo "Check table exists"
-psql -d ${POSTGRES_DBNAME} -c "\dt" | grep test_replication_table
-
-exit $?

sample/replication/tests/replication_test_slave.sh

@@ -1,14 +0,0 @@
-#!/usr/bin/env bash
-
-. /scripts/env-data.sh
-
-set -e
-
-echo "Check slave replication"
-
-# Check table exists in slave
-
-echo "Check table exists"
-psql -d ${POSTGRES_DBNAME} -c "\dt" | grep test_replication_table
-
-exit $?

scripts/env-data.sh

@@ -17,7 +17,7 @@ SQLDIR="/usr/share/postgresql/${POSTGRES_MAJOR_VERSION}/contrib/postgis-${POSTGI
 SETVARS="POSTGIS_ENABLE_OUTDB_RASTERS=1 POSTGIS_GDAL_ENABLED_DRIVERS=ENABLE_ALL"
 LOCALONLY="-c listen_addresses='127.0.0.1'"
 PG_BASEBACKUP="/usr/bin/pg_basebackup"
-PROMOTE_FILE="/tmp/pg_promote_master"
+NODE_PROMOTION="/usr/lib/postgresql/${POSTGRES_MAJOR_VERSION}/bin/pg_ctl"
 PGSTAT_TMP="/var/run/postgresql/"
 PG_PID="/var/run/postgresql/${POSTGRES_MAJOR_VERSION}-main.pid"
 

scripts/setup-replication.sh

@@ -35,7 +35,7 @@ if [[ "$WAL_LEVEL" == 'replica' && "${REPLICATION}" =~ [Tt][Rr][Uu][Ee] ]]; then
  fi
  # Promote to master if desired
 if [[ ! -z "${PROMOTE_MASTER}" ]]; then
-	touch ${PROMOTE_FILE}
+  su - postgres -c "${NODE_PROMOTION} promote -D ${DATADIR}"
 fi
 
 fi