Replication changes (#153)

* Fix typo in helper script

* Port work in develop to 10 branch (#90)

* Port 9.6 to develop (#89)

* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly

* Update README.md

* Backport from branch: 9.6-2.4

Fix default datadir

- Change into default datadir
- Add small unittest

* Optimise PostgreSQL performance and align with the changes done in 9.6
version

* Fix version numbers

* Minor change to correct env file

* Merged 10 branch into develop

* Remove reduntant conf file from dockerfile

* Remove reduntant conf file directive from setup script

* Remove reduntant conf file directive from  script

* Migrate settings to be compatible with PostgreSQL version 11

* fix ssl setup

* Align changes to postgresql version 11

* Set destroy database to false

* Commented out code for destroy database

* Fix version number for build

* Merge develop to master to align with PostgreSQL 11 (#117)

* Added note about version number

* README update

* Revert to using pg 9.3 in latest build - prevent breaking downstream apps for now

* Revert to pg 9.3

* Fix references to 9.4 to make them 9.3 rather

* Fix incorrect version in 9.4 branch

* Fix TOPOLOGY conditional typo

* Update run-postgis-docker.sh

* Update run-postgis-docker.sh

added IPADDRESS

* Updates from Marco

* Allow connections from 192.168 private network too

* start temporary server in local-only mode, poll instead of blind wait, preserve pid 1

* Enable PostGIS Out DB raster support

http://postgis.net/2015/05/02/tip_enable_raster_drivers/
http://postgis.net/docs/postgis_installation.html#install_short_version

* Remove hardcoded reference to container name "docker"

* upgrade postgres to 9.5 and postgis to 2.2

* Added flexible IP range as arg on docker run

* Added better description to docker env vars

* Updated references to pg and postgis versions in readme

* Followup 0745c488, fix references to 9.4

* problem version

postgresql version is 9.4, instead of 9.5

* Added note about allowing external ports.

* Update README.md

* Cleanups to 9.5 branch - remove old supervisor files

* Fix for README

* 9.5 2.2 (#49)

* Allow connections from 192.168 private network too

* Updated references to pg and postgis versions in readme

* Cleanups to 9.5 branch - remove old supervisor files

* Fix for README

* Dont add supervisor stuff which is deprecated

* Fix 9.4 reference

* Allow changing the default database name (#50)

* Allow changing the default database name (#50)

* Allow changing the default database name (#50) (#52)

* Commented out legacy scripts since it seems they are removed in 2.2

* 9.5 2.2 (#58)

* Allow changing the default database name (#50)

* Commented out legacy scripts since it seems they are removed in 2.2

* 9.5 2.2 (#59)

* Allow changing the default database name (#50)

* Commented out legacy scripts since it seems they are removed in 2.2

* Improve start (#57)

* Make initial dir optional

* Improve termination of background initial process #56

* Merge improvements from 9.5 branch (#60)

* Allow changing the default database name (#50)

* Commented out legacy scripts since it seems they are removed in 2.2

* 9.5 2.2 (#59)

* Allow changing the default database name (#50)

* Commented out legacy scripts since it seems they are removed in 2.2

* Improve start (#57)

* Make initial dir optional

* Improve termination of background initial process #56

* Added more options to convenience run script

* Added missing l from getopts

* Added missing l from getopts

* 9.5 2.2 (#61)

* Allow changing the default database name (#50)

* Allow changing the default database name (#50)

* Allow changing the default database name (#50) (#52)

* Commented out legacy scripts since it seems they are removed in 2.2

* Added more options to convenience run script

* Added missing l from getopts

* Added missing l from getopts

* Tweak convenience scripts (#62)


* Added missing l from getopts

* allow connection when using docker compose (#65)

see https://github.com/docker/compose/issues/4336
and https://github.com/kartoza/docker-postgis/issues/40

* Install gnupg for fetching keys first

* Fix key fetching

* Updated to 9.6 and postgis 2.4

* Ditch apt-cacher stuff

* Upgraded to PG 10

* change version from postgresql-10.0 to 10 and fix typo in run-postgis… (#76)

* change version from postgresql-10.0 to 10 and fix typo in run-postgis file

* edit setup.sh to version 10

* Change from version 10.0 to 10 in Dockerfile (#75)

I checked that postgresql-10.0 is an invalid name. It should be just postgresql-10

* Fixes for pg 10 to start nicely

* Port 9.6 to develop (#89)

* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly

* Update README.md

* Fix #90 and replace references to slave with replicant

(Slave is a pejorative term)

* Backport from branch: 9.6-2.4

Fix default datadir

- Change into default datadir
- Add small unittest

* Merge branch 10 into develop (#113)

* Fix typo in helper script

* Port work in develop to 10 branch (#90)

* Port 9.6 to develop (#89)

* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly

* Update README.md

* Backport from branch: 9.6-2.4

Fix default datadir

- Change into default datadir
- Add small unittest

* Optimise PostgreSQL performance and align with the changes done in 9.6
version

* Fix version numbers

* Minor change to correct env file

* Merged 10 branch into develop

* Remove reduntant conf file from dockerfile

* Remove reduntant conf file directive from setup script

* Remove reduntant conf file directive from  script

* Port changes for Postgres version 11 (#114)

* Fix typo in helper script

* Port work in develop to 10 branch (#90)

* Port 9.6 to develop (#89)

* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly

* Update README.md

* Backport from branch: 9.6-2.4

Fix default datadir

- Change into default datadir
- Add small unittest

* Optimise PostgreSQL performance and align with the changes done in 9.6
version

* Fix version numbers

* Minor change to correct env file

* Merged 10 branch into develop

* Remove reduntant conf file from dockerfile

* Remove reduntant conf file directive from setup script

* Remove reduntant conf file directive from  script

* Migrate settings to be compatible with PostgreSQL version 11

* fix ssl setup

* Port changes to 11 branch (#115)

* Fix typo in helper script

* Port work in develop to 10 branch (#90)

* Port 9.6 to develop (#89)

* Part one of porting work from 9.6 to 10

* Backported more scripts from 9.6 branch

* Added missing apt update in dockerfile

* Updates to entrypoint to reference image and update docker-compose to reference 10 pg

* Added sample and docs from 9.6 branch

* Removed my diagram as Rizky had already added one

* Fix env paths for pg 10

* Fixes for backporting work from 9.6 to 10 - dbb now spins up and accepts connections properly

* Update README.md

* Backport from branch: 9.6-2.4

Fix default datadir

- Change into default datadir
- Add small unittest

* Optimise PostgreSQL performance and align with the changes done in 9.6
version

* Fix version numbers

* Minor change to correct env file

* Merged 10 branch into develop

* Remove reduntant conf file from dockerfile

* Remove reduntant conf file directive from setup script

* Remove reduntant conf file directive from  script

* Migrate settings to be compatible with PostgreSQL version 11

* fix ssl setup

* Align changes to postgresql version 11

* Set destroy database to false

* Commented out code for destroy database

* Fix version number for build

* Implement conf lock file check (#116)

It will make sure that the conf file will only be generated once
for a given container.

* Added new configuration in recovery.conf and postgres optimisations for master-slave replication

* Added new configuration in recovery.conf and postgres optimisations for master-slave replication (#118)

* Remove template logic and use plain create extension (#119)

* Remove template logic and added option to create multiple databases and extensions

* Fix travis error

* More optimisations for replication

* Fix logic for checking if database exists since we now can create
multiple databases.

* Added Licence file

* add option to mount certificates

* fix comments in PR about mounting SSL

* Align to develop upstream

* Add OGR FDW to the installation and activate it in the docker-compose

* Fix spacing in readme

* Add replication user and enable postgis rasters drivers

* Replicate from restricted IP address

* use replication user for streaming changes

* update README to higlight new changes

* change permisions of default schema to enable replication

* add missing env variables

* Add logic to cater for destroy database on restart. It can be True or False on start and the logic will handle it

* Fix issues due to feedback
pull/156/head
mazano 2019-10-22 11:31:59 +02:00 zatwierdzone przez GitHub
rodzic f26e638d21
commit ee88848870
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
8 zmienionych plików z 64 dodań i 41 usunięć

Wyświetl plik

@ -9,7 +9,7 @@ RUN export DEBIAN_FRONTEND=noninteractive
ENV DEBIAN_FRONTEND noninteractive
RUN dpkg-divert --local --rename --add /sbin/initctl
RUN apt-get -y update; apt-get -y install gnupg2 wget ca-certificates rpl pwgen
RUN apt-get -y update; apt-get -y install gnupg2 wget ca-certificates rpl pwgen gdal-bin
RUN sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ ${IMAGE_VERSION}-pgdg main" > /etc/apt/sources.list.d/postgresql.list'
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc -O- | apt-key add -
@ -31,6 +31,13 @@ ADD setup.sh /setup.sh
RUN chmod +x /setup.sh
RUN /setup.sh
ADD locale.gen /etc/locale.gen
RUN /usr/sbin/locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN update-locale ${LANG}
# We will run any commands in this when the container starts
ADD docker-entrypoint.sh /docker-entrypoint.sh
ADD setup-conf.sh /
@ -41,11 +48,4 @@ ADD setup-ssl.sh /
ADD setup-user.sh /
RUN chmod +x /docker-entrypoint.sh
ADD locale.gen /etc/locale.gen
RUN /usr/sbin/locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN update-locale ${LANG}
ENTRYPOINT /docker-entrypoint.sh

Wyświetl plik

@ -16,6 +16,8 @@ differentiates itself by:
* replication support included
* Ability to create multiple database when you spin the database.
* Enable multiple extensions in the database when setting it up
* Gdal drivers automatically registered for pg raster
* Support for out-of-db rasters
We will work to add more security features to this container in the future with
the aim of making a PostGIS image that is ready to be used in a production
@ -218,6 +220,14 @@ mirror database content from a designated master. This replication scheme allows
us to sync databases. However a `replicant` is only for read-only transaction, thus
we can't write new data to it. The whole database cluster will be replicated.
### Database permissions
Since we are using a role ${REPLICATION_USER}, we need to ensure that it has access to all
the tables in a particular schema. So if a user adds another schema called `data`
to the database `gis` he also has to update the permission for the user
with the following SQL assuming the ${REPLICATION_USER} is called replicator
ALTER DEFAULT PRIVILEGES IN SCHEMA data GRANT SELECT ON TABLES TO replicator;
To experiment with the replication abilities, you can see a (docker-compose.yml)[sample/replication/docker-compose.yml]
sample. There are several environment variables that you can set, such as:
@ -225,8 +235,8 @@ Master settings:
- **ALLOW_IP_RANGE**: A pg_hba.conf domain format which will allow specified host(s)
to connect into the container. This is needed to allow the `slave` to connect
into `master`, so specifically this settings should allow `slave` address. It is also needed to allow clients on other hosts to connect to either the slave or the master.
- Both POSTGRES_USER and POSTGRES_PASS will be used as credentials for the slave to
connect, so make sure you change this into something secure.
- **REPLICATION_USER** Username to initiate streaming replication
- **REPLICATION_PASS** Password for a user with streaming replication role
Slave settings:
- **REPLICATE_FROM**: This should be the domain name or IP address of the `master`
@ -244,6 +254,8 @@ Slave settings:
as `master` for a while. However, the promoted replicant will break consistencies and
is not able to revert to replicant anymore, unless it is destroyed and resynced
with the new master.
- **REPLICATION_USER** Username to initiate streaming replication
- **REPLICATION_PASS** Password for a user with streaming replication role
To run the sample replication, follow these instructions:

Wyświetl plik

@ -2,6 +2,7 @@
DATADIR="/var/lib/postgresql/11/main"
ROOT_CONF="/etc/postgresql/11/main"
PG_ENV="$ROOT_CONF/environment"
CONF="$ROOT_CONF/postgresql.conf"
WAL_ARCHIVE="/opt/archivedir"
RECOVERY_CONF="$ROOT_CONF/recovery.conf"
@ -97,7 +98,9 @@ if [ -z "${POSTGRES_MULTIPLE_EXTENSIONS}" ]; then
POSTGRES_MULTIPLE_EXTENSIONS='postgis,hstore,postgis_topology'
fi
if [ -z "${ALLOW_IP_RANGE}" ]; then
ALLOW_IP_RANGE='0.0.0.0/0'
fi
if [ -z "${DEFAULT_ENCODING}" ]; then
DEFAULT_ENCODING="UTF8"
fi
@ -108,6 +111,13 @@ if [ -z "${DEFAULT_CTYPE}" ]; then
DEFAULT_CTYPE="en_US.UTF-8"
fi
if [ -z "${REPLICATION_USER}" ]; then
REPLICATION_USER=replicator
fi
if [ -z "${REPLICATION_PASS}" ]; then
REPLICATION_PASS=replicator
fi
# Compatibility with official postgres variable
# Official postgres variable gets priority
if [ ! -z "${POSTGRES_PASSWORD}" ]; then

Wyświetl plik

@ -22,8 +22,8 @@ services:
ALLOW_IP_RANGE: '0.0.0.0/0'
# We can specify optional credentials
POSTGRES_USER: 'docker'
POSTGRES_PASS: 'docker'
REPLICATION_USER: 'replicator'
REPLICATION_PASS: 'replicator'
# Setup master replication variables
#PG_MAX_WAL_SENDERS: 8
#PG_WAL_KEEP_SEGMENTS: 100
@ -80,8 +80,8 @@ services:
# For now we don't support different credentials for replication
# so we use the same credentials as master's superuser, or anything that
# have replication role.
POSTGRES_USER: 'docker'
POSTGRES_PASS: 'docker'
REPLICATION_USER: 'replicator'
REPLICATION_PASS: 'replicator'
depends_on:
pg-master:
condition: service_healthy

Wyświetl plik

@ -47,7 +47,10 @@ source /setup-user.sh
# Create a default db called 'gis' or $POSTGRES_DBNAME that you can use to get up and running quickly
# It will be owned by the docker db user
# Since we now pass a comma separated list in database creation we need to search for all databases as a test
touch custom.sql
cat >> custom.sql <<EOF
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO ${REPLICATION_USER}
EOF
for db in $(echo ${POSTGRES_DBNAME} | tr ',' ' '); do
RESULT=`su - postgres -c "psql -t -c \"SELECT count(1) from pg_database where datname='${db}';\""`
if [[ ${RESULT} -eq 0 ]]; then
@ -60,11 +63,13 @@ for db in $(echo ${POSTGRES_DBNAME} | tr ',' ' '); do
echo "Loading legacy sql"
su - postgres -c "psql ${db} -f ${SQLDIR}/legacy_minimal.sql" || true
su - postgres -c "psql ${db} -f ${SQLDIR}/legacy_gist.sql" || true
export PGPASSWORD=${POSTGRES_PASS}
psql ${db} -U ${POSTGRES_USER} -p 5432 -h localhost -f custom.sql
else
echo "${db} db already exists"
fi
done
rm custom.sql
# This should show up in docker logs afterwards
su - postgres -c "psql -l"

Wyświetl plik

@ -48,8 +48,8 @@ fi
if [[ -z "$REPLICATE_FROM" ]]; then
# if env not set, then assume this is master instance
# add rules to pg_hba.conf to allow replication from all
echo "Add rule to pg_hba: replication user"
echo "host replication all 0.0.0.0/0 $authMethod" >> ${ROOT_CONF}/pg_hba.conf
echo "Add rule to pg_hba: replication ${REPLICATION_USER} "
echo "host replication ${REPLICATION_USER} ${ALLOW_IP_RANGE} $authMethod" >> ${ROOT_CONF}/pg_hba.conf
fi
# Put lock file to make sure conf was not reinitialized

Wyświetl plik

@ -25,12 +25,12 @@ function configure_replication_permissions {
echo "Setup data permissions"
echo "----------------------"
chown -R postgres:postgres $(getent passwd postgres | cut -d: -f6)
su - postgres -c "echo \"${REPLICATE_FROM}:${REPLICATE_PORT}:*:${POSTGRES_USER}:${POSTGRES_PASS}\" > ~/.pgpass"
su - postgres -c "echo \"${REPLICATE_FROM}:${REPLICATE_PORT}:*:${REPLICATION_USER}:${REPLICATION_PASS}\" > ~/.pgpass"
su - postgres -c "chmod 0600 ~/.pgpass"
}
function streaming_replication {
until su - postgres -c "${PG_BASEBACKUP} -X stream -h ${REPLICATE_FROM} -p ${REPLICATE_PORT} -D ${DATADIR} -U ${POSTGRES_USER} -vP -w"
until su - postgres -c "${PG_BASEBACKUP} -X stream -h ${REPLICATE_FROM} -p ${REPLICATE_PORT} -D ${DATADIR} -U ${REPLICATION_USER} -vP -w --label=gis_pg_custer"
do
echo "Waiting for master to connect..."
sleep 1s
@ -46,33 +46,19 @@ until su - postgres -c "${PG_BASEBACKUP} -X stream -h ${REPLICATE_FROM} -p ${REP
if [[ "$DESTROY_DATABASE_ON_RESTART" =~ [Tt][Rr][Uu][Ee] ]]; then
echo "Get initial database from master"
configure_replication_permissions
streaming_replication
if [ -f "${DATADIR}/backup_label.old" ]; then
echo "PG Basebackup already exists so proceed to start the DB"
else
streaming_replication
fi
fi
#TODO We need a clever way to identify if base backup exists - Incoperate it as an else statement in destroy logic
#configure_replication_permissions
#var=`du -sh /var/lib/postgresql/11/main/pg_wal | awk '{print $1}'`
#var_size=${var:0:2}
#if [[ "${var_size} -gt 33 " ]]; then
#echo ${var_size}
#echo "Base directory exist - Please startup the database"
#else
#echo "Base directory does not exists- Create a new one"
#streaming_replication
#fi
# Setup recovery.conf, a configuration file for slave
cat > ${DATADIR}/recovery.conf <<EOF
standby_mode = on
primary_conninfo = 'host=${REPLICATE_FROM} port=${REPLICATE_PORT} user=${POSTGRES_USER} password=${POSTGRES_PASS} sslmode=${PGSSLMODE}'
primary_conninfo = 'host=${REPLICATE_FROM} port=${REPLICATE_PORT} user=${REPLICATION_USER} password=${REPLICATION_PASS} sslmode=${PGSSLMODE}'
trigger_file = '${PROMOTE_FILE}'
recovery_target_timeline='latest'
recovery_target_action='promote'

Wyświetl plik

@ -19,9 +19,19 @@ echo "postgresql user: $POSTGRES_USER" > /tmp/PGPASSWORD.txt
echo "postgresql password: $POSTGRES_PASS" >> /tmp/PGPASSWORD.txt
# Check user already exists
echo "Creating superuser $POSTGRES_USER"
RESULT=`su - postgres -c "psql postgres -t -c \"SELECT 1 FROM pg_roles WHERE rolname = '$POSTGRES_USER'\""`
COMMAND="ALTER"
if [ -z "$RESULT" ]; then
COMMAND="CREATE"
fi
su - postgres -c "psql postgres -c \"$COMMAND USER $POSTGRES_USER WITH SUPERUSER ENCRYPTED PASSWORD '$POSTGRES_PASS';\""
echo "Creating replication user $REPLICATION_USER"
RESULT_REPLICATION=`su - postgres -c "psql postgres -t -c \"SELECT 1 FROM pg_roles WHERE rolname = '$REPLICATION_USER'\""`
COMMANDS="ALTER"
if [ -z "$RESULT_REPLICATION" ]; then
COMMANDS="CREATE"
fi
su - postgres -c "psql postgres -c \"$COMMANDS USER $REPLICATION_USER WITH REPLICATION ENCRYPTED PASSWORD '$REPLICATION_PASS';\""