2021-08-19 13:55:32 +00:00
[![Scenario Tests ](https://github.com/kartoza/docker-postgis/actions/workflows/build-latest.yaml/badge.svg?branch=develop&event=push )](https://github.com/kartoza/docker-postgis/actions/workflows/build-latest.yaml)
[![deploy-image ](https://github.com/kartoza/docker-postgis/actions/workflows/deploy-image.yaml/badge.svg )](https://github.com/kartoza/docker-postgis/actions/workflows/deploy-image.yaml)
2020-01-26 06:54:13 +00:00
2014-05-04 19:14:46 +00:00
# docker-postgis
2014-05-04 18:52:28 +00:00
2014-05-04 19:14:46 +00:00
A simple docker container that runs PostGIS
2019-01-25 12:57:23 +00:00
Visit our page on the docker hub at: https://hub.docker.com/r/kartoza/postgis/
2014-12-06 04:50:34 +00:00
2014-08-05 09:16:50 +00:00
There are a number of other docker postgis containers out there. This one
differentiates itself by:
2021-05-18 07:52:45 +00:00
* provides ssl support out of the box and enforces ssl client connections
2014-08-05 09:16:50 +00:00
* connections are restricted to the docker subnet
2017-01-20 15:32:24 +00:00
* a default database 'gis' is created for you so you can use this container 'out of the
2014-08-05 09:16:50 +00:00
box' when it runs with e.g. QGIS
2021-05-18 07:52:45 +00:00
* Streaming replication and logical replication support included (turned off by default)
2019-02-20 10:28:14 +00:00
* Ability to create multiple database when you spin the database.
2021-05-18 07:52:45 +00:00
* Ability to create multiple schemas when spinning the database.
* Enable multiple extensions in the database when setting it up.
* Gdal drivers automatically registered for pg raster.
* Support for out-of-db rasters.
2014-08-05 09:16:50 +00:00
2019-12-03 09:39:15 +00:00
We will work to add more security features to this container in the future with
the aim of making a PostGIS image that is ready to be used in a production
2014-08-05 09:16:50 +00:00
environment (though probably not for heavy load databases).
2018-03-21 20:53:39 +00:00
There is a nice 'from scratch' tutorial on using this docker image on Alex Urquhart's
blog [here ](https://alexurquhart.com/post/set-up-postgis-with-docker/ ) - if you are
just getting started with docker, PostGIS and QGIS, we really recommend that you use it.
2015-08-11 09:14:55 +00:00
## Tagged versions
The following convention is used for tagging the images we build:
2019-12-27 15:38:23 +00:00
kartoza/postgis:[postgres_major_version]-[postgis-point-releases]
2015-08-11 09:14:55 +00:00
So for example:
2020-10-09 06:09:56 +00:00
``kartoza/postgis:13.0`` Provides PostgreSQL 13.0, PostGIS 3.0
2015-08-11 09:14:55 +00:00
**Note:** We highly recommend that you use tagged versions because
successive minor versions of PostgreSQL write their database clusters
into different database directories - which will cause your database
to appear to be empty if you are using persistent volumes for your
database storage.
2014-09-10 11:17:37 +00:00
## Getting the image
2014-05-04 19:14:46 +00:00
2014-09-10 11:17:37 +00:00
There are various ways to get the image onto your system:
The preferred way (but using most bandwidth for the initial image) is to
get our docker trusted build like this:
2021-06-01 13:19:05 +00:00
```shell
2020-06-14 16:22:13 +00:00
docker pull kartoza/postgis:image_version
2014-09-10 11:17:37 +00:00
```
2020-06-14 16:22:13 +00:00
## Building the image
2021-01-02 21:34:42 +00:00
To build the image yourself do:
2014-05-04 19:14:46 +00:00
2021-06-01 13:19:05 +00:00
```shell
2014-08-05 09:16:50 +00:00
docker build -t kartoza/postgis git://github.com/kartoza/docker-postgis
2014-05-04 19:47:26 +00:00
```
2021-03-01 13:00:39 +00:00
Alternatively clone the repository and build against any preferred branch
2014-05-04 19:47:26 +00:00
2021-06-01 13:19:05 +00:00
```shell
2014-09-10 11:17:37 +00:00
git clone git://github.com/kartoza/docker-postgis
2020-06-14 16:22:13 +00:00
git checkout branch_name
2014-05-04 19:47:26 +00:00
```
2020-06-14 16:22:13 +00:00
Then do:
2014-05-04 19:47:26 +00:00
2021-06-01 13:19:05 +00:00
```shell
2014-05-04 20:31:00 +00:00
docker build -t kartoza/postgis .
2014-05-04 19:14:46 +00:00
```
2021-01-02 21:34:42 +00:00
Or build against a specific PostgreSQL version
2020-10-09 06:09:56 +00:00
2021-06-01 13:19:05 +00:00
```shell
2020-10-09 06:09:56 +00:00
docker build --build-arg POSTGRES_MAJOR_VERSION=13 --build-arg POSTGIS_MAJOR=3 -t kartoza/postgis:POSTGRES_MAJOR_VERSION .
```
2021-06-01 13:19:05 +00:00
2020-06-14 16:22:13 +00:00
#### Alternative base distributions builds
2020-04-20 10:05:39 +00:00
There are build args for `DISTRO` (=debian), `IMAGE_VERSION` (=buster)
and `IMAGE_VARIANT` (=slim) which can be used to control the base image used
(but it still needs to be Debian based and have PostgreSQL official apt repo).
For example making Ubuntu 20.04 based build (for better arm64 support)
2021-05-18 07:52:45 +00:00
Edit the `.env` file to change the build arguments
2020-10-01 15:55:53 +00:00
2021-06-01 13:19:05 +00:00
```dotenv
2021-05-18 07:52:45 +00:00
DISTRO=ubuntu
IMAGE_VERSION=focal
IMAGE_VARIANT=""
```
Then run the script
2020-10-01 15:55:53 +00:00
2021-06-01 13:19:05 +00:00
```shell
2021-05-18 07:52:45 +00:00
./build.sh
2020-04-20 10:05:39 +00:00
```
2020-04-24 10:24:59 +00:00
#### Locales
2021-03-01 13:00:39 +00:00
By default, the image build will include **all** `locales` to cover any value for `locale` settings such as `DEFAULT_COLLATION` , `DEFAULT_CTYPE` or `DEFAULT_ENCODING` .
2020-04-24 10:24:59 +00:00
2021-05-18 07:52:45 +00:00
You can use the build argument: `GENERATE_ALL_LOCALE=0`
This will build with the default locate and speed up the build considerably.
2020-04-24 10:24:59 +00:00
2020-06-14 16:22:13 +00:00
### Environment variables
2016-09-06 07:15:07 +00:00
2020-04-08 08:15:45 +00:00
#### Cluster Initializations
2021-01-02 21:34:42 +00:00
With a minimum setup, our image will use an initial cluster located in the
2020-04-23 12:10:09 +00:00
`DATADIR` environment variable. If you want to use persistence, mount these
2021-01-02 21:34:42 +00:00
locations into your volume/host. By default, `DATADIR` will point to `/var/lib/postgresql/{major-version}` .
2020-04-08 08:15:45 +00:00
You can instead mount the parent location like this:
* `-v data-volume:/var/lib/postgresql`
This default cluster will be initialized with default locale settings `C.UTF-8` .
If, for instance, you want to create a new cluster with your own settings (not using the default cluster).
You need to specify different empty directory, like this
2021-06-01 13:19:05 +00:00
```shell
2020-04-08 08:15:45 +00:00
-v data-volume:/opt/postgres/data \
-e DATADIR:/opt/postgres/data \
-e DEFAULT_ENCODING="UTF8" \
-e DEFAULT_COLLATION="id_ID.utf8" \
-e DEFAULT_CTYPE="id_ID.utf8" \
2021-05-18 07:52:45 +00:00
-e PASSWORD_AUTHENTICATION="md5" \
2021-07-01 08:31:07 +00:00
-e INITDB_EXTRA_ARGS="< some more initdb command args > " \
-v pgwal-volume:/opt/postgres/pg_wal \
-e POSTGRES_INITDB_WALDIR=/opt/postgres/pg_wal
2020-04-08 08:15:45 +00:00
```
2020-04-23 12:10:09 +00:00
The containers will use above parameters to initialize a new db cluster in the
2020-04-08 08:15:45 +00:00
specified directory. If the directory is not empty, then initialization parameter will be ignored.
2021-08-19 13:55:32 +00:00
These are some initialization parameters that will only be used to initialize a new cluster.
2021-01-02 21:34:42 +00:00
If the container uses an existing cluster, it is ignored (for example, when the container restarts).
2020-04-08 08:15:45 +00:00
* `DEFAULT_ENCODING` : cluster encoding
* `DEFAULT_COLLATION` : cluster collation
* `DEFAULT_CTYPE` : cluster ctype
* `WAL_SEGSIZE` : WAL segsize option
2021-05-18 07:52:45 +00:00
* `PASSWORD_AUTHENTICATION` : PASSWORD AUTHENTICATION
2020-04-08 08:15:45 +00:00
* `INITDB_EXTRA_ARGS` : extra parameter that will be passed down to `initdb` command
2021-08-19 13:55:32 +00:00
* `POSTGRES_INITDB_WALDIR` : parameter to tell postgres about the initial waldir location. Note that you must always mount persistent volume to this location. Postgres will expect that the directory will always be available, even though it doesn't need the environment variable anymore. If you didn't persist this location, postgres will not be able to find the `pg_wal` directory and consider the instance to be broken.
2020-04-08 08:15:45 +00:00
In addition to that, we have another parameter: `RECREATE_DATADIR` that can be used to force database reinitializations.
2020-04-23 12:10:09 +00:00
If this parameter is specified as `TRUE` it will act as explicit consent to delete `DATADIR` and create
2020-04-08 08:15:45 +00:00
new db cluster.
2020-08-28 18:40:31 +00:00
* `RECREATE_DATADIR` : Force database reinitialization in the location `DATADIR`
2020-04-08 08:15:45 +00:00
2021-01-02 21:34:42 +00:00
If you used `RECREATE_DATADIR` and successfully created a new cluster. Remember
2020-04-23 12:10:09 +00:00
that you should remove this parameter afterwards. Because, if it was not omitted,
2020-04-08 08:15:45 +00:00
it will always recreate new db cluster after every container restarts.
2020-06-14 16:22:13 +00:00
#### Postgres Encoding
The database cluster is initialised with the following encoding settings
`
-E "UTF8" --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8"
`
or
`
-E "UTF8" --lc-collate="C.UTF-8" --lc-ctype="C.UTF-8"
`
If you use default `DATADIR` location.
2021-01-02 21:34:42 +00:00
If you need to set up a database cluster with other encoding parameters you need
2020-06-14 16:22:13 +00:00
to pass the environment variables when you initialize the cluster.
2021-06-01 13:19:05 +00:00
* `-e DEFAULT_ENCODING="UTF8"`
* `-e DEFAULT_COLLATION="en_US.UTF-8"`
* `-e DEFAULT_CTYPE="en_US.UTF-8"`
2020-06-14 16:22:13 +00:00
Initializing a new cluster can be done by using different `DATADIR` location and
mounting an empty volume. Or use parameter `RECREATE_DATADIR` to forcefully
delete the current cluster and create a new one. Make sure to remove parameter
`RECREATE_DATADIR` after creating the cluster.
2021-05-17 19:50:50 +00:00
See [the postgres documentation about encoding ](https://www.postgresql.org/docs/11/multibyte.html ) for more information.
2020-06-14 16:22:13 +00:00
2020-02-12 09:34:49 +00:00
#### Basic configuration
You can use the following environment variables to pass a
2021-01-02 21:34:42 +00:00
username, password and/or default database name(or multiple databases comma separated).
2014-10-02 14:52:45 +00:00
2020-02-12 09:34:49 +00:00
* `-e POSTGRES_USER=<PGUSER>`
* `-e POSTGRES_PASS=<PGPASSWORD>`
2020-08-28 18:40:31 +00:00
**NB** You should use a strong passwords. If you are using docker-compose make sure
2021-03-01 13:00:39 +00:00
docker can interpolate the password. Example using a password with a `$` you will
2020-08-28 18:40:31 +00:00
need to escape it ie `$$`
2020-02-12 09:34:49 +00:00
* `-e POSTGRES_DBNAME=<PGDBNAME>`
2020-04-23 12:10:09 +00:00
* `-e POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting`
2019-12-27 15:38:23 +00:00
You can pass as many extensions as you need.
2020-04-14 17:00:36 +00:00
* `-e SHARED_PRELOAD_LIBRARIES='pg_cron'`
2021-06-01 13:19:05 +00:00
Some extensions need to be registered in the `postgresql.conf`
2020-04-14 17:00:36 +00:00
as shared_preload_libraries. pg_cron should always be added because
the extension is installed with the image.
2020-02-12 09:34:49 +00:00
* `-e SSL_CERT_FILE=/your/own/ssl_cert_file.pem`
* `-e SSL_KEY_FILE=/your/own/ssl_key_file.key`
* `-e SSL_CA_FILE=/your/own/ssl_ca_file.pem`
* `-e DEFAULT_ENCODING="UTF8"`
* `-e DEFAULT_COLLATION="en_US.UTF-8"`
* `-e DEFAULT_CTYPE="en_US.UTF-8"`
2020-04-23 12:10:09 +00:00
2020-02-12 09:34:49 +00:00
* `-e POSTGRES_TEMPLATE_EXTENSIONS=true`
2019-09-09 06:20:12 +00:00
2021-06-01 13:19:05 +00:00
Specifies whether extensions will also be installed in template1 database.
2020-06-14 16:22:13 +00:00
2021-03-01 06:42:38 +00:00
### Schema Initialisation
2021-06-01 13:19:05 +00:00
2021-03-01 06:42:38 +00:00
* `-e SCHEMA_NAME=<PGSCHEMA>`
You can pass a comma separated value of schema names which will be created when the
database initialises. The default behaviour is to create the schema in the first
database specified in the environment variable `POSTGRES_DBNAME` . If you need to
create matching schemas in all the databases that will be created you use
the environment variable `ALL_DATABASES=TRUE`
2021-03-02 10:48:46 +00:00
2020-02-12 09:34:49 +00:00
#### Configures archive mode
This image uses the initial PostgreSQL values which disables the archiving option by default.
When `ARCHIVE_MODE` is changed to `on` , the archiving command will copy WAL files to `/opt/archivedir`
[More info: 19.5. Write Ahead Log ](https://www.postgresql.org/docs/12/runtime-config-wal.html )
2019-09-06 11:47:42 +00:00
2020-04-22 10:25:05 +00:00
* `-e ARCHIVE_MODE=off`
* `-e ARCHIVE_COMMAND="test ! -f /opt/archivedir/%f && cp %p /opt/archivedir/%f"`
2020-02-12 09:34:49 +00:00
[More info ](https://www.postgresql.org/docs/12/continuous-archiving.html#BACKUP-ARCHIVING-WAL )
2020-04-22 10:25:05 +00:00
* `-e ARCHIVE_CLEANUP_COMMAND="pg_archivecleanup /opt/archivedir %r"`
* `-e RESTORE_COMMAND='cp /opt/archivedir/%f "%p"'`
2014-10-02 14:52:45 +00:00
2020-02-12 09:34:49 +00:00
#### Configure WAL level
2021-06-01 13:19:05 +00:00
2020-02-12 09:34:49 +00:00
* `-e WAL_LEVEL=replica`
[More info ](https://www.postgresql.org/docs/12/runtime-config-wal.html )
Maximum size to let the WAL grow to between automatic WAL checkpoints.
* `-e WAL_SIZE=4GB`
* `-e MIN_WAL_SIZE=2048MB`
* `-e WAL_SEGSIZE=1024`
* `-e MAINTAINANCE_WORK_MEM=128MB`
#### Configure networking
2021-06-01 13:19:05 +00:00
2021-01-02 21:34:42 +00:00
You can open up the PG port by using the following environment variable. By default,
2016-09-06 07:14:20 +00:00
the container will allow connections only from the docker private subnet.
2020-02-12 09:34:49 +00:00
* `-e ALLOW_IP_RANGE=<0.0.0.0/0> By default`
2019-01-25 12:57:23 +00:00
2021-01-02 21:34:42 +00:00
Postgres conf is set up to listen to all connections and if a user needs to restrict which IP address
2019-12-03 09:39:15 +00:00
PostgreSQL listens to you can define it with the following environment variable. The default is set to listen to
2019-01-25 12:57:23 +00:00
all connections.
2020-02-12 09:34:49 +00:00
* `-e IP_LIST=<*>`
#### Additional configuration
2019-01-25 12:57:23 +00:00
2021-03-01 06:42:38 +00:00
You can also define any other configuration to add to `extra.conf` , separated by '\n' e.g.:
2019-10-25 12:43:15 +00:00
2020-02-12 09:34:49 +00:00
* `-e EXTRA_CONF="log_destination = 'stderr'\nlogging_collector = on"`
2016-09-06 07:14:20 +00:00
2021-03-01 06:42:38 +00:00
You can alternatively mount an extra config file into the setting's folder i.e
2021-06-01 13:19:05 +00:00
```shell
2021-03-01 06:42:38 +00:00
docker run --name "postgis" -v /data/extra.conf:/settings/extra.conf -p 25432:5432 -d -t kartoza/postgis
```
2020-04-08 08:15:45 +00:00
If you want to reinitialize the data directory from scratch, you need to do:
2020-02-25 08:19:59 +00:00
2020-04-08 08:15:45 +00:00
1. Do backup, move data, etc. Any preparations before deleting your data directory.
2. Set environment variables `RECREATE_DATADIR=TRUE` . Restart the service
2020-04-23 12:10:09 +00:00
3. The service will delete your `DATADIR` directory and start reinitializing your data directory from scratch.
2016-09-06 07:14:20 +00:00
2020-01-23 13:07:23 +00:00
## Docker secrets
To avoid passing sensitive information in environment variables, `_FILE` can be appended to
some of the variables to read from files present in the container. This is particularly useful
in conjunction with Docker secrets, as passwords can be loaded from `/run/secrets/<secret_name>` e.g.:
2021-06-01 13:19:05 +00:00
* `-e POSTGRES_PASS_FILE=/run/secrets/<pg_pass_secret>`
2020-01-23 13:07:23 +00:00
For more information see [https://docs.docker.com/engine/swarm/secrets/ ](https://docs.docker.com/engine/swarm/secrets/ ).
2021-05-18 07:52:45 +00:00
Currently, `POSTGRES_PASS` , `POSTGRES_USER` , `POSTGRES_DB` , `SSL_CERT_FILE` ,
`SSL_KEY_FILE` , `SSL_CA_FILE` are supported.
2020-01-23 13:07:23 +00:00
2019-12-27 15:38:23 +00:00
2020-06-14 16:22:13 +00:00
## Running the container
### Using the terminal
To create a running container do:
2021-06-01 13:19:05 +00:00
```shell
2020-06-14 16:22:13 +00:00
docker run --name "postgis" -p 25432:5432 -d -t kartoza/postgis
```
2021-06-27 17:12:37 +00:00
**Note** If you do not pass the env variable `POSTGRES_PASS` a random password
2021-06-07 06:49:30 +00:00
will be generated and will be visible from the logs or within the container in
`/tmp/PGPASSWORD.txt`
2018-03-21 20:53:39 +00:00
## Convenience docker-compose.yml
2014-10-02 14:52:45 +00:00
2021-05-18 07:52:45 +00:00
For convenience, we provide a ``docker-compose.yml`` that will run a
2019-12-03 09:39:15 +00:00
copy of the database image and also our related database backup image (see
2018-03-21 20:53:39 +00:00
[https://github.com/kartoza/docker-pg-backup ](https://github.com/kartoza/docker-pg-backup )).
2014-10-02 14:52:45 +00:00
2018-03-21 20:53:39 +00:00
The docker compose recipe will expose PostgreSQL on port 25432 (to prevent
potential conflicts with any local database instance you may have).
2014-10-02 14:52:45 +00:00
Example usage:
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
docker-compose up -d
2014-10-02 14:52:45 +00:00
```
2018-03-21 20:53:39 +00:00
**Note:** The docker-compose recipe above will not persist your data on your local
disk, only in a docker volume.
2014-05-04 19:14:46 +00:00
## Connect via psql
Connect with psql (make sure you first install postgresql client tools on your
host / client):
2021-06-01 13:19:05 +00:00
```shell
2014-05-04 19:14:46 +00:00
psql -h localhost -U docker -p 25432 -l
```
2014-05-04 20:31:00 +00:00
**Note:** Default postgresql user is 'docker' with password 'docker'.
2014-05-04 19:14:46 +00:00
You can then go on to use any normal postgresql commands against the container.
2019-01-25 12:57:23 +00:00
Under ubuntu 16.04 the postgresql client can be installed like this:
2014-05-04 19:14:46 +00:00
2021-06-01 13:19:05 +00:00
```shell
2019-12-02 13:14:29 +00:00
sudo apt-get install postgresql-client-12
2014-05-04 19:14:46 +00:00
```
2019-04-29 14:00:29 +00:00
## Running SQL scripts on container startup.
2019-12-03 09:39:15 +00:00
In some instances users want to run some SQL scripts to populate the
2021-05-18 07:52:45 +00:00
database. The environment variable POSTGRES_DB allows
2019-04-29 14:00:29 +00:00
us to specify multiple database that can be created on startup.
2019-12-03 09:39:15 +00:00
When running scripts they will only be executed against the
2019-04-29 14:00:29 +00:00
first database ie POSTGRES_DB=gis,data,sample
2021-05-18 07:52:45 +00:00
The SQL script will be executed against the `gis` database. Additionally, a lock file is generated in
`/docker-entrypoint-initdb.d` , which will prevent the scripts from getting executed after the first
container startup. Provide `IGNORE_INIT_HOOK_LOCKFILE=true` to execute the scripts on _every_ container start.
2019-04-29 14:00:29 +00:00
2021-06-01 13:19:05 +00:00
Currently, you can pass `.sql` , `.sql.gz` and `.sh` files as mounted volumes.
2019-04-29 14:00:29 +00:00
2021-06-01 13:19:05 +00:00
```shell
2021-06-20 13:47:21 +00:00
docker run -d -v `pwd` /setup-db.sql:/docker-entrypoint-initdb.d/setup-db.sql kartoza/postgis
2019-04-29 14:00:29 +00:00
```
2014-05-04 20:31:00 +00:00
## Storing data on the host rather than the container.
2014-05-04 19:14:46 +00:00
Docker volumes can be used to persist your data.
2021-06-01 13:19:05 +00:00
```shell
2014-05-04 19:14:46 +00:00
mkdir -p ~/postgres_data
2021-06-01 13:19:05 +00:00
docker run -d -v $HOME/postgres_data:/var/lib/postgresql kartoza/postgis
2014-05-04 19:14:46 +00:00
```
2015-08-08 12:11:45 +00:00
You need to ensure the ``postgres_data`` directory has sufficient permissions
2014-05-04 19:14:46 +00:00
for the docker process to read / write it.
2020-06-14 16:22:13 +00:00
## Postgres SSL setup
2021-05-17 15:33:27 +00:00
There are three modalities in which you can work with SSL:
1. Optional: using the shipped snakeoil certificates
2. Forced SSL: forced using the shipped snakeoil certificates
3. Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority
By default, the image is delivered with an unsigned SSL certificate.
This helps to have an encrypted connection to clients and avoid eavesdropping but does not help to mitigate
2020-06-14 16:22:13 +00:00
man in the middle (MITM) attacks.
You need to provide your own, signed private key to avoid this kind of attacks (and make
sure clients connect with verify-ca or verify-full sslmode).
2021-05-17 15:33:27 +00:00
Although SSL is enabled by default, connection to PostgreSQL with other clients
i.e (PSQL or QGIS) still doesn't enforce SSL encryption. To force SSL connection between clients you
need to use the environment variable
2021-06-01 13:19:05 +00:00
```shell
2021-05-17 15:33:27 +00:00
FORCE_SSL=TRUE
```
2021-05-17 19:50:50 +00:00
The following example sets up a container with custom ssl private key and certificate:
2020-06-14 16:22:13 +00:00
2021-06-01 13:19:05 +00:00
```shell
2021-05-17 19:50:50 +00:00
docker run -p 25432:5432 -e FORCE_SSL=TRUE -e SSL_DIR="/etc/ssl_certificates" -e SSL_CERT_FILE='/etc/ssl_certificates/fullchain.pem' -e SSL_KEY_FILE='/etc/ssl_certificates/privkey.pem' -e SSL_CA_FILE='/etc/ssl_certificates/root.crt' -v /tmp/postgres/letsencrypt:/etc/ssl_certificates --name ssl -d kartoza/postgis:13-3.1
2020-06-14 16:22:13 +00:00
```
2021-05-17 19:50:50 +00:00
The environment variable `SSL_DIR` allows a user to specify the location
where custom SSL certificates will be located. The environment variable currently
defaults to `SSL_DIR=/ssl_certificates`
2020-06-14 16:22:13 +00:00
See [the postgres documentation about SSL ](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBQ-SSL-CERTIFICATES ) for more information.
2021-05-17 15:33:27 +00:00
### Forced SSL: forced using the shipped snakeoil certificates
2020-06-14 16:22:13 +00:00
2021-05-16 09:20:18 +00:00
If you are using the default certificates provided by the image when connecting
to the database you will need to set `SSL Mode` to any value besides
`verify-full` or `verify-ca`
2021-05-18 07:52:45 +00:00
The pg_hba.con will have entries like:
2021-06-01 13:19:05 +00:00
```shell
2021-05-18 07:52:45 +00:00
hostssl all all 0.0.0.0/0 scram-sha-256 clientcert=0
```
2021-06-01 13:19:05 +00:00
where `PASSWORD_AUTHENTICATION=scram-sha-256` and `ALLOW_IP_RANGE=0.0.0.0/0`
2021-05-18 07:52:45 +00:00
2021-05-17 15:33:27 +00:00
### Forced SSL with Certificate Exchange: using SSL certificates signed by a certificate authority
2021-05-16 09:20:18 +00:00
2021-05-17 15:33:27 +00:00
When setting up the database you need to define the following environment variables.
2021-05-16 09:20:18 +00:00
SSL_CERT_FILE:
SSL_KEY_FILE:
SSL_CA_FILE:
2021-05-17 15:33:27 +00:00
Example:
2021-06-01 13:19:05 +00:00
```shell
2021-05-17 15:33:27 +00:00
docker run -p 5432:5432 -e FORCE_SSL=TRUE -e SSL_CERT_FILE='/ssl_certificates/fullchain.pem' -e SSL_KEY_FILE='/ssl_certificates/privkey.pem' -e SSL_CA_FILE='/ssl_certificates/root.crt' --name ssl -d kartoza/postgis:13-3.1
```
2021-05-16 09:20:18 +00:00
On the host machine where you need to connect to the database you also
need to copy the `SSL_CA_FILE` file to the location `/home/$user/.postgresql/root.crt`
or define an environment variable pointing to location of the `SSL_CA_FILE`
example: `PGSSLROOTCERT=/etc/letsencrypt/root.crt`
2021-06-01 13:19:05 +00:00
The `pg_hba.conf` will have entries like:
2021-05-18 07:52:45 +00:00
```
hostssl all all 0.0.0.0/0 cert
```
where `ALLOW_IP_RANGE=0.0.0.0/0`
2021-05-16 09:20:18 +00:00
#### SSL connection inside the docker container using openssl certificates
Generate the certificates inside the container
2021-06-01 13:19:05 +00:00
```shell
2021-05-17 15:33:27 +00:00
CERT_DIR=/ssl_certificates
mkdir $CERT_DIR
openssl req -x509 -newkey rsa:4096 -keyout ${CERT_DIR}/privkey.pem -out \
${CERT_DIR}/fullchain.pem -days 3650 -nodes -sha256 -subj '/CN=localhost'
2021-05-16 09:20:18 +00:00
2021-05-17 15:33:27 +00:00
cp $CERT_DIR/fullchain.pem $CERT_DIR/root.crt
chmod -R 0700 ${CERT_DIR}
chown -R postgres ${CERT_DIR}
2021-05-16 09:20:18 +00:00
```
Set up your ssl config to point to the new location
2021-06-01 13:19:05 +00:00
2021-05-16 09:20:18 +00:00
```
ssl = true
2021-05-17 15:33:27 +00:00
ssl_cert_file = '/ssl_certificates/fullchain.pem'
ssl_key_file = '/ssl_certificates/privkey.pem'
ssl_ca_file = '/ssl_certificates/root.crt'
2021-05-16 09:20:18 +00:00
```
2021-05-17 15:33:27 +00:00
2021-05-16 09:20:18 +00:00
Then connect to the database using the psql command:
2021-06-01 13:19:05 +00:00
```shell
psql "dbname=gis port=5432 user=docker host=localhost sslmode=verify-full sslcert=/etc/letsencrypt/fullchain.pem sslkey=/etc/letsencrypt/privkey.pem sslrootcert=/etc/letsencrypt/root.crt"
2021-05-16 09:20:18 +00:00
```
2018-03-21 20:53:39 +00:00
## Postgres Replication Setup
2021-05-17 15:33:27 +00:00
The image supports replication out of the box. By default, replication is turned off.
2020-10-09 06:09:56 +00:00
The two mains replication methods allowed are
2020-10-01 15:55:53 +00:00
* Streaming replication
* Logical replication
### Streaming replication
2020-10-09 06:09:56 +00:00
2018-03-22 07:06:45 +00:00
Replication allows you to maintain two or more synchronised copies of a database, with a
2019-12-03 09:39:15 +00:00
single **master** copy and one or more **replicant** copies. The animation below illustrates
this - the layer with the red boundary is accessed from the master database and the layer
with the green fill is accessed from the replicant database. When edits to the master
layer are saved, they are automatically propagated to the replicant. Note also that the
2018-03-22 07:06:45 +00:00
replicant is read-only.
2021-06-01 13:19:05 +00:00
```shell
2020-10-09 06:09:56 +00:00
docker run --name "streaming-replication" -e REPLICATION=true -e WAL_LEVEL='replica' -d -p 25432:5432 kartoza/postgis:13.0
```
2021-06-27 17:12:37 +00:00
**Note** If you do not pass the env variable `REPLICATION_PASS` a random password
will be generated and will be visible from the logs or within the container in
`/tmp/REPLPASSWORD.txt`
2020-10-09 06:09:56 +00:00
2018-03-22 07:06:45 +00:00
![qgis ](https://user-images.githubusercontent.com/178003/37755610-dd3b774a-2dae-11e8-9fa1-4877e2034675.gif )
2019-12-03 09:39:15 +00:00
This image is provided with replication abilities. We can
categorize an instance of the container as `master` or `replicant` . A `master`
instance means that a particular container has a role as a single point of
database write. A `replicant` instance means that a particular container will
mirror database content from a designated master. This replication scheme allows
2021-05-18 07:52:45 +00:00
us to sync databases. However, a `replicant` is only for read-only transaction, thus
2019-01-25 12:57:23 +00:00
we can't write new data to it. The whole database cluster will be replicated.
2018-03-21 20:53:39 +00:00
2020-10-01 15:55:53 +00:00
#### Database permissions
2021-06-01 13:19:05 +00:00
2019-12-03 09:39:15 +00:00
Since we are using a role ${REPLICATION_USER}, we need to ensure that it has access to all
2019-10-22 09:31:59 +00:00
the tables in a particular schema. So if a user adds another schema called `data`
to the database `gis` he also has to update the permission for the user
with the following SQL assuming the ${REPLICATION_USER} is called replicator
2021-06-01 13:19:05 +00:00
```sql
ALTER DEFAULT PRIVILEGES IN SCHEMA data GRANT SELECT ON TABLES TO replicator;
```
2020-06-14 16:22:13 +00:00
2021-05-18 07:52:45 +00:00
**NB** You need to set up a strong password for replication otherwise the
2020-06-14 16:22:13 +00:00
default password for ${REPLICATION_USER} will default to `replicator`
2020-07-12 06:58:31 +00:00
To experiment with the replication abilities, you can see a [docker-compose.yml ](sample/replication/docker-compose.yml )
2019-01-25 12:57:23 +00:00
sample. There are several environment variables that you can set, such as:
2018-03-21 20:53:39 +00:00
Master settings:
2021-06-01 13:19:05 +00:00
- **ALLOW_IP_RANGE**: A `pg_hba.conf` domain format which will allow specified host(s)
2019-12-03 09:39:15 +00:00
to connect into the container. This is needed to allow the `slave` to connect
2021-06-01 13:19:05 +00:00
into `master` , so specifically these settings should allow `slave` address. It is also needed to allow clients on other hosts to connect to either the slave or the master.
2019-11-25 12:47:29 +00:00
- **REPLICATION_USER** User to initiate streaming replication
2020-01-26 06:54:13 +00:00
- **REPLICATION_PASS** Password for a user with streaming replication role
2019-12-03 09:39:15 +00:00
2018-03-21 20:53:39 +00:00
Slave settings:
2019-12-03 09:39:15 +00:00
- **REPLICATE_FROM**: This should be the domain name or IP address of the `master`
instance. It can be anything from the docker resolved name like that written in the sample,
or the IP address of the actual machine where you expose `master` . This is
2018-03-21 20:53:39 +00:00
useful to create cross machine replication, or cross stack/server.
2019-12-03 09:39:15 +00:00
- **REPLICATE_PORT**: This should be the port number of `master` postgres instance.
2018-03-21 20:53:39 +00:00
Will default to 5432 (default postgres port), if not specified.
2019-12-03 09:39:15 +00:00
- **DESTROY_DATABASE_ON_RESTART**: Default is `True` . Set to 'False' to prevent
this behaviour. A replicant will always destroy its current database on
2018-03-21 20:53:39 +00:00
restart, because it will try to sync again from `master` and avoid inconsistencies.
2019-12-03 09:39:15 +00:00
- **PROMOTE_MASTER**: Default none. If set to any value then the current replicant
will be promoted to master.
In some cases when the `master` container has failed, we might want to use our `replicant`
as `master` for a while. However, the promoted replicant will break consistencies and
is not able to revert to replicant anymore, unless it is destroyed and resynced
2018-03-21 20:53:39 +00:00
with the new master.
2019-11-25 12:47:29 +00:00
- **REPLICATION_USER** User to initiate streaming replication
2020-01-26 06:54:13 +00:00
- **REPLICATION_PASS** Password for a user with streaming replication role
2018-03-21 20:53:39 +00:00
2019-01-25 12:57:23 +00:00
To run the sample replication, follow these instructions:
2018-03-21 20:53:39 +00:00
2019-01-25 12:57:23 +00:00
Do a manual image build by executing the `build.sh` script
2018-03-21 20:53:39 +00:00
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
./build.sh
```
2019-12-03 09:39:15 +00:00
Go into the `sample/replication` directory and experiment with the following Make
2018-03-21 20:53:39 +00:00
command to run both master and slave services.
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
make up
```
2021-05-18 07:52:45 +00:00
To shut down services, execute:
2018-03-21 20:53:39 +00:00
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
make down
```
To view logs for master and slave respectively, use the following command:
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
make master-log
make slave-log
```
You can try experiment with several scenarios to see how replication works
2020-10-01 15:55:53 +00:00
#### Sync changes from master to replicant
2018-03-21 20:53:39 +00:00
2019-12-03 09:39:15 +00:00
You can use any postgres database tools to create new tables in master, by
2018-03-21 20:53:39 +00:00
connecting using POSTGRES_USER and POSTGRES_PASS credentials using exposed port.
2019-01-25 12:57:23 +00:00
In the sample, the master database was exposed on port 7777.
2018-03-21 20:53:39 +00:00
Or you can do it via command line, by entering the shell:
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
make master-shell
```
Then made any database changes using psql.
2019-01-25 12:57:23 +00:00
After that, you can see that the replicant follows the changes by inspecting the
2019-12-03 09:39:15 +00:00
slave database. You can, again, use database management tools using connection
credentials, hostname, and ports for replicant. Or you can do it via command line,
2018-03-21 20:53:39 +00:00
by entering the shell:
2021-06-01 13:19:05 +00:00
```shell
2018-03-21 20:53:39 +00:00
make slave-shell
```
Then view your changes using psql.
2020-10-01 15:55:53 +00:00
#### Promoting replicant to master
2018-03-21 20:53:39 +00:00
2019-01-25 12:57:23 +00:00
You will notice that you cannot make changes in replicant, because it is read-only.
2019-12-03 09:39:15 +00:00
If somehow you want to promote it to master, you can specify `PROMOTE_MASTER: 'True'`
into slave environment and set `DESTROY_DATABASE_ON_RESTART: 'False'` .
2018-03-21 20:53:39 +00:00
2019-12-03 09:39:15 +00:00
After this, you can make changes to your replicant, but master and replicant will not
be in sync anymore. This is useful if the replicant needs to take over a failover master.
However it is recommended to take additional action, such as creating a backup from the
2019-01-25 12:57:23 +00:00
slave so a dedicated master can be created again.
2018-03-21 20:53:39 +00:00
2020-10-01 15:55:53 +00:00
#### Preventing replicant database destroy on restart
2018-03-21 20:53:39 +00:00
2019-12-03 09:39:15 +00:00
You can optionally set `DESTROY_DATABASE_ON_RESTART: 'False'` after successful sync
to prevent the database from being destroyed on restart. With this setting you can
shut down your replicant and restart it later and it will continue to sync using the existing
2019-01-25 12:57:23 +00:00
database (as long as there are no consistencies conflicts).
2014-05-04 19:14:46 +00:00
2019-12-03 09:39:15 +00:00
However, you should note that this option doesn't mean anything if you didn't
persist your database volume. Because if it is not persisted, then it will be lost
2018-03-21 20:53:39 +00:00
on restart because docker will recreate the container.
2014-05-04 19:14:46 +00:00
2021-06-01 13:19:05 +00:00
### Logical replication
2020-10-01 15:55:53 +00:00
To activate the following you need to use the environment variable
`WAL_LEVEL=logical` to get a running instance like
2021-06-01 13:19:05 +00:00
```shell
2020-10-09 06:09:56 +00:00
docker run --name "logical-replication" -e WAL_LEVEL=logical -d kartoza/postgis:13.0
2020-10-01 15:55:53 +00:00
```
2021-06-01 13:19:05 +00:00
2020-10-01 15:55:53 +00:00
For a detailed example see the docker-compose in the folder `sample/logical_replication` .
2021-05-04 11:10:19 +00:00
### Docker image versions
2021-06-01 13:19:05 +00:00
2021-05-04 11:10:19 +00:00
All instructions mentioned in the README are valid for the latest running image.
Other docker images might have a few missing features than the ones in the
latest image. We mainly do not back port changes to current stable images that are being
used in production. However, if you feel that some changes included
in the latest tagged version of the image are essential for the previous image
2021-05-18 07:52:45 +00:00
you can cherry-pick the changes against that specific branch and we will
2021-05-04 11:10:19 +00:00
test and merge.
2019-09-06 11:47:42 +00:00
2020-06-14 15:06:54 +00:00
### Support
If you require more substantial assistance from [kartoza ](https://kartoza.com ) (because our work and interaction on docker-postgis is pro bono),
2021-03-01 13:00:39 +00:00
please consider taking out a [Support Level Agreeement ](https://kartoza.com/en/shop/product/support )
2019-06-13 15:39:40 +00:00
2014-05-04 20:31:00 +00:00
## Credits
2021-06-01 13:19:05 +00:00
- Tim Sutton (tim@kartoza.com)
- Gavin Fleming (gavin@kartoza.com)
- Rizky Maulana (rizky@kartoza.com)
- Admire Nyakudya (admire@kartoza.com)
2021-03-02 10:48:46 +00:00
March 2021