Problem setting environment variables through docker-compose

[datracka]

I have problems to create a custom DB in docker-compose using the environment variables and postgres docker image.

having this docker.compose.yml file and running docker-compose up -d

postgres:
  restart: 'true'
  image: postgres
  ports:
    - "5432:5432"
  environment:
    - POSTGRES_DB:test_db 

the container created does not contain the test_db database

But if I run the following command:

docker run -e POSTGRES_DB=test_db --name postgres-test -d -p 5432:5432 postgres

the created container DO have the test_db

I was reading the following links and everything looks good.

https://hub.docker.com/_/postgres/
https://docs.docker.com/engine/reference/run/#/env-environment-variables
https://docs.docker.com/compose/environment-variables/

Sure I am missing something because it should be straightforward. But what?

thank for your time and help!

[mrafayaleem]

I suspect this is happening because you have a previous image of this in your docker where you ran docker-compose up without providing the envs and hence it is not able to overwrite. Check and delete postgres image by doing docker ps -a and docker rm <image> respectively.

[datracka]

Hi,

it did not work :( I removed all the images in my local machine and regardless still not working.

Below the log output when I run $ docker-compose up

postgres_1  | The files belonging to this database system will be owned by user "postgres".
postgres_1  | This user must also own the server process.
postgres_1  | 
postgres_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres_1  | The default database encoding has accordingly been set to "UTF8".
postgres_1  | The default text search configuration will be set to "english".
postgres_1  | 
postgres_1  | Data page checksums are disabled.
postgres_1  | 
postgres_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1  | creating subdirectories ... ok
postgres_1  | selecting default max_connections ... 100
postgres_1  | selecting default shared_buffers ... 128MB
postgres_1  | selecting dynamic shared memory implementation ... posix
postgres_1  | creating configuration files ... ok
postgres_1  | running bootstrap script ... ok
postgres_1  | performing post-bootstrap initialization ... ok
postgres_1  | syncing data to disk ... ok
postgres_1  | 
postgres_1  | WARNING: enabling "trust" authentication for local connections
postgres_1  | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1  | --auth-local and --auth-host, the next time you run initdb.
postgres_1  | 
postgres_1  | Success. You can now start the database server using:
postgres_1  | 
postgres_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1  | 
postgres_1  | ****************************************************
postgres_1  | WARNING: No password has been set for the database.
postgres_1  |          This will allow anyone with access to the
postgres_1  |          Postgres port to access your database. In
postgres_1  |          Docker's default configuration, this is
postgres_1  |          effectively any other container on the same
postgres_1  |          system.
postgres_1  | 
postgres_1  |          Use "-e POSTGRES_PASSWORD=password" to set
postgres_1  |          it in "docker run".
postgres_1  | ****************************************************
postgres_1  | waiting for server to start....LOG:  database system was shut down at 2016-10-17 08:45:42 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
postgres_1  |  done
postgres_1  | server started
postgres_1  | ALTER ROLE
postgres_1  | 
postgres_1  | 
postgres_1  | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1  | 
postgres_1  | LOG:  received fast shutdown request
postgres_1  | LOG:  aborting any active transactions
postgres_1  | LOG:  autovacuum launcher shutting down
postgres_1  | LOG:  shutting down
postgres_1  | waiting for server to shut down....LOG:  database system is shut down
postgres_1  |  done
postgres_1  | server stopped
postgres_1  | 
postgres_1  | PostgreSQL init process complete; ready for start up.
postgres_1  | 
postgres_1  | LOG:  database system was shut down at 2016-10-17 08:45:43 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started

has something to do this line? /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* I am pointing in this direction: I assume my docker configuration should be tweak in some way... (I am using Docker for Mac Version 1.12.1 (build: 12133))

at the moment and as workaround I create the DB / user / pass manually.

[mrafayaleem]

I still don't get it but I am using the following docker-compose.yml and it works perfectly fine. I was in a similar problem that you have mentioned and the trick was to just remove the previous docker image and data dir and re-run the container from scratch.

version: '2'
services:
  postgres:
    image: postgres:9.3
    ports:
    - "5432:5432"
    environment:
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
      - POSTGRES_DB=test_db
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

[datracka]

your script works like a charm! Still I do not get exactly what I was doing wrong (maybe it was the version:'2' or nest the services under 'services' tag..), but it does not mind, I move forward. Thanks for your help!

[Chrisissorry]

Had the exact same problem. Removed all images and deleted the data directory as @mrafayaleem suggested - works!

[yosifkit]

Yeah, the problem is not the existing images on your host, but it stems from the volume (or bind mount directory) being already initialized after your first start. The postgres user, and database creation only happens on the first start (ie, /var/lib/postgresql/data must not already contain database files).

Here is an example flow

1. declare volume for my image via compose: ./data/postgres:/var/lib/postgresql/data
2. set env in yaml: POSTGRES_PASSWORD=test
3. start containers docker-compose up -d
4. decide to add database to postgres env POSTGRES_DB=test_db or change the password
5. restart my containers docker-compose up -d
6. database test_db does not exist or password is unchanged 😢
   1. db files must be removed in order for postres entrypoint to re-initialize: docker-compose stop; sudo rm -rf ./data/postgres/
   2. now it can be restarted docker-compose up -d 😃

If you do not declare a volume mount point, then the VOLUME declared in the postgres image will apply and docker will create and manage the directory independent of the life-cycle of the container. This gets more complicated when using compose, since it will keep the volume to re-use later even when you docker-compose rm -f all of your running containers.

An example without a bind mounted volume:

1. say we do steps 2-5 above
2. we still have the same problem of the test_db database not existing (or the password not changing) since the data from the volume still exists 😞
   1. so, get rid of all containers and their volumes docker-compose rm -fv
   2. or docker-compose rm -fv postgres to get rid of just the postres service and its volumes
   3. now we can start up a new postgres container with a new empty volume docker-compose up -d 😃

You can see what volumes you have on your host by doing a docker volume ls (bind mounts will not show up in this list). There is currently no easy way to see what containers are attached to a volume, so you would have to docker inspect the container to see which ones are attached to it.

If you want to clean up all local volumes that are not attached to containers (WARNING this could delete important data 😲): docker volume ls | awk '$1 == "local" { print $2 }' | xargs --no-run-if-empty docker volume rm. On my development machine I usually precede this by removing stopped containers.

Also, in case it is not obvious, do not delete your postres data directory or volume if you have important data stored there. 😱

[erikj]

I do not know for sure if it was the root cause, but the syntax used to specify the test_db environment variable appears to be incorrect and will not set POSTGRES_DB:

  environment:
    - POSTGRES_DB:test_db 

You can either specify it as a hash (no -):

  environment:
    POSTGRES_DB:test_db 

Or as an array, -, w/ = assignment rather than key:value

  environment:
    - POSTGRES_DB=test_db 

[i-pip]

This issue is still not resolved for me. I tried removing the images after shutdown with docker-compose down --rmi all then deleted my data/postgres folder, confirmed that there are no volumes still attached with docker volume ls and even tried --force-recreate with docker-compose up --force-recreate. docker-compose still ignores the environment variables in my docker-compose.yml
which looks like so:

services:
  web:
    restart: always
    build: . 
    ports:
      - "3000:3000"
    links:
      - db
  db:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=someuser
      - POSTGRES_PASSWORD=s0me-p4sswd
      - POSTGRES_DB=test-db
      - POSTGRES_PORT=5432
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    restart: always

I still get the following output

db_1   | syncing data to disk ... ok
db_1   |
db_1   | Success. You can now start the database server using:
db_1   |
db_1   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1   |
db_1   |
db_1   | WARNING: enabling "trust" authentication for local connections
db_1   | You can change this by editing pg_hba.conf or using the option -A, or
db_1   | --auth-local and --auth-host, the next time you run initdb.
db_1   | waiting for server to start....2018-02-16 05:10:10.335 UTC [40] LOG:  listening on IPv4 address "127.0.0.1", port 5432
db_1   | 2018-02-16 05:10:10.335 UTC [40] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
db_1   | 2018-02-16 05:10:10.335 UTC [40] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
db_1   | 2018-02-16 05:10:10.338 UTC [40] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2018-02-16 05:10:10.397 UTC [41] LOG:  database system was shut down at 2018-02-16 05:10:08 UTC
db_1   | 2018-02-16 05:10:10.420 UTC [40] LOG:  database system is ready to accept connections
db_1   |  done
db_1   | server started
db_1   | CREATE DATABASE
db_1   |
db_1   | CREATE ROLE
db_1   |
db_1   |
db_1   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1   |
db_1   | waiting for server to shut down...2018-02-16 05:10:12.424 UTC [40] LOG:  received fast shutdown request
db_1   | .2018-02-16 05:10:12.426 UTC [40] LOG:  aborting any active transactions
db_1   | 2018-02-16 05:10:12.429 UTC [40] LOG:  worker process: logical replication launcher (PID 47) exited with exit code 1
db_1   | 2018-02-16 05:10:12.430 UTC [42] LOG:  shutting down
db_1   | 2018-02-16 05:10:12.490 UTC [40] LOG:  database system is shut down
db_1   |  done
db_1   | server stopped
db_1   |
db_1   | PostgreSQL init process complete; ready for start up.
db_1   |
db_1   | 2018-02-16 05:10:12.549 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1   | 2018-02-16 05:10:12.549 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1   | 2018-02-16 05:10:12.552 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2018-02-16 05:10:12.636 UTC [67] LOG:  database system was shut down at 2018-02-16 05:10:12 UTC
db_1   | 2018-02-16 05:10:12.667 UTC [1] LOG:  database system is ready to accept connections```

[yosifkit]

@doc-phily, the output of your logs look correct. The CREATE DATABASE and CREATE ROLE output is where postgres is logging the fact that the entrypoint script created them. What problems are you seeing?

On a side note, POSTGRES_PORT isn't doing anything, since it is not a variable the entrypoint scripts looks for (docs).

[RomarQ]

Fix: nextcloud/docker#345 (comment)

[BLockhartCC]

There may be an older instance of the image. Be sure to run docker-compose build.

[wansiedler]

Yeah, the problem is not the existing images on your host, but it stems from the volume (or bind mount directory) being already initialized after your first start. The postgres user, and database creation only happens on the first start (ie, /var/lib/postgresql/data must not already contain database files).

Here is an example flow

1. declare volume for my image via compose: ./data/postgres:/var/lib/postgresql/data

2. set env in yaml: POSTGRES_PASSWORD=test

3. start containers docker-compose up -d

4. decide to add database to postgres env POSTGRES_DB=test_db or change the password

5. restart my containers docker-compose up -d

6. database test_db does not exist or password is unchanged 😢

   1. db files must be removed in order for postres entrypoint to re-initialize: docker-compose stop; sudo rm -rf ./data/postgres/
   2. now it can be restarted docker-compose up -d 😃

If you do not declare a volume mount point, then the VOLUME declared in the postgres image will apply and docker will create and manage the directory independent of the life-cycle of the container. This gets more complicated when using compose, since it will keep the volume to re-use later even when you docker-compose rm -f all of your running containers.

An example without a bind mounted volume:

1. say we do steps 2-5 above

2. we still have the same problem of the test_db database not existing (or the password not changing) since the data from the volume still exists 😞

   1. so, get rid of all containers and their volumes docker-compose rm -fv
   2. or docker-compose rm -fv postgres to get rid of just the postres service and its volumes
   3. now we can start up a new postgres container with a new empty volume docker-compose up -d 😃

You can see what volumes you have on your host by doing a docker volume ls (bind mounts will not show up in this list). There is currently no easy way to see what containers are attached to a volume, so you would have to docker inspect the container to see which ones are attached to it.

If you want to clean up all local volumes that are not attached to containers (WARNING this could delete important data 😲): docker volume ls | awk '$1 == "local" { print $2 }' | xargs --no-run-if-empty docker volume rm. On my development machine I usually precede this by removing stopped containers.

Also, in case it is not obvious, do not delete your postres data directory or volume if you have important data stored there. 😱

WORKED! =)

[escalonn]

anybody know a convenient way to have postgres run a script on startup even if the data directory is not empty? i have an idempotent migration script i just want to run any time the container starts.

[thomascrha]

anybody know a convenient way to have postgres run a script on startup even if the data directory is not empty? i have an idempotent migration script i just want to run any time the container starts.

@escalonn Did you find a way to do this ?

[escalonn]

@thomascrha nope. i don't think it exists so when i get back to that project i plan to just psql it from another container.

[djheru]

FYI, was helping a teammate with this issue today. It turns out they had a local installation of PostgreSQL running, and that was causing the issue. Once we stopped the (brew packaged) service with brew services stop postgresql, docker-compose worked as expected, setting the correct user and database name from the environment variables.

[falk-stefan]

For me adding --renew-anon-volumes did the trick (for now).

[lachezar]

--renew-anon-volumes seems to fix it for me... I also had a "." (dot) in the name if the parent folder and for some reason after renaming it to something without a dot in its name it started working. It could've been the dot, but it could have been the act of renaming it... idk 🤷