docker-compose up: PermissionError: [Errno 13] Permission denied: './.env'
Shmorgel opened this issue · comments
Hello,
when I entered the "docker" directory and entered the command 'docker-compose up' I receive following error message:
$ docker-compose up
Traceback (most recent call last):
File "/snap/docker/1779/bin/docker-compose", line 33, in <module>
sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
File "/snap/docker/1779/lib/python3.6/site-packages/compose/cli/main.py", line 81, in main
command_func()
File "/snap/docker/1779/lib/python3.6/site-packages/compose/cli/main.py", line 200, in perform_command
project = project_from_options('.', options)
File "/snap/docker/1779/lib/python3.6/site-packages/compose/cli/command.py", line 40, in project_from_options
environment = Environment.from_env_file(override_dir or project_dir, environment_file)
File "/snap/docker/1779/lib/python3.6/site-packages/compose/config/environment.py", line 67, in from_env_file
instance = _initialize()
File "/snap/docker/1779/lib/python3.6/site-packages/compose/config/environment.py", line 62, in _initialize
return cls(env_vars_from_file(env_file_path))
File "/snap/docker/1779/lib/python3.6/site-packages/compose/config/environment.py", line 38, in env_vars_from_file
env = dotenv.dotenv_values(dotenv_path=filename, encoding='utf-8-sig', interpolate=interpolate)
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 363, in dotenv_values
encoding=encoding,
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 74, in dict
self._dict = OrderedDict(resolve_variables(raw_values, override=self.override))
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 222, in resolve_variables
for (name, value) in values:
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 81, in parse
with self._get_stream() as stream:
File "/snap/docker/1779/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 54, in _get_stream
with io.open(self.dotenv_path, encoding=self.encoding) as stream:
PermissionError: [Errno 13] Permission denied: './.env'
System is Kubuntu 20.04
Would you know how to fix it? I tried
$ sudo chmod 777 /var/run/docker.sock
with no avail.
I'm sorry I am new to Docker and the Frameworks etc used by this example which is why I am interested in your probably great example. I would really love to test it and learn from it. So help would be highly appreciated.
Thank you very much!
Hi @Shmorgel ,
It looks like you don't have permission to read the ".env" file inside the "docker" folder. Can you try executing "docker-compose up" with sudo? Or "sudo chmod 777 .env" inside the "docker" folder.
If that doesn't help, take a look at this: https://stackoverflow.com/questions/48957195/how-to-fix-docker-got-permission-denied-issue
Hello hdimitrieski,
I really appreciate your help!
Sweet lord Jebus, why does the devil in the details always seem to prefer me to pick up on.
I followed your advice and neither running docker-compose with sudo nor changing the rights of .env to 777 nor putting myself in the usergroup docker seems to change anything. Still the same error message !
Hi @Shmorgel,
I'm not sure what might be the problem. But one last thing that comes to my mind is reinstalling "docker-compose", and running the docker service with "sudo".
If that doesn't work, just copy everything from the ".env" file in the "docker-compose" configuration, and delete it. Basically, you just have to replace the variables in the docker-compose files with the values in .env. Ex. replace ${MONGO_USER} with user in all docker-compose files. :)
Hey hdimitrieski !
Thank you very much for your friendly and fast support ! Upvote!
I will try this and maybe start a thread on stackoverflow. If that doesn't help I will try your last idea as "matter of last resort"
Best wishes !
Shmorgel
No problem @Shmorgel :)
I think that removing the .env will solve your issue, but it would be better if you manage to fix it without having to remove it.
Dear hdimitrieski,
I have uninstalled docker by this receipt:
https://stackoverflow.com/questions/61876116/how-to-remove-a-snap-application-docker-completely
But obviously there is still another installation as I witnessed following these instructions to install the docker-ce community edition:
https://www.cherryservers.com/blog/how-to-install-and-start-using-docker-on-ubuntu-20-04
I tried sudo systemctl status docker before sudo apt install docker-ce and it showed a running docker daemon.
$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-07-31 19:27:27 CEST; 2 days ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2746 (dockerd)
Tasks: 14
Memory: 109.9M
CGroup: /system.slice/docker.service
└─2746 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
But this installation seems to be broken because when I run docker run hello-world I get the message
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
So I first ask myself:
First: Which deinstallation strategy to follow?
Second: What is the right way to install Docker if I don't want Docker Desktop ?
I would like to ask you if you would be so friendly to tell me
- Should I follow this deinstallation strategy (adding the installed docker-ce to the list to be uninstalled):
https://docs.docker.com/engine/install/ubuntu/#uninstall-old-versions
or - rather purge with this one
https://docs.docker.com/engine/install/ubuntu/#uninstall-docker-engine
There seems to be a caveat between the two because the first list of packages to be removed is:
sudo apt-get remove docker docker-engine docker.io containerd runc
whereas the second:
sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-compose-plugin
If following either strategy, should I add those lists together?
- Should I follow neither deinstallation strategy but another?
And then if I have uninstalled docker, what are the correct packages to install and how to add docker-compose to the list of binaries? I have read docker-ce, docker.io ? Or first docker-engine ? I can haz halp with all these different packages plox? :)
Hi @Shmorgel,
I use Colima as a container runtime instead of Docker. It's pretty straightforward to set up. You might want to give it a try.
When it comes to the deinstallation, I'd say It depends on what you have installed. As I can see, the first one is for old docker versions, and the second one is for the newer versions. So you can try them both. :)
And for the installation, I use macOS. I haven't used docker on Linux, so I can't help you there... I'd follow this guide: https://docs.docker.com/engine/install/ubuntu/.
Hi @Shmorgel,
Can I close this issue? Did you find a solution to your problem?
Dear Hristijan ,
I finally managed to get $ docker compose up to run after some attempts to purge and install docker packages following the official docs.
So I ran $ docker compose up in $PROJECT_FOLDER/docker and it successfully downloaded images and magical docker stuff I am not yet knowledgeable about. I could just confirm that it worked by witnessing how it put my laptop under heavy load :)
Then I built the services via $ ./gradlew build, following to that I imported it into IntelliJ which liked to build it again obviously. Not sure if that causes any trouble.
However, now I wanted to start the services in docker containers and, following the project instructions here on github issued docker compose -f docker-compose.infra.yml up --build
I didn't want to make another ticket for this. I hope it is ok if I append this to my questions above, since it's essentially not bugs in your code but questions on my behalf and my ineperiencedness which I hope to alleviate by making myself accostumed with that great demo of yours !
Anyway, this is the output:
max@max-Precision-M4800:/media/max/Informatik/codebases/gradle microservice github examples/hdimitrieski/e-shop/docker$ docker compose -f docker-compose.infra.yml up --build
[+] Building 34.3s (17/17) FINISHED
=> [docker_gateway internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 524B 0.0s
=> [docker_discovery internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 656B 0.0s
=> [docker_config internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 516B 0.0s
=> [docker_gateway internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [docker_config internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [docker_discovery internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [docker_discovery internal] load metadata for docker.io/library/openjdk:17.0.2-slim 1.8s
=> [docker_gateway 1/2] FROM docker.io/library/openjdk:17.0.2-slim@sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f97 28.2s
=> => resolve docker.io/library/openjdk:17.0.2-slim@sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f9774 0.0s
=> => sha256:aaa3b3cb27e3e520b8f116863d0580c438ed55ecfa0bc126b41f68c3f62f9774 547B / 547B 0.0s
=> => sha256:779635c0c3d23cc8dbab2d8c1ee4cf2a9202e198dfc8f4c0b279824d9b8e0f22 953B / 953B 0.0s
=> => sha256:37cb44321d0423bc57266a3bff658daf00478e4cdf2d3b8091f785310534256d 4.80kB / 4.80kB 0.0s
=> => sha256:1fe172e4850f03bb45d41a20174112bc119fbfec42a650edbbd8491aee32e3c3 31.38MB / 31.38MB 20.0s
=> => sha256:44d3aa8d076675d49d85180b0ced9daef210fe4fdff4bdbb422b9cf384e591d0 1.58MB / 1.58MB 1.1s
=> => sha256:6ce99fdf16e86bd02f6ad66a0e1334878528b5a4b5487850a76e0c08a7a27d56 187.90MB / 187.90MB 26.1s
=> => extracting sha256:1fe172e4850f03bb45d41a20174112bc119fbfec42a650edbbd8491aee32e3c3 0.7s
=> => extracting sha256:44d3aa8d076675d49d85180b0ced9daef210fe4fdff4bdbb422b9cf384e591d0 0.1s
=> => extracting sha256:6ce99fdf16e86bd02f6ad66a0e1334878528b5a4b5487850a76e0c08a7a27d56 1.7s
=> [docker_config internal] load build context 1.8s
=> => transferring context: 61.75MB 1.7s
=> [docker_gateway internal] load build context 2.1s
=> => transferring context: 76.33MB 2.0s
=> [docker_discovery internal] load build context 2.1s
=> => transferring context: 73.79MB 2.0s
=> [docker_gateway 2/2] ADD ./build/libs/gateway.jar gateway.jar 1.7s
=> [docker_config 2/2] ADD ./build/libs/config.jar config.jar 1.8s
=> [docker_discovery 2/4] ADD ./wait-for-it.sh wait-for-it.sh 1.7s
=> [docker_discovery 3/4] RUN chmod +x wait-for-it.sh 1.0s
=> [docker_discovery] exporting to image 1.7s
=> => exporting layers 0.7s
=> => writing image sha256:09ec1459eb6a76cef74bc39c40c9211db93dcbf59f0ec506bf27d8034cdcafe7 0.0s
=> => naming to docker.io/library/docker_config 0.0s
=> => writing image sha256:73dfcab45e463baedf65709143904c7fdc941e99e3439ae51cb927b1e0ac54d9 0.0s
=> => naming to docker.io/library/docker_gateway 0.0s
=> => writing image sha256:9210d66722fa7c9d3d81d3941a327e86007432ae654a42f71fb5eeb29d799a46 0.0s
=> => naming to docker.io/library/docker_discovery 0.0s
=> [docker_discovery 4/4] ADD ./build/libs/discovery.jar discovery.jar 0.6s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
WARN[0035] Found orphan containers ([docker-kafka-1 docker-authorization-service-1 docker-zookeeper-1 docker-mongo-1 docker-portainer-1 docker-postgres-1 docker-redis-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[+] Running 3/3
⠿ Container docker-config-1 Created 0.1s
⠿ Container docker-discovery-1 Created 0.1s
⠿ Container docker-gateway-1 Created 0.1s
Attaching to docker-config-1, docker-discovery-1, docker-gateway-1
Error response from daemon: driver failed programming external connectivity on endpoint docker-config-1 (4aa42dbec782dcb5b03e659835a8dcd1e8951cfa827170aad25b21e758f805da): Error starting userland proxy: listen tcp4 0.0.0.0:8888: bind: address already in use
I guess I receive the error messages because of the containers already built before by $ docker compose up ?
Can I remove them via $ docker compose down before issuing the above command again? Would this even help? And if so, would this undo everything in .env , like, removing volumes etc. ?
Thank you so much for your help ("big heart for noobs" ^^) !
In my stackoverflow threads I didn't even receive an answer. I'd be lost.
Yours sincerly
Maximilian
Hi @Shmorgel,
The config service container binds to port 8888.
You either have some other process that's using port 8888 on your machine (you can check that with sudo netstat -PNA | grep 8888. Or, you have the same container running at some other instance (You can use docker ps to check the running and stopped containers.)
In the first case, you can stop the process on your machine, and in the second, you can remove or stop the containers.
I hope this helps, but let me know if you need anything else. :)
Hello,
well, I have stopped and removed the containers but the key point is that always when I run docker compose -f docker-compose.infra.yml up --build I receive the last line as error message for docker-config-1:
Error response from daemon: driver failed programming external connectivity on endpoint docker-config-1 (4aa42dbec782dcb5b03e659835a8dcd1e8951cfa827170aad25b21e758f805da): Error starting userland proxy: listen tcp4 0.0.0.0:8888: bind: address already in use
Everything else seems to be ok. ! Now the question is, if it speaks of "driver failed programming..." but I don't know enough about networks and endpoints yet. May it be that it is not my installation? Because otherwise it runs flawless.
Best Wishes
Maximilian
Dear Hristijan,
I am sorry, I haven't seen that you have already answered! Normally I receive an email notification and then I immediately look and when I updated last time either I haven't seen it (need for new glasses?) or you in fact hadn't answered then.
Well, thanks for your advice with netstat. On Kubuntu/Linux it obviously has some different flags/options.
It only accepts the -N option but not -P nor A or rather A only with some addition -A <af> but I dont' know if you would've meant that.
Alas, the port 8888 is really occupied.
ax@max-Precision-M4800:~$ sudo netstat -N | grep 8888
tcp 0 0 localhost:8888 localhost:59666 VERBUNDEN
tcp 0 0 localhost:8888 localhost:57440 VERBUNDEN
tcp 0 0 localhost:51760 localhost:8888 VERBUNDEN
tcp 0 0 localhost:57440 localhost:8888 VERBUNDEN
tcp 0 0 localhost:59666 localhost:8888 VERBUNDEN
tcp 0 0 localhost:8888 localhost:51760 VERBUNDE
I'm sorry, German keyboard and language settings. I should switch to English because it makes it easier to search for error messages and problems. "VERBUNDEN" = "CONNECTED"
What do the - P and A do, so I can look for similar options in the linux version ? Do they show the names of the services that occupy these ports?
So, while waiting for your answer, I could try to set config to another port, right?
Thank you so much for your patience again.
Best Wishes
Maximilian
Ok, I first changed the port in docker-compose.infra.yml from '8888:8888' to '8761:8761' and then a lot of console output happened. But obviously 'wait-for-it.sh' looked for the config service on '8888'. So do the other two services.
In ../infrastructure/config/Dockerfile I changed EXPOSE from '8888' to '8849'
And in ../infrastructure/gateway/Dockerfile I changed it the port as argument for the 'wait-for-it.sh' script.
# Run the application
ENTRYPOINT ["/bin/sh", "-c", "./wait-for-it.sh config:8849/actuator/health -t 5 -- java -jar discovery.jar"]
Now I get the information about Spring being started from the containers, but the script still complains about timeout after 5s and the other services are still looking for config under port '8888'
: (
Stupid me! D'oh!
I don't know why I ask you about the options of netstat. I'm sorry. I haven't slept all night and I'm still having aches. (Not dr*nk! ^^)
I used -pN and now I know the delinquent service. It was Jupyter. Ran through virtualenv which is obviously a Python flavour kind of flexible, 'slash not-a-container'. Will have to look to configure those to run on another port. Now I will try to run docker compose with my changes undone and port 8888 being free.
Stupid me! D'oh!
I don't know why I ask you about the options of netstat. I'm sorry. I haven't slept all night and I'm still having aches. (Not dr*nk! ^^)
I used -pN and now I know the delinquent service. It was Jupyter. Ran through virtualenv which is obviously a Python flavour kind of flexible, 'slash not-a-container'. Will have to look to configure those to run on another port. Now I will try to run docker compose with my changes undone and port 8888 being free.
Your description doesn't mention that one needs Kafka daemon running but the error messages mostly complain about Kafka not being available. Then also about a ton of other stuff. But if the problem is that I have Kafka not (correctly) installed, then I should tackle this first, right. I have downloaded the already compiled package from the official site and put it to /usr/local/kafka or something and made it available in ~/.profiles I think. But that's long ago already. Guess the proper way to install it is another one anyway... If, well if I have to manually install it and your scripts don't take care of it. But I think they don't ?
Hi @Shmorgel,
Yes, just run the config service on port 8888, so you don't need to change anything else.
You don't need Kafka installed on your machine. You can see that in docker/docker-compose.yml a service for Kafka is defined. So first use docker-compose up to start the services for Kafka and all other databases, then docker-compose -f docker-compose.infra.yml up --build for the infrastructure services, and after that run docker-compose -f docker-compose.services.yml up --build to start the domain services.
Dear @hdimitrieski
I don't know what is wrong. Yes I read the documentation wrong. I thought either docker compose up or docker compose -f docker-compose.infra.yml and docker-compose.services.yml up --build
But even if I run docker compose up I get about the same error messages as if I hadn't issued the command ! Although Kafka seems to run and I see output of it about leadership etc. But also the output of these services doesn't seem calm enough. It has warnings here and there.
Foremost, I am still getting the same Kafka errors as before by the second command and by the services config, gateway, discovery...
docker-config-1 | 2022-08-08 14:48:01.736 WARN [,,] 1 --- [| adminclient-3] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-3] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
Now everything should actually run fine but nah. It's frustrating. Don't know what to do any more.
Best wishes !
Seems like my only option is to make small experiments with a Kafka Dockerfile/Container first and so on, huh? Damn. What the hell is wrong.
May I ask one further thing? Is it possible to iteratively develop the domain services of a microservice and automagically start them in containers? If not, what is the protocol? Run Images of Kafka, of Databases etc. in containers and while developing the Java classes or running them set for instance the database or CQRS server data, probably in .yml files in the directory structure (?) to those containers available as localhost:1234 with credentials "userword" und "passname" ?
Or is there a way to create containers in such a way that they "pull in" or "run" code from a certain directory ? That would be best. Developing the domain services in their appropriate containers without having to actually leave IntelliJ or whatever IDE. Maybe theres plugins for Gradle that do that from inside IntelliJ if you push "build" or "run" ?
Thanks !
@Shmorgel Just ignore the warnings in the config service, you'll still be able to run the services and the clients. I'll check what's going on there later.
@Shmorgel To develop individual domain/infrastructure services you don't need to use docker. You can just start all infrastructure components(Kafka, Postgres, Mongo,...) because the services require those components, and then start the services that you need manually as described in the docs.
@Shmorgel I fixed the Kafka connection warning. After updating the project, remove or stop the running containers on your machine, build the project ./gradlew build, and start the containers again: docker-compose up, docker-compose -f docker-compose.infra.yml up --build, docker-compose -f docker-compose.services.yml up --build.
Hello @hdimitrieski !!!
Thank you for updating your code. It seems to me that things run better now. But there is, for instance, still the "error" (?)
wait-for-it.sh: timeout occurred after waiting 5 seconds for config:8888/actuator/health
When issuing docker compose -f docker-compose.infra.yml up --build
docker-gateway-1 seems to get a connection to Kafka now but it spawns errors and warnings, errors like
docker-gateway-1 | 2022-08-09 08:49:19.691 WARN [api-gateway,,] 1 --- [nfoReplicator-0] eactorLoadBalancerExchangeFilterFunction : LoadBalancer does not contain an instance for the service catalog-query
docker-gateway-1 | 2022-08-09 08:49:19.691 WARN [api-gateway,,] 1 --- [nfoReplicator-0] o.s.c.l.core.RoundRobinLoadBalancer : No servers available for service: catalog-query
Thank you
Looking forward for your answer
Best wishes!
Maximilian/Shmorgel
@Shmorgel Those are not errors actually. Those are only warnings that the gateway service cannot connect to some domain services like catalog-query, order-processing, basket, etc. because they are not available. After running docker-compose -f docker-compose.services.yml up --build those services will become available to the gateway service, so it would be able to establish connections.
@hdimitrieski
I have a problem big time.
When I ran docker-compose -f docker-compose.services.yml up --build my system was totally overburdened.
I had to shut down and boot up again, but when I tried to issue docker compose stop it told me that it wouldn't find a configuration. I have the codebase of e-shop on an USB stick. To this directory it was that I cloned the repository and pulled your fix. And of course it was in this directory on the USB stick, where I ran the docker compose commands to build the projects, and to build and run the docker containers. Shouldn't docker compose, as docker, write those things into my home directory? Not right into place? Could it be that the configuration file was written onto the USB and that it got lost when I had to choke off the computer?
The system was really pretty much frozen and additionally like every two seconds system notifications warned me that the containers didn't respond. I could only end it by sudo systemctl stop docker. Thank God then the CPU load and ram load instantaneously went down.
What do you estimate how much gig RAM all of the packages take?
I have 32 gig ram but I must admit I probably am a case of abuse of session saving. I have many browser tabs and windows with webpages. The reason is that I seek information, for instance compiler writing, microservices, maths, and normally I like to archive the URLs into text files rather than putting them into the bookmarks. When I can really classify and integrate them into a scheme, into the right files. But so it comes that I do not immediately have enough overview over a new field of matter such that it takes some time before I can do so. But then comes a new topic around the corner or develops out of the things opened last such that things become really messy. I have to close many firefox windows that strain the performance of my system before I can carry on now, Ugh! I will write a plugin or addon for firefox to save those URLs for me ! Impossible to do this in feasible time by hand. So first I must look up how to do that.
In the meantime, do you know an answer to my question whether the docker compose would really write important information of the containers created and run right into place on the USB stick and not into my $HOME folder ? When I ran like docker ps docker, not compose, knew about the containers. If theres no way with compose then I will have to shut the containers down through just docker.
Ok, see you asap :)
@Shmorgel To stop the running containers, you have to execute docker-compose -f docker-compose.yml -f docker-compose.infra.yml -f docker-compose.services.yml down inside the docker folder. Most likely it can't find the configuration because you are trying to execute the command outside the docker folder.
32GB of RAM is more than enough to run this demo. I have 16 GB, and it works very well on my machine. You just need to allow docker to use around 8GB of RAM. Also, you don't need to use docker to run the services on your local machine because it takes too much memory. Just run docker-compose up inside the docker folder, and then run the services manually.
No, docker won't write anything on the USB stick.
Under Kubuntu there is a bug with session management and usb devices. My USB stick was mounted with a different label after I re-booted but there is a copy of the e-shop and docker directory still accessible under the old moniker and hence directory name. But this copy is probably erroneous, I don't know how this even happens. But my plan is: 1 write the firefox plugin, 2) reboot, remove usb stick, delete both mount points 3) insert usb stick so it hopefully creates the correct mountpoint with correct label again and then [because of open files in kate i don't want to close in order to unmount the stick. If I remove it when the pc is off I can remove it, turn it on again, then do Nr. 3 on my list and reinsert the stick] 4) If I have closed a significant number of firefox instances retry to close them through docker compose.
@Shmorgel I'm closing this issue. If you have any further questions, please open a discussion. :)