Using LetsEncrypt with a webserver in a docker

I have a docker which contains a webserver (caddy) that runs on ports 8080 and 8443 (instead of 80 and 443).

CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS                    PORTS                                                                                                                               NAMES
820e77fbb3b9   api-platform_caddy   "caddy run --config …"   29 minutes ago   Up 24 minutes             2019/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, 0.0.0.0:8443->443/udp, :::8443->443/tcp, :::8443->443/udp   api-platform_caddy_1
067da3c186d5   api-platform_php     "docker-entrypoint p…"   30 minutes ago   Up 24 minutes (healthy)   9000/tcp                                                                                                                            api-platform_php_1
a8522cb58871   api-platform_pwa     "docker-entrypoint.s…"   30 minutes ago   Up 24 minutes             3000/tcp                                                                                                                            api-platform_pwa_1
0e34c1583cfe   postgres:13-alpine   "docker-entrypoint.s…"   30 minutes ago   Up 24 minutes             0.0.0.0:5433->5432/tcp, :::5433->5432/tcp                                                                                           api-platform_database_1

I then have an Apache virtual host which acts as a proxy between 80 to 8080 and 443 to 8443 (80 just redirects to 443 and is not shown), and use letsencrypt for certificates.

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName my-site.net
    ProxyPreserveHost On
    ProxyRequests Off
#       <Proxy *>
#               Order deny,allow
#               Allow from all
#       </Proxy>
    SSLProxyEngine On
    SSLProxyVerify none
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerExpire Off
    SSLProxyCheckPeerName Off
    #SSLEngine On       #Not sure what this is
    ProxyPass / https://127.0.0.1:8443/
    ProxyPassReverse / https://127.0.0.1:8443/

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/my-site.net/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my-site.net/privkey.pem
</VirtualHost>
</IfModule>

I've gotten it working in the past but then I start over and I get TLS errors. I thought I did the exact steps but can't get things working. I am not sure whether it has something to do with executing "sudo certbot --apache" when the docker is up or not or whether I need to first make a traditional virtual host pointing to some directory and then execute cerbot. I also thought that maybe it was selinux related and have it disabled. Tried uninstalling and reinstalling cerbot and not sure whether doing so would have any effect.

I am going round and round and hoping for some advise how to troubleshoot and fix this. Thanks!!!

[michael@my-site api-platform]$ docker-compose up
Creating network "api-platform_default" with the default driver
Creating api-platform_database_1 ... done
Creating api-platform_pwa_1      ... done
Creating api-platform_php_1      ... done
Creating api-platform_caddy_1    ... done
Attaching to api-platform_database_1, api-platform_php_1, api-platform_pwa_1, api-platform_caddy_1
database_1  |
database_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
database_1  |
php_1       | Installing dependencies from lock file (including require-dev)
database_1  | 2021-09-25 14:22:25.743 UTC [1] LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
database_1  | 2021-09-25 14:22:25.743 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
database_1  | 2021-09-25 14:22:25.743 UTC [1] LOG:  listening on IPv6 address "::", port 5432
php_1       | Verifying lock file contents can be installed on current platform.
pwa_1       | yarn run v1.22.5
php_1       | Nothing to install, update or remove
pwa_1       | $ next dev
pwa_1       | ready - started server on 0.0.0.0:3000, url: http://localhost:3000
database_1  | 2021-09-25 14:22:25.754 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
php_1       | Generating optimized autoload files
php_1       | composer/package-versions-deprecated: Generating version class...
caddy_1     | {"level":"info","ts":1632579750.9612656,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
php_1       | composer/package-versions-deprecated: ...done generating version class
php_1       | 86 packages you are using are looking for funding.
php_1       | Use the `composer fund` command to find out more!
database_1  | 2021-09-25 14:22:25.771 UTC [22] LOG:  database system was shut down at 2021-09-25 14:22:16 UTC
php_1       |
php_1       | Run composer recipes at any time to see the status of your Symfony recipes.
php_1       |
database_1  | 2021-09-25 14:22:25.776 UTC [1] LOG:  database system is ready to accept connections
php_1       | Executing script cache:clear [OK]
caddy_1     | {"level":"warn","ts":1632579750.966388,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy_1     | {"level":"info","ts":1632579750.967962,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
caddy_1     | {"level":"info","ts":1632579750.9681666,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1     | {"level":"info","ts":1632579750.9681878,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1     | {"level":"info","ts":1632579750.9682405,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00023c620"}
caddy_1     | {"level":"info","ts":1632579750.9960306,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
caddy_1     | {"level":"info","ts":1632579750.9960551,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1     | {"level":"info","ts":1632579750.9961016,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1     | {"level":"info","ts":1632579750.9961822,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["my-site.net"]}
caddy_1     | 2021/09/25 14:22:30 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
caddy_1     | {"level":"info","ts":1632579750.9966419,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1     | {"level":"info","ts":1632579750.996675,"msg":"serving initial configuration"}
caddy_1     | {"level":"info","ts":1632579750.996811,"logger":"tls.obtain","msg":"acquiring lock","identifier":"my-site.net"}
caddy_1     | {"level":"info","ts":1632579751.0015178,"logger":"tls.obtain","msg":"lock acquired","identifier":"my-site.net"}
caddy_1     | {"level":"info","ts":1632579751.0027425,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["my-site.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy_1     | {"level":"info","ts":1632579751.0027647,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["my-site.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
pwa_1       | info  - Using webpack 5. Reason: no custom webpack configuration in next.config.js https://nextjs.org/docs/messages/webpack5
php_1       | Executing script assets:install public [OK]
php_1       |
php_1       | Waiting for database to be ready...
php_1       | The database is now ready and reachable
caddy_1     | {"level":"info","ts":1632579751.5622032,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"my-site.net","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
php_1       |
php_1       |  [OK] Already at the latest version ("DoctrineMigrations\Version20200828181711")
php_1       |
php_1       | [25-Sep-2021 14:22:31] NOTICE: fpm is running, pid 1
php_1       | [25-Sep-2021 14:22:31] NOTICE: ready to handle connections
pwa_1       | event - compiled successfully
caddy_1     | {"level":"error","ts":1632579753.0027852,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"my-site.net","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Error getting validation data"}
caddy_1     | {"level":"error","ts":1632579753.0028174,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"my-site.net","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Error getting validation data","order":"https://acme-v02.api.letsencrypt.org/acme/order/214512190/27121325760","attempt":1,"max_attempts":3}
caddy_1     | {"level":"info","ts":1632579754.2444031,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"my-site.net","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1632579754.9741173,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"my-site.net","challenge_type":"http-01","status_code":403,"problem_type":"urn:ietf:params:acme:error:unauthorized","error":"Invalid response from https://my-site.net/.well-known/acme-challenge/Bu2hYZHx9JZjzEVOW8F447Hcc_wwySx2Q0g1WOoE4g8 [12.34.56.78]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>502 Proxy Error</title>\\n</head><body>\\n<h1>Proxy Error</h1\""}
caddy_1     | {"level":"error","ts":1632579754.974146,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"my-site.net","error":"authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Invalid response from https://my-site.net/.well-known/acme-challenge/Bu2hYZHx9JZjzEVOW8F447Hcc_wwySx2Q0g1WOoE4g8 [12.34.56.78]: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>502 Proxy Error</title>\\n</head><body>\\n<h1>Proxy Error</h1\"","order":"https://acme-v02.api.letsencrypt.org/acme/order/214512190/27121341370","attempt":2,"max_attempts":3}
caddy_1     | {"level":"error","ts":1632579756.206941,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"my-site.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[my-site.net] solving challenges: my-site.net: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/214512190/27121347190) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy_1     | {"level":"warn","ts":1632579756.207511,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
php_1       | - -  25/Sep/2021:14:22:36 +0000 "GET /ping" 200
caddy_1     | {"level":"info","ts":1632579757.8707886,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"GvpqzyR0WQ1UQkPFUyguLQ"}
caddy_1     | {"level":"info","ts":1632579760.632394,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["my-site.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
caddy_1     | {"level":"info","ts":1632579760.6324198,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["my-site.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
caddy_1     | {"level":"info","ts":1632579762.0576348,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"my-site.net","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
php_1       | - -  25/Sep/2021:14:22:46 +0000 "GET /ping" 200
php_1       | - -  25/Sep/2021:14:22:57 +0000 "GET /ping" 200
^CGracefully stopping... (press Ctrl+C again to force)
Stopping api-platform_caddy_1    ... done
Stopping api-platform_php_1      ... done
Stopping api-platform_pwa_1      ... done
Stopping api-platform_database_1 ... done
[michael@my-site api-platform]$

If you're terminating tls in Apache running on the host why is caddy trying to obtain a certificate from let's encrypt? Why are you using https on your proxy? Is the certificate for caddy shared with Apache and is it in persistent storage (is it in a docker volume or bind mounted)?

Originally Posted by APB_4 If you're terminating tls in Apache running on the host why is caddy trying to obtain a certificate from let's encrypt? Why are you using https on your proxy? Is the certificate for caddy shared with Apache and is it in persistent storage (is it in a docker volume or bind mounted)?

Thanks APB_4, I just got things working again but really do not know exactly why. It appears that caddy will automatically try to obtain a certificate from Let's Encrypt, and I was also using cerbot to obtain a certificate for the Apache virtual host proxy. If either using certificates for Apache or the downstream Caddy, shouldn't they just be used on Apache? Certificate isn't shared AFAIK and when used with caddy it is persistent and I believe in a docker volume. What was frustrating was not getting more details from the logs and likely it was there but I was not looking at the right place. While immediate issue is solved, would still like to know more why it was happening and how to troubleshoot similar issues in the future.

For Caddy to run certbot successfully it probably needs to be visible on port 80 so if Apache isn't running first that will fail. The second part is that the proxy is done over https, I see that you aren't verifying the certificates so you should be able to use snakeoil but if Caddy doesn't have a key to encrypt https then it will fail. As your connection is local to the machine I would proxy using http://127.0.0.1:8080 rather than https://127.0.0.1:8443 thereby skipping caddy trying to do tls and just leaving it to Apache.

On another note, I hope you have a firewall because you have port 8080 and 8443 open on all addresses so your Apache proxy could be bypassed otherwise.