Yeah I agree.
I moved my stack from podman run to quadlets, but god damn was it frustrating to deal with them. I kept running into weird issues such as: the containers not starting every time on reboot, all containers taking like two minutes to start even without needing to download the image, the unit files not being found by systemd.
I ended up moving back to podman run, because they just worked. It is a shame, to be honest, because I would like to use quadlets.
Rootless podman caddy doesn’t need those priviliged ports, if you have your server behind a firewall device. You can map your ports on the firewall/router 80:8080 and then on the caddy container 8080:80. This way there is no need for priviliged ports and the traffic seems to go on ports 80 (and 443 the same way).