Traefik, Docker and Cloudflare: A recipe for not having any idea how to get Real IP addresses

After much wrangling with Traefik plugins configs, and some good old trial and error, finally managed to find a way to restore original visitor IPs to my docker containers on a Cloudflare proxy + Traefik reverse proxy deployed in a docker container on a bridged network set up.

The fact that Traefik in a docker container doesn't pass on client X-forwarded-for or X-Real-IP headers for reverse proxied requests, but instead substitutes them with docker bridged network IPs, is a bit unintuitive especially when you consider that HTTP Requests are part of the TCP protocol, which runs at Layer 4 of the OSI model and not Layer 3.

But wait... Traefik is an application? Doesn't it run on Layer 7?

To make things muddier, reverse proxies in particular complicate the entire thing as they perform routing functions as well. Traefik in particular functions as a virtual router in specific instances. Throw in the fact that docker itself has networking capabilities that are either host or bridge dependent and you get the perfect recipe for a stew of wtf is going on here.

So it turns out that there are a few problems to solve.

1. Cloudflare send X-Real-IP in a CF-Connecting-IP HTTP Header instead of X-Forwarded-For or X-Real-IP because Cloudflare itself acts as a proxy. More details here.
2. Traefik is a reverse proxy, it takes in requests and forwards them. However, when you deploy Traefik in a docker container and assign it a bridge network, it takes in traffic through the docker NAT, not the host network.
3. Traefik also forwards requests, and again, in a docker container on a bridged network, it forwards the requests to containers through the docker NAT (which makes perfect sense)

So the solution required resolving each of these problems.

First by exposing the ports Traefik connects to on the host interface. More info here.

Next, by allowing all forwarded headers into our Traefik entrypoints to be forwarded to the containers by modifying the static Traefik config. (Option of being lazy and going insecure: true or defining the cloudflare IP CIDRs one by one)

Lastly, rewriting the received HTTP Header that Traefik receives with a middleware plugin (shoutout to github.com/Paxxs/traefik-get-real-ip) so that CF-Connecting-IP is now forwarded as X-Forwarded-For or X-Real-IP when the containers receive it.

tl;dr: Don't use Traefik unless you know exactly what the hell it does. Addtionally, don't use Traefik in a docker container unless you are intimately familiar with networking.

Afterthought: idk how the hell the average web developer who just builds apps is going to figure this out. This requires some understanding about networking + containerization tech. Just looking at the number of posts on the Traefik forums asking the same question is quite telling.