elmiko's notes

For some time now I’ve wanted to become more familiar with load balancers. Mainly to help my learning as I do some custom OpenShift installations, but also to fuel a dream of building my own simple container hosting workflow. Over the weekend I had a thought about using Podman and an Nginx container to experiment with balancing network traffic.

To start with I decided to create as simple an architecture as possible: 1 load balancer in round robin mode feeding traffic to 2 servers. Each application deployed as a container. The servers would be simple echo-golem deployments that reflect body text back to the sender.

Nginx configuration and network names

To make my Nginx configuration simple and also reslient to future change, I want to use well known names instead of IP addresses for my echo-golem containers. To do that in the wild I would use DNS tro server the proper IPs or if running on OpenShift, I would just let the edge routers do all the work. But here, I need to give the server containers names and I need those names to be meaningful in the network namespace.

I need to create a network that Podman will use to address the container by name. To do that I use:

podman network create echogolemnet

When I create the container for this experiment I will use the --network echogolemnet to ensure they are all placed in the same network space.

Now that I have a network namespace to keep track of my servers I will use the arbitrary names “echogolem1” and “echogolem2” for them. This allows me to create an Nginx configuration like this:

events {}
http {
    upstream echogolem {
        server echogolem1:8080;
        server echogolem2:8080;
    }

    server {
        listen 8080;

        location / {
            proxy_pass http://echogolem;
        }
    }
}

which I save in a file named nginx.conf.

One thing to note are the ports. I needed to keep these straight as I plumbed the traffic through to the servers. I could have used anything at the server listen port though, this is useful to keep in mind for moving the Nginx container around.

Starting the experiment

I need to start the echo-golem servers first otherwise Nginx will crash when it tries to connect to the upstream servers. To start them I use the following commands:

podman run --rm -d --name echogolem1 --network echogolemnet quay.io/elmiko/echo-golem:python-flask
podman run --rm -d --name echogolem2 --network echogolemnet quay.io/elmiko/echo-golem:python-flask

then check their status with a quick podman ps.

$ podman ps --filter name=echogolem*
CONTAINER ID  IMAGE                                   COMMAND               CREATED        STATUS            PORTS   NAMES
e11ebe415c59  quay.io/elmiko/echo-golem:python-flask  /bin/sh -c /usr/l...  2 minutes ago  Up 2 minutes ago          echogolem2
9f88bdf1eb2b  quay.io/elmiko/echo-golem:python-flask  /bin/sh -c /usr/l...  2 minutes ago  Up 2 minutes ago          echogolem1

It’s worth noting that I’ve started these containers with -d and --rm, so they will detach from the terminal and be removed when they end, respectively.

Now I start the Nginx container with:

podman run --rm -it -v `pwd`/nginx.conf:/etc/nginx/nginx.conf:Z -p 8080:8080 --network echogolemnet docker.io/library/nginx

this time I’m letting the container stay attached to the terminal so I can watch its output. I also attach a volume to the nginx.conf file that I created earlier, publish a port to the host on 8080, and connect the container to the echogolemnet network.

After starting, I see this output:

$ podman run --rm -it -v `pwd`/nginx.conf:/etc/nginx/nginx.cong:Z -p 8080:8080 --network echogolemnet docker.io/library/nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

Sending messages and watching traffic

Now comes the fun. I will use the HTTPie project to make some quick calls to the load balancer. If it is truly doing a round robin, then I should see the first request in one of the echo-golem’s logs and then the second request in the other’s.

Send first message

$ http localhost:8080 foo=bar
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 15
Content-Type: text/html; charset=utf-8
Date: Sun, 27 Dec 2020 17:31:50 GMT
Server: nginx/1.19.5

{
        "foo": "bar"
}

Nginx log

127.0.0.1 - - [27/Dec/2020:17:31:50 +0000] "POST / HTTP/1.1" 200 15 "-" "HTTPie/1.0.3"

echogolem1 logs

$ podman logs echogolem1
---> Running application from Python script (app.py) ...
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
10.88.2.6 - - [27/Dec/2020 17:31:50] "POST / HTTP/1.0" 200 -

echogolem2 logs

$ podman logs echogolem2
---> Running application from Python script (app.py) ...
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)

So far, so good. The request has only gone to echogolem1.

Send second message

$ http localhost:8080 foo=bar
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 15
Content-Type: text/html; charset=utf-8
Date: Sun, 27 Dec 2020 17:37:29 GMT
Server: nginx/1.19.5

{
        "foo": "bar"
}

Nginx log

127.0.0.1 - - [27/Dec/2020:17:31:50 +0000] "POST / HTTP/1.1" 200 15 "-" "HTTPie/1.0.3"
127.0.0.1 - - [27/Dec/2020:17:37:29 +0000] "POST / HTTP/1.1" 200 15 "-" "HTTPie/1.0.3"

echogolem1 logs

$ podman logs echogolem1
---> Running application from Python script (app.py) ...
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
10.88.2.6 - - [27/Dec/2020 17:31:50] "POST / HTTP/1.0" 200 -

echogolem2 logs

$ podman logs echogolem2
---> Running application from Python script (app.py) ...
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
10.88.2.6 - - [27/Dec/2020 17:37:29] "POST / HTTP/1.0" 200 -

And there we have it. Fairly definitive proof of how the round robin works.

Wrap up

I know this is a super basic intro to how this works, but I tend to learn best by taking apart the small pieces and playing with them to understand how to build bigger things. I credit that to a youth spent playing with Legos ;)

The next things I want to do with this knowledge is setup an external DNS to provide name service for the API and application endpoints for an OpenShift cluster. Then setup a load balancer in a virtual machine that will balance the traffic between the nodes I am deploying. It should be straightforward, but you never know lol.