Building a load balancer in Go part 3

Learning Golang by building an HTTP load balancer

Introduction

This is the last part of a 3-part series where we’re building a basic HTTP load balancer in Go.

In the last part, we added health checks. The last feature, every load balancer should have is connection pools.

Announcement

I’m working on a new project called ReelCode. It’s a coding platform such as LeetCode but with real-world problems. You can learn important computer science topics while solving real-world challenges. Supported by high quality descriptions and tutorials.

Connection pools

A connection pool is an optimization technique that means storing and reusing existing connections. Instead of creating a new HTTP or TCP connection each time a client connects to your server, you reuse older, existing connections. This saves you the overhead of creating HTTP connections.

They are used in many places, including:

• Databases

• Web servers

• Message queues

• HTTP clients

• E-mail servers

• Load balancers

• And the list goes on

In our case, the connection pool is going to be a hash map. Each backend server can have X number of connections that the load balancer will reuse when needed. For example, if you have 3 backend servers the connection can be visualized like this:

Each server has a list of existing connections that can be reused when serving new requests.

This is what the ConnectionPool struct looks like:

type ConnectionPool struct {
    *Opts
    clients map[string][]*http.Client
}

A connection is an http.Client struct so I called the hashmap clients.

The type

map[string][]*http.Client

means it’s a map with string keys and the values are arrays of http.Client pointers. Keys are going to be server addresses.

You might wonder what is this *Opts thing:

type ConnectionPool struct {
    *Opts
}

As I mentioned before, there’s no inheritance in Go. The language favors enforces composition over inheritance (the good old mantra you’ve probably already heard).

What you see here with *Opts is called struct embedding. Opts is another struct that contains options for connections (you’ll see in a minute).

When we want to use a struct in another struct we have two options.

The first one is to use it as a normal field:

type ConnectionPool struct {
    Options *Opts
}

This is exactly what you think it is. ConnectionPool has a field called Options which is a type of Opts.

The second one is struct embedding:

type ConnectionPool struct {
    *Opts
}

In this case, there’s no field name because ConnectionPool will “inherit” every field from Opts. So if Opts has a field called “foo” then ConnectionPool also has a field called “foo”:

type Opts struct {
    string foo
}

type ConnectionPool struct {
    *Opts
}

cp := &ConnectionPool

// This is valid
cp.foo

But the fields of Opts can also be accessed via a field called “Opts”:

type Opts struct {
    string foo
}

type ConnectionPool struct {
    *Opts
}

cp := &ConnectionPool

// Both of them valid
cp.Opts.foo
cp.foo

So struct embedding is just syntactic sugar that makes composition feel a bit cleaner.

The real Opts struct defines two fields:

type Opts struct {
    maxConnections int
    timeout        time.Duration
}

The number of max connections for a pool and the timeout for the connections.

This is how the ConnectionPool will be used:

• Before forwarding a request to a backend server, the load balancer (LB) will request a connection for the given server

• The connection pool (CP) checks if there are available clients. If there are, it returns one of them. If there aren’t any, it creates a new one

• After forwarding the request to the backend LB will put the connection back

ConnectionPool needs two functions:

• Get

• Put

Get connection

Let’s start with Get():

func (cp *ConnectionPool) Get(server string) *http.Client {
    if clients, ok := cp.clients[server]; ok && len(clients) > 0 {
    	client := clients[len(clients)-1]
    	clients = clients[:len(clients)-1]
    	cp.clients[server] = clients

    	return client
    }

    return &http.Client{
    	Timeout: cp.timeout,
    }
}

The first line might look a bit messy. This is another version of it:

clients, ok := cp.clients[server]

if ok && len(clients) > 0 {
    // ...
}

In Go, accessing a key of a map returns two values. The value itself, and a bool which is true if the key exists and false if it doesn’t. The if statement checks if the given server exists in the pool and it has at least one client. The clients variable is an array and len() returns the length of an array.

In Go, we can declare variables in an if statement. So the two steps (accessing the key and checking it has clients) can be merged into one if statement:

if clients, ok := cp.clients[server]; ok && len(clients) > 0

If there are cached connections for the given server we need to remove and return the last one:

client := clients[len(clients)-1]
clients = clients[:len(clients)-1]
cp.clients[server] = clients

return client

Array indices work similarly to Python. For example:

clients[:3]

Will return items #0, #1, and #2. So everything before 3 (it’s an exclusive operator).

This line removes the last elements:

clients = clients[:len(clients)-1]

So the Get() function removes and returns the last client for the given server. If there are no clients it creates a new one.

Put connection

After the load balancer forwarded the request it puts back the connection to the pool.

This is quite straightforward:

func (cp *ConnectionPool) Push(server string, client *http.Client) error {
    if len(cp.clients[server]) > cp.maxConnections {
       return fmt.Errorf("connection pool limit exceeded for server '%s'", server)
    }

    cp.clients[server] = append(cp.clients[server], client)

    return nil
}

If the number of max connections is exceeded the function returns an error. Otherwise, it appends the client to the clients array.

In Go, there’s no operator such as:

$clients[] = $client

We need to use the append function that returns a new array with the new length. Don’t worry, it’s quite efficient and won’t actually create a new array each time.

That’s the basic behavior the ConnectionPool needs to implement.

Constructors and builder

As I said in a previous post, Go doesn’t have constructors. It’s not a problem since a constructor is just a regular function with a special name. So this is a constructor in Go:

func NewOpts() *Opts {
    return &Opts{
       maxConnections: 10,
       timeout:        5 * time.Second,
    }
}

This is a convention in Go. A “constructor” function starts with “New” and ends with the struct’s name.

As you can see, it uses default values but doesn’t accept arguments. This struct uses a simple builder pattern. It has two other functions:

func (opts *Opts) MaxConnections(maxConnections int) *Opts {
    opts.maxConnections = maxConnections

    return opts
}

func (opts *Opts) Timeout(timeout time.Duration) *Opts {
    opts.timeout = timeout

    return opts
}

They are setters that return the struct itself. It is also known as “fluent API.” It can be used like this:

connectionpool.NewOpts().
    Timeout(10*time.Second).
    MaxConnections(100)

It’s a nice pattern, I find it especially useful in Go with structs. Probably because Go doesn’t support default values for function arguments (a very poor decision in my opinion).

ConnectionPool also has a constructor that accepts an Opts instance and initializes an empty map:

func NewConnectionPool(opts *Opts) *ConnectionPool {
    return &ConnectionPool{
       Opts:    opts,
       clients: make(map[string][]*http.Client),
    }
}

And finally LoadBalancer also exposes a constructor:

func NewLoadBalancer(urls []string, opts *connectionpool.Opts) *LoadBalancer {
    servers := make([]*Server, len(urls))
    for i, url := range urls {
       servers[i] = &Server{
          URL:     url,
          Healthy: true,
       }
    }

    return &LoadBalancer{
       servers: servers,
       cp:      connectionpool.NewConnectionPool(opts),
    }
}

For convenience reasons, it accepts a list of strings and creates a list of servers.

In the main function you can find the following:

lb := loadbalancer.NewLoadBalancer(
    []string{"http://127.0.0.1:8000", "http://127.0.0.1:8001"},
    connectionpool.NewOpts().
       Timeout(10*time.Second).
       MaxConnections(100),
)

Config

As you can see, server addresses are hardcoded in the code. A better way would be to add a config. I’m going to use YML in this example. The config file is really simple:

port: 8080
servers:
  - http://127.0.0.1:8000
  - http://127.0.0.1:8001

I added an example config.yml to the root folder of the Git repo.

There’s a struct that maps to this exact config file:

type Config struct {
    Port    int
    Servers []string
}

There’s a package called go-yaml/yaml that can read YML files. We can use it to read and parse the config.

I created a configparser.go file with a Parse() function:

import (
    "bufio"
    "gopkg.in/yaml.v3"
    "os"
)

func Parse() (*Config, error) {
    file, err := os.Open("config.yaml")
    if err != nil {
    	return nil, err
    }

    defer file.Close()

    stat, err := file.Stat()
    if err != nil {
    	return nil, err
    }

    data := make([]byte, stat.Size())
    reader := bufio.NewReader(file)

    _, err = reader.Read(data)
    if err != nil {
    	return nil, err
    }

    config := Config{}

    err = yaml.Unmarshal(data, &config)
    if err != nil {
    	return nil, err
    }

    return &config, nil
}

First, the function opens the config file:

file, err := os.Open("config.yaml")
if err != nil {
    return nil, err
}

defer file.Close()

os is a first-party language that contains OS-specific functions such as opening a file. Once a file is opened it has to be closed at some point. It’s a perfect opportunity to use defer just as before.

The next part reads stats about the file:

stat, err := file.Stat()
if err != nil {
    return nil, err
}

stat is a struct that includes information such as the size of the file.

In order to read the content of the file, we need a buffer, a byte array that holds the content. If we know the size of the file we can create the right-sized buffer:

data := make([]byte, stat.Size())

After that, the file can be read:

reader := bufio.NewReader(file)

_, err = reader.Read(data)
if err != nil {
    return nil, err
}

bufio is another first-party package that implements buffered I/O. It’s outside of the scope of this article but in short: even if you want to read only 10 characters it will read a bigger chunk and store it internally in a buffer so the next read won’t run an I/O operation. It’s a great technique to reduce I/O operations. Every good language implements a buffered reader and writer.

reader.Read will read the file’s content into the data array:

reader.Read(data)

The final step is encoding the data to valid YAML and creating a Config struct:

config := Config{}

err = yaml.Unmarshal(data, &config)
if err != nil {
    return nil, err
}

yaml.Unmarshal comes from the 3rd party package mentioned earlier. It parses data and creates a new Config struct.

Now the main function looks like this:

func main() {
    cnf, err := config.Parse()
    if err != nil {
       panic(err)
    }

    lb := loadbalancer.NewLoadBalancer(
       cnf.Servers,
       connectionpool.NewOpts().
          Timeout(10*time.Second).
          MaxConnections(100),
    )

    lb.RunHealthCheck()

    fmt.Printf("Load balancer listening on port :%d\n", cnf.Port)

    err = http.ListenAndServe(fmt.Sprintf(":%d", cnf.Port), lb)
    if err != nil {
       panic(err)
    }
}

Everything is simple, easy to read, and easy to understand.

This simple load balancer can:

• Forward requests to the right servers

• Run health checks and skip unhealthy servers

• Use connection pools

• Read from config

Concurrency considerations

As I said earlier, we won’t go into details about concurrency in this series. That’s quite a big topic for another post.

However, this load balancer is concurrent by nature:

err = http.ListenAndServe(fmt.Sprintf(":%d", cnf.Port), lb)
if err != nil {
   panic(err)
}

ListenAndServe will run a go routine for each individual request. You can test it by:

• Write a simple script that waits for 2 seconds and returns HTTP OK

• Start it on :8000 and :8001

• Start the load balancer and configure the two servers

• Send two requests to it

If it was synchronous the first request would take 2s but the second one would take 4s to complete since it has to wait for the first one as well. This is how Nodejs works if you don’t run async stuff (for example, spin up an API endpoint that calculates prime numbers from 1 to 1 billion and then send two requests to it).

But in this case, both requests will finish in 2s because ListenAndServe runs go routines. ListenAndServe runs LoadBalancer.ServeHTTP that calculates the next server running LoadBalancer.NextServer().

This is a snippet from NestServer:

server := lb.servers[lb.idx]
lb.idx = (lb.idx + 1) % len(lb.servers)

The function accesses and changes the idx field on the LoadBalancer struct.

What happens when you run 20 go routines and each calls NextServer? Potentially, 20 threads try to read and write lb.idx. It leads to a race condition.

I didn’t show you this until now, but the LoadBalancer struct has a Mutex:

type LoadBalancer struct {
    servers []*Server
    mu      sync.Mutex
    idx     int
    cp      *connectionpool.ConnectionPool
}

And NextServer starts with these two lines:

func (lb *LoadBalancer) NextServer() (*Server, error) {
    lb.mu.Lock()
    defer lb.mu.Unlock()

    // ...
}

Mutex is a synchronization technique.

lb (which is an instance of LoadBalancer) lives in the memory. It has a memory address in the heap:

In this illustration, it lives in the first memory slot. When a thread calls

lb.mu.Lock()

This memory slot is locked. It means if another thread tries to read or write this memory address it has to wait. This memory address is locked.

When the original thread finishes its job and calls

lb.mu.Unlock()

It means that the given memory slock is unlocked so it’s open for other threads.

A mutex synchronizes the data access across threads.

ConnectionPool also uses a mutex:

type ConnectionPool struct {
    *Opts
    clients map[string][]*http.Client
    mu      sync.Mutex
}

func (cp *ConnectionPool) Get(server string) *http.Client {
    cp.mu.Lock()
    defer cp.mu.Unlock()
}

func (cp *ConnectionPool) Push(server string, client *http.Client) error {
    cp.mu.Lock()
    defer cp.mu.Unlock()
}

As I said before, we’re going to talk about concurrency in a different post, because it’s quite a complicated topic.