Testing is an important part of software development, which gives information about the quality of a product. It is that process by which we ensure that the product we deliver meets the expectations and requirements by testing and verifying its functionality, performance and reliability.

By testing a product we also ensure that we don’t introduce any regressions into our codebase.

There are different techniques which can be used when it comes to testing a software product.

For instance in unit testing we test the individual components of a product in order to determine and evaluate the functionality of a single isolated component of our product.

In integration testing for instance we test groups of components or modules as a whole in order to test the functionality and performance of our product in a real-world environment.

The topic on testing a software product has been thoroughly documented and discussed already, so for more information on this topic please refer to the software testing page on Wikipedia.

In this post we will see how we can record and replay HTTP interactions in Go in order to provide fast, deterministic and accurate testing of our product.

Recently I’ve been working on Gru - A simple orchestration framework written in Go, which is a personal project I work on in my spare time and I needed to get my tests done. Gru uses etcd, as the backend for coordination and management of nodes under it’s control and I was in a need to record and replay all HTTP interactions between my managed nodes and clients, so that I can create tests of these interactions.

And that’s how go-vcr was created. With go-vcr you can record and later on replay all HTTP interactions created by your code. The principle work of go-vcr involves creating a httptest.Server, which is used to mock server responses as if your client was talking to the real server.

By using dependency injection, we inject an http.Transport into our client, which routes all HTTP traffic through our own httptest.Server instance, which allows us to record and/or replay these HTTP interactions.

The Go code below shows how to record and replay the HTTP interactions that were performed against an etcd cluster.

package main

import (
        "log"
        "time"

        "github.com/dnaeon/go-vcr/recorder"

        "github.com/coreos/etcd/client"
        "golang.org/x/net/context"
)

func main() {
        // Start our recorder
        r, err := recorder.New("fixtures/etcd")
        if err != nil {
                log.Fatal(err)
        }
        defer r.Stop() // Make sure recorder is stopped once done with it

        // Create an etcd configuration using our transport
        cfg := client.Config{
                Endpoints:               []string{"http://127.0.0.1:2379"},
                HeaderTimeoutPerRequest: time.Second,
                Transport:               r.Transport, // Inject our transport!
        }

        // Create an etcd client using the above configuration
        c, err := client.New(cfg)
        if err != nil {
                log.Fatalf("Failed to create etcd client: %s", err)
        }

        // Get an example key from etcd
        etcdKey := "/foo"
        kapi := client.NewKeysAPI(c)
        resp, err := kapi.Get(context.Background(), etcdKey, nil)

        if err != nil {
                log.Fatalf("Failed to get etcd key %s: %s", etcdKey, err)
        }

        log.Printf("Successfully retrieved etcd key %s: %s", etcdKey, resp.Node.Value)
}

The first time we run this code the performed HTTP interactions will be recorded in the fixtures/etcd.yaml cassette file, which go-vcr will use during the next runs in order to replay the interactions during our tests.

All interactions are stored within the cassette file and are in YAML format. And here is an example cassette file generated by go-vcr for the above code.

---
version: 1
interactions:
- request:
    body: ""
    headers:
      Accept-Encoding:
      - gzip
      User-Agent:
      - Go-http-client/1.1
    url: http://127.0.0.1:2379/v2/keys/foo?quorum=false&recursive=false&sorted=false
    method: GET
  response:
    body: |
      {"action":"get","node":{"key":"/foo","value":"bar","modifiedIndex":523,"createdIndex":523}}
    headers:
      Content-Length:
      - "92"
      Content-Type:
      - application/json
      Date:
      - Wed, 16 Dec 2015 13:45:47 GMT
      X-Etcd-Cluster-Id:
      - 7e27652122e8b2ae
      X-Etcd-Index:
      - "526"
      X-Raft-Index:
      - "807547"
      X-Raft-Term:
      - "23"
    status: 200 OK
    code: 200

This cassette file now contains all HTTP interactions that our client has made and the respective responses it received from the server.

By replaying this cassette in future test runs we avoid the needed for having the real etcd cluster to be present, which essentially turns this into a unit test.