Reading and Writing Redis Protocol in Go

In this post, I outline a simple, easy to understand implementation for two components of a Redis client in Go as a way of understanding how the Redis protocol works and what makes it great.

If you’re looking for a full-featured, production-ready Redis client in Go, we recommend taking a look at Gary Burd’s redigo library.

Before we get started, be sure you read our gentle introduction to the Redis protocol - it covers the basics of the protocol that you’ll need to understand for this guide.

A RESP command writer in Go

For our hypothetical Redis client, there’s only one kind of object that we’ll need to write: an array of bulk strings for sending commands to Redis. Here is a simple implementation of a command-to-RESP writer:

package redis

import (
  "bufio"
  "io"
  "strconv"     // for converting integers to strings
)

var (
  arrayPrefixSlice      = []byte{'*'}
  bulkStringPrefixSlice = []byte{'$'}
  lineEndingSlice       = []byte{'\r', '\n'}
)

type RESPWriter struct {
  *bufio.Writer
}

func NewRESPWriter(writer io.Writer) *RESPWriter {
  return &RESPWriter{
    Writer: bufio.NewWriter(writer),
  }
}

func (w *RESPWriter) WriteCommand(args ...string) (err error) {
  // Write the array prefix and the number of arguments in the array.
  w.Write(arrayPrefixSlice)
  w.WriteString(strconv.Itoa(len(args)))
  w.Write(lineEndingSlice)

  // Write a bulk string for each argument.
  for _, arg := range args {
    w.Write(bulkStringPrefixSlice)
    w.WriteString(strconv.Itoa(len(arg)))
    w.Write(lineEndingSlice)
    w.WriteString(arg)
    w.Write(lineEndingSlice)
  }

  return w.Flush()
}

Rather than writing to a net.Conn object, RESPWriter writes to a io.Writer object. This allows us to test our parser without tightly coupling to the net stack. We simply test the network protocol the way we would any other io.

For example, we can pass it a bytes.Buffer to inspect the final RESP:

var buf bytes.Buffer
writer := NewRESPWriter(&buf)
writer.WriteCommand("GET", "foo")
buf.Bytes() // *2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n

A simple RESP reader in Go

After sending a command to Redis with RESPWriter, our client would use RESPReader to read from the TCP connection until it has received a full RESP reply. To start with, we’ll need a few packages to handle buffering and parsing the incoming data:

package redis

import (
  "bufio"
  "bytes"
  "errors"
  "io"
  "strconv"
)

And we’ll use a few variables and constants to make our code a little easier to read:

const (
  SIMPLE_STRING = '+'
  BULK_STRING   = '$'
  INTEGER       = ':'
  ARRAY         = '*'
  ERROR         = '-'
)

var (
  ErrInvalidSyntax = errors.New("resp: invalid syntax")
)

Like RESPWriter, RESPReader doesn’t care about the implementation details of the object that it’s reading RESP from. All it needs the ability to read bytes until it has read a full RESP object. In this case, it needs an io.Reader, which it wraps with a bufio.Reader to handle the buffering of the incoming data.

Our object and initializer are simple:

type RESPReader struct {
  *bufio.Reader
}

func NewReader(reader io.Reader) *RESPReader {
  return &RESPReader{
    Reader: bufio.NewReaderSize(reader, 32*1024),
  }
}

The buffer size for bufio.Reader is just a guess during development. In an actual client, you’d want to make its size configurable and perhaps test to find the optimal size. 32KB will work fine for development.

RESPReader has only one method: ReadObject(), which returns a byte slice containing a full RESP object on each call. It will pass back any errors encountered from io.Reader, and will also return errors when it encounters any invalid RESP syntax.

The prefix nature of RESP means we only need to read the first byte to decide how to handle the following bytes. However, because we’ll always need to read at least the first full line (i.e. up until the first \r\n), we can start by reading the whole first line:

func (r *RESPReader) ReadObject() ([]byte, error) {
  line, err := r.readLine()
  if err != nil {
    return nil, err
  }

  switch line[0] {
  case SIMPLE_STRING, INTEGER, ERROR:
    return line, nil
  case BULK_STRING:
    return r.readBulkString(line)
  case ARRAY:
    return r.readArray(line)
  default:
    return nil, ErrInvalidSyntax
  }
}

When the line that we read has a simple string, integer, or error prefix, we return the full line as the received RESP object because those object types are contained entirely within one line.

In readLine(), we read up until the first occurrence of \n and then check to make sure that it was preceded by a \r before returning the line as a byte slice:

func (r *RESPReader) readLine() (line []byte, err error) {
  line, err = r.ReadSlice('\n')
  if err != nil {
    return nil, err
  }

  if len(line) > 1 && line[len(line)-2] == '\r' {
    return line, nil
  } else {
    // Line was too short or \n wasn't preceded by \r.
    return nil, ErrInvalidSyntax
  }
}

In readBulkString() we parse the length specification for the bulk string to know how many bytes we need to read. Once we do, we read that count of bytes and the \r\n line terminator:

func (r *RESPReader) readBulkString(line []byte) ([]byte, error) {
  count, err := r.getCount(line)
  if err != nil {
    return nil, err
  }
  if count == -1 {
    return line, nil
  }

  buf := make([]byte, len(line)+count+2)
  copy(buf, line)
  _, err = r.Read(buf[len(line):])
  if err != nil {
    return nil, err
  }

  return buf, nil
}

I’ve pulled getCount() out to a separate method because the length specification is also used for arrays:

func (r *RESPReader) getCount(line []byte) (int, error) {
  end := bytes.IndexByte(line, '\r')
  return strconv.Atoi(string(line[1:end]))
}

To handle arrays, we get the number of array elements, and then call ReadObject() recursively, adding the resulting objects to our current RESP buffer:

func (r *RESPReader) readArray(line []byte) ([]byte, error) {
  // Get number of array elements.
  count, err := r.getCount(line)
  if err != nil {
    return nil, err
  }

  // Read `count` number of RESP objects in the array.
  for i := 0; i < count; i++ {
    buf, err := r.ReadObject()
    if err != nil {
      return nil, err
    }
    line = append(line, buf...)
  }

  return line, nil
}

Wrapping up

The above hundred lines are all that’s needed to read any RESP object from Redis. However, there are a number of missing pieces we’d need to implement before using this library in a production environment:

• The ability to extract actual values from the RESP. RESPReader currently only returns the full RESP response, it does not, for example, return a string from a bulk string response. However, implementing this would be easy.
• RESPReader needs better syntax error handling.

This code is also entirely unoptimized and does more allocations and copies than it needs to. For example, the readArray() method: for each object in the array, we read in the object and then copies it to our local buffer.

If you’re interested in learning how to implement these pieces, I recommend looking at how popular libraries like hiredis or redigo implement them.

Ready to get started with a production-ready Redis host backed by excellent support? Get started now with your own RedisGreen server.

Last updated 30 Apr 2015. Originally posted 16 May 2014 by Tyson Tate