SpiderBB - Go-State-Store

In the last posts I’ve shown how I setup the motion detection on my Raspberry Pi. In this post, I will show you the state store of SpiderBB.

The state store is written in go which has three reasons:

1. Go is advised as “makes it easy to build simple, reliable, and efficient software.”
2. Go supports REST, WebSocket, and concurrency straightforward
3. I wanted to learn some Go basics ;)

State store Idea

The idea of the state store is that processes from localhost can manipulate key-value pairs. Processes that access state stores from elsewhere only can read and subscribe to values within the store. Every subscripted process will receive updated information via a WebSocket channel. To reduce network traffic, the state store collects changes and sends them as a group.

In the schematic above, we see that the state store has two APIs and three internal blocks. The APIs are REST, and the listeners receive updates via WebSocket.

The APIs

The API is split into two APIs a manipulation and a client API.

Manipulation is very quickly done via the Rest endpoint http://localhost:8081/value/{key} where key is the key within the store (1). The HTTPS adjectives select the CRUD operation.

• GET tries to respond with the currently stored value for the key. If for the requested key no value is present, the result is HTTP status code BadRequest.
• PUT and POST expect the key’s new value as a query parameter with the key value, response with BadRequest if the value is nil.
• DELETE tries to remove the value for a key from the datastore, answer with BadRequest if the value is nil. The manipulation API is hardcoded only to accept requests from localhost.

Processes can use the client API to react to state changes within the datastore. To inform the clients go-state-store uses Websockets as defined by RFC 6455. The WebSocket endpoint listens to ws://<IP>:8080/updates. The endpoint is accessible from everywhere. The endpoint sends every 5s a message containing the changes keys (4). Clients can read data from the store via the GET endpoint http://<IP>:8080/get/{key} where key is the requested string key. The endpoint answers with the BadRequest code if the key is not present (5).

The Process

We will follow the process in the schematics and see how it is implemented.

A local process calls the manipulation API and sets a value (1). Let’s say we call POST http://localhost:8081/value/x?value=2. The registered handler extracts the value 2 and the key x and sets the value into the data store (2). The handler function puts the key x into the channel to the event cache, which puts it into the queue.

Every 5 seconds, the event cache copies the queued key and clears the queue. At this point, the cache is ready to queue new keys. Now the copy is sent to the event publisher via a second channel (3).

The events publisher parses all incoming data into JSON and sends it via web socket (5) to the listeners. Every listener can now decide if a key of interest has changed and query the value via the GET http://<IP>:8080/get/<key>.

Some implementation details

The complete source code is publicly available on https://github.com/MBoegers/go-state-store. I will show some essential parts here.

Send data across goroutines

A goroutine is a lightweight thread for concurrent execution. Goroutines can send and receive data via channels. Go requires the payload to be a primitive type or a struct.

1
2
3
4
5
6
7
8
9
10
11
12
13

func SendA42(ch chan int) {
    ch <- 42 // 4
}

func main() {
    channel := make(chan int) // 1
    
    go SendA42AndClose(channel) // 2
 
    var v int
    v = <-ch // 3
    fmt.Println(v) // prints 42 to console, hurray!
}

This small go program shows the basic use of channels. It creates a channel (1) and passes it to a goroutine (2). Now the execution proceeds and waits for data to arrive via the channel (3). The goroutine sends the integer value 42 to the channel and terminates (4).

By passing the channel to two different goroutines, like the data store and the event store, these two can communicate without knowing each other. So they are loosely coupled and independent, the same hold for the state store and the event publisher.

Handling multiple channels

In the events store I have a timer to trigger the cyclic event publication. The time and the event store also interact via a channel and so I have to handle two channels. Luckily Go has a select expression which can handle them.

1
2
3
4
5
6
7
8
9
10

func handleMessages() {
    for { // 1
        select {
            case <-sendSchedule.C: // 2
                sendEvents()
            case event := <-newChan: // 3
                addEvent(event)
        }
    }
}

First of all, we are in the main goroutine of the event store, which is spawned a startup and loops forever (1). In every iteration, we wait for a message on a channel. If the timer fires the message containing no data, we don’t extract data but call sendEvents() to trigger the notifications. On the other hand, if a new event occurs (3), we extract the key and add it to the queue. It is straightforward to handle multiple channels with go.

Summary

In this post, I’ve presented the main idea behind the go-state-store. This small go program handles concurrent manipulation, preserves the order, and sends events on time to the listener. I’ve shown how easy it is to develop loosely coupled small blocks with go. One could say they are picoservices :-D. In this series’s next and probably last post, I’ll show how motion and the go-state-store care are tied together to get notifications whenever motion is detected.