Cloud Wu wrote a piece of code back in 2012 here about a socket server that opens a TCP service, maintains an in-memory ring buffer itself, uses epoll/kevent to poll requests, and runs its own event loop to collect them. His original code is here.

I recently took some time to read through it and found it quite noteworthy, so I rewrote it and added it to my utility library. You can find the code here.

The core idea is to detect the system and wrap BSD's kqueue and Linux's epoll into a unified set of APIs for callers.

Looking at the concrete implementation, there are many interesting details. I won't go deep into the map implementation inside the ring buffer; the version I copied has a hidden linked list. During initialization, a connection array is allocated, where each element's fd stores the offset of the next element, and the last one points to -1. A head pointer is then initialized to 0.

Each time a connection is taken, the offset is fetched from the head, and the head is updated to the fd value at that offset. Each time a connection is returned, the fd of the slot being returned is set to the current head value, and the head is updated to that slot's offset. If the head points to -1, it means there are no free slots left.

However, kqueue and epoll only cover Linux and BSD, so they won't work on Windows Server. If I ever get interested, I plan to add select as well so it can run anywhere. Besides, the ring buffer itself is a great concept; once I have time to fully understand it, I'll add that too.

A usage example is attached:

C
#include <cstdio>
#include <unistd.h>

#include "argcv/nio/tcp_listen.hh"

using namespace argcv::nio;

int main(int argc, char * argv[]) {
    tcp_listen pool(9527, 200000);
    if (pool._error_no() != 0) {
        printf("pool establish failed .. %d \n", pool._error_no());
    } else {
        printf("pool established .. %d \n", pool._error_no());
        for (;;) {
            int id = pool.poll(0);
            if (id != -1) {
                printf("#### id: %d\n", id);
                co_lacus::conn &c = pool[id];
                bool st = pool.pull(id, 1);
                if (st) {
                    // printf("data:[%s] %lu \n",c.to_str().c_str(), c.to_str().length());
                    std::string s = c.to_str();
                    for (size_t i = 0; i < c.to_str().length(); i++) {
                        printf("%lu %d %c\n", i, c.to_str()[i], c.to_str()[i]);
                    }
                    // sleep(3);
                    c.write(c.to_str(), c.to_str().length());
                } else {
                    if (c.closed()) {
                        printf("is closed .. \n");
                    } else {
                        printf("unknown error ? \n status : %hhu\n", c._status());
                    }
                }
                c.flush();
            }
        }
    }
}