Chatter3 engineering notes

Note 3 · Cloudflare Workers · WebSockets

A Durable Object that forgot one of the two people

The call looked perfect on both phones: an offer, an answer, ICE candidates going out. None of them arrived. One side of the conversation was sending into a room the other side had left.

Chatter3 signalling server · one Durable Object per call session

The symptom

Chatter3 runs its signalling server on Cloudflare Workers: one Durable Object per call session, holding the two WebSockets of the two people. The symptom was a call that failed after it had been set up. The offer was created, the answer was created and received, the candidates were sent — and the connection never came up. The failure was not random: it happened when the second person had been waiting a while before the first one arrived.

Waiting a while is the clue. Waiting is what an idle Durable Object does: the platform takes it out of memory.

Why the object lost half the call

The WebSocket hibernation API is what makes a Durable Object a good home for a signalling server. state.acceptWebSocket(ws) hands the socket to the platform, which keeps it open even while the object itself is evicted from memory. The object can be woken up for every message.

That is exactly the trap. The socket survives the eviction. A field in your object does not.

export class SignalingServer implements DurableObject {
  state: DurableObjectState;
  sessions: Set = new Set();   // ← lost on eviction

  async webSocketMessage(ws: WebSocket, msg: string): Promise {
    for (const o of this.sessions) {          // ← the set is empty now
      if (o === ws) continue;
      try { o.send(msg); } catch { }
    }
  }
}

After the eviction, the object was woken by a message, the loop walked a set that no longer contained the first socket, and the message went to the second socket only. The first person could still send — the message reached the object — but never received anything again, because the forward loop no longer knew about their socket.

Measured against the live server

After a gap of 70 seconds without traffic, the first socket received nothing while the second socket received everything. Same object, same session, same message. The difference was a field that had been emptied by the platform.

The fix: ask the platform who is here

The list of sockets is never kept. It is read from the platform every time it is needed, because the platform is the only party that knows it:

export class SignalingServer implements DurableObject {
  state: DurableObjectState;

  constructor(s: DurableObjectState) { this.state = s; }

  async fetch(r: Request): Promise {
    if (r.headers.get('Upgrade') !== 'websocket') {
      return new Response('Expected websocket', { status: 426 });
    }
    const { 0: cl, 1: sv } = new WebSocketPair();
    // The hibernation API keeps the socket across the eviction of this
    // object and hands it back through getWebSockets().
    this.state.acceptWebSocket(sv);
    return new Response(null, { status: 101, webSocket: cl });
  }

  async webSocketMessage(ws: WebSocket, msg: string): Promise {
    // A heartbeat is only there to keep the socket alive at the edge.
    // It is swallowed here and never passed on to the other person.
    try {
      const parsed = JSON.parse(msg);
      if (parsed && parsed.type === 'ping') return;
    } catch { /* not JSON: pass it on as it is */ }
    for (const o of this.state.getWebSockets()) {
      if (o === ws) continue;
      try { o.send(msg); } catch { /* the other end is gone; nothing to do */ }
    }
  }
}

Three things are worth noticing in the fixed version:

  • The state of the session lives in the platform. state.getWebSockets() is the list, and it is correct after an eviction because the platform never let go of the sockets.
  • The heartbeat is swallowed. A ping keeps the socket alive at the edge and must never be forwarded, or the other side receives it as an application message.
  • A dead socket is not an error. The catch around send is the normal case of the other person leaving, not an exception to be reported.

The general rule

With the hibernation API, the object is a function, not a container. Anything that has to survive an eviction lives in the platform: sockets through getWebSockets(), state through state.storage, alarms through the alarm handler. A field is a scratchpad that happens to be empty some of the time — and the times it is empty are the times nobody is looking.

The same lesson shows up in the next note, where a field was not the problem but the order of two blocks of code was.


← All notes · Next: a wider route ate a GDPR export request →