dgen_server Design

dgen_server brings the gen_server programming model (init/1, handle_call/3, handle_cast/2, ...) to a distributed system. A dgen_server is not an Erlang process. It is durable state plus a durable message queue in a strongly-consistent backend (default: FoundationDB). It exists as long as that state does, independent of any process, node, or cluster.

The Guarantees section is normative. Behaviour that contradicts it is a bug.

This document is LLM generated.

1. The one-minute model

• Zero or more consumer processes, on any node, pull messages off the queue and run your callback. Each one commits the new state, and the reply if there is one, in a single database transaction.
• Standard messages (call, cast) go through the queue and are processed with strict serializability. Priority messages (priority_call, priority_cast, handle_info) skip the queue and run immediately, at the cost of ordering.
• Callbacks run inside that transaction and must be pure. Side effects go in Actions, which run after the commit (§4.3), or in handle_locked/4 (§4.4).
• If a consumer crashes mid-transaction, nothing commits. The next consumer retries a queued message. Priority messages are lost (§5).

2. Quick start

Erlang

-module(counter).
-behavior(dgen_server).

-export([start/1, increment/1, value/1]).
-export([init/1, handle_call/3]).

start(Tenant) -> dgen_server:start(?MODULE, [], [{tenant, Tenant}]).
increment(Pid) -> dgen_server:call(Pid, increment).
value(Pid) -> dgen_server:call(Pid, value).

init([]) -> {ok, 0}.

handle_call(increment, _From, State) -> {reply, ok, State + 1};
handle_call(value, _From, State) -> {reply, State, State}.

Tenant = dgen_erlfdb:sandbox_open(<<"demo">>, <<"counter">>),
{ok, Pid} = counter:start(Tenant),
counter:increment(Pid), counter:increment(Pid),
dgen_server:stop(Pid),
{ok, Pid2} = counter:start(Tenant),
2 = counter:value(Pid2).  %% State persisted

Elixir

defmodule Counter do
  use DGen.Server

  def start(tenant), do: DGen.Server.start(__MODULE__, [], tenant: tenant)
  def increment(pid), do: DGen.Server.cast(pid, :increment)
  def value(pid), do: DGen.Server.call(pid, :value)

  @impl true
  def init([]), do: {:ok, 0}

  @impl true
  def handle_call(:value, _from, state), do: {:reply, state, state}

  @impl true
  def handle_cast(:increment, state), do: {:noreply, state + 1}
end

tenant = :dgen_erlfdb.sandbox_open("demo", "counter")
{:ok, pid} = Counter.start(tenant)
Counter.increment(pid); Counter.increment(pid)
GenServer.stop(pid)
{:ok, pid2} = Counter.start(tenant)
2 = Counter.value(pid2)  # State persisted

3. Core concepts

• backend: the database interface; the provided one is FoundationDB via erlfdb.
• tenant: a database handle plus a directory handle defining a key subspace. Every dgen_server is started against one.
• key-tuple: a tuple encoded as a key in a tenant. Extending it makes the original a prefix that contains other keys.
• tuid: the key-tuple identifying a dgen_server in its tenant. init/1 may return {ok, Tuid, State} to choose it.
• message-queue: the durable queue a dgen_server processes requests from.
• call-request / cast-request: a queued operation that does / does not expect a response.
• priority-request: a call or cast handled immediately, ignoring the queue and its ordering. It is meant for urgent, usually read-only work.
• dgen_server process: any process that can push requests onto the queue.
• consumer: a dgen_server process that also processes requests. A dgen_server may have zero, one, or many consumers.

4. How it works

4.1 Message processing paths

4.2 consume_k and inlining

consume_k (default 1) is the maximum number of queued messages a consumer processes per transaction. A consume cycle peeks up to consume_k messages and processes them in order, carrying the state in memory. It commits the batch in one transaction, so the state is read once and written once. Then it starts the next cycle right away.

A larger consume_k spreads the cost of each transaction over more messages and raises throughput when callbacks are cheap. The price is larger transactions and all-or-nothing batches, because a conflict retries the whole batch. A busy consumer keeps draining, so it also tends to keep a single node as the active consumer. That helps when it matters which node consumes, as in election-style callbacks.

Inlining. When consume_k =:= 1, a call/2 whose queue is empty and unlocked is processed immediately in the caller's own transaction, with the same result as going through the queue. When consume_k > 1, inlining is disabled and every call/2 goes through the queue.

priority_call/2 and priority_cast/2 always bypass the queue and the lock (§4.1).

4.3 Actions

Callbacks may return {reply, Reply, State, Actions} or {noreply, State, Actions}, where Actions is a list of 1-arity functions that:

• run after the transaction commits.
• receive the committed State but cannot modify it.
• are the place for side effects such as logging, telemetry, and external publishing.
• may return halt to skip the remaining actions.

With consume_k > 1, a batch's actions all run after its single commit and each receives the state as of the end of the batch.

4.4 Locking

A callback may return {lock, State} to enter locked mode. Locked mode is for work that must finish before the next queued message is processed but cannot run inside a transaction, such as external API calls or long computations.

While locked:

• Standard call and cast messages are queued but not processed.
• Priority messages and handle_info still execute (§4.1).
• handle_locked/4 runs outside a transaction. It has no transaction limits and may perform side effects, and the state it returns is written back.

lock_timeout (default infinity) bounds how long a lock may be held. The holder normally clears the lock when handle_locked/4 returns. If the holder is killed (SIGKILL, VM abort), another consumer clears the lock once lock_timeout ms have passed since it was set. We call that busting the lock. With infinity, a dead holder blocks all consumers permanently.

Busting is a guess that the holder is dead, and a slow holder may still be running. So the lock is fenced (§4.7). If a holder was busted, we don't commit its result. Its message goes back on the queue, and the holder exits with {lock_fenced, requeued}. A priority or inline lock has no queued message, so the holder exits with {lock_fenced, not_requeued} and the caller receives the exit. Either way, the side effects of handle_locked/4 have already happened, so handle_locked/4 must be safe to run twice.

Because of fencing, a wrong guess costs a retry rather than lost data. So set lock_timeout to how quickly you want to recover from a dead holder, rather than above the worst-case handle_locked/4 duration.

4.5 Persisted state

State is encoded to optimise partial updates:

1. Assigns map: a map whose keys are all atoms. Each entry gets its own key, unordered.

    #{mykey => <<"my value">>, otherkey => 42}

2. Components list: a list of maps, each with an atom id key holding a binary. Each item gets its own key, ordered by fractional indexing.

    [#{id => "item1", value => 1}, #{id => "item2", value => 2}]

3. Term: anything else, via term_to_binary, chunked into 100KB values.

4. Nesting: encodings nest recursively, e.g. a components list inside an assigns map.

On update, assigns maps and components lists write only the entries that changed. Removed entries are cleared, and reordering updates the fractional indices. Terms are rewritten in full. If the encoding type changes, we clear the old keys and write the new encoding in full.

4.6 Caching

A consumer may cache the state with its versionstamp and skip the read when the versionstamp still matches the database. The cache is invalidated when another consumer modifies the state.

4.7 Request flows

Call: the caller enqueues the request with its from-key and watches the reply-sentinel-key. A consumer runs handle_call/3 and writes the new state. If the sentinel still exists, it also writes the reply. Then it commits. After that, the consumer runs the actions while the caller reads and clears the reply. A caller that times out clears its reply keys so none leak. As with gen_server, the callback still runs and the state still changes.

Cast: the caller enqueues the request. A consumer runs handle_cast/2, commits, then runs the actions.

Lock:

1. The callback returns {lock, NewState}. In one transaction the consumer commits NewState, dequeues the message, and writes a lock key holding {Timestamp, Token}. Token is unique to this acquisition.
2. Other consumers see the lock and wait on a DB watch.
3. The holder calls handle_locked/4 with the triggering event, message, and committed NewState.
4. The holder commits the result in a second transaction, but only if the lock key still carries its Token. Otherwise nothing is written, the message is requeued in that transaction, and the holder exits with {lock_fenced, requeued}. A priority or inline lock exits with {lock_fenced, not_requeued} instead.
5. However handle_locked/4 ends, even with an exception, the holder clears the lock key if it still carries its Token. Clearing notifies the queue watch, so waiting consumers resume.

4.8 Internal terms

• from-key: {WaitingKey..., Timestamp, term_to_binary(Ref)} identifies one waiting caller under the waiting-key prefix. The timestamp lets abandoned keys be garbage-collected.
• reply-sentinel-key: chunk 0 of the reply. Replies are stored in chunks as {From, <<"t">>, N} so they may exceed the backend value-size limit. The caller watches this key.
• quid: the prefix of all queue keys. Under it are the item-keys, which hold queued requests, and the push-key and pop-key counters.

5. Crash and failure behaviour

Standard call and cast messages are processed exactly-once under normal operation and at-least-once if a consumer crashes before commit (bounded by dead_letter_threshold when set). Make callbacks idempotent where possible.

During init/1: nothing is persisted yet; on restart init/1 runs again from scratch.

During handle_call, handle_cast, handle_info: the transaction aborts and state is unchanged. A call or cast stays in the queue, and the next consumer retries it, subject to dead_letter_threshold. A priority_call, priority_cast, or handle_info message is lost.

Dead-letter queue. A poison message consistently crashes consumers. With dead_letter_threshold set to a positive integer, each failed attempt increments a counter in the message envelope. At the threshold, the message is moved to the dead-letter queue (DLQ) instead of being retried, and:

• a blocked call caller raises {dead_letter, N}, where N is the attempt count.
• the optional handle_dead_letter/2 callback receives the message and count.
• a warning is logged.

Dead-lettering is disabled by default (infinity):

Erlang

dgen_server:start(MyMod, [], [{tenant, Tenant}, {dead_letter_threshold, 3}])

Elixir

DGen.Server.start_link(MyMod, [], tenant: tenant, dead_letter_threshold: 3)

Supervisor restart intensity. Each consumer crash is a supervisor restart. If the supervisor's intensity limit (default 3 restarts in 5 seconds) is hit before dead_letter_threshold, the supervisor terminates instead. Set intensity/max_restarts >= dead_letter_threshold with a period long enough to cover the crash-restart cycle, e.g. 5 restarts in 60 seconds for a threshold of 3. With infinity, a poison message crash-loops until the supervisor gives up.

Managing the DLQ. Dead-lettered messages are never cleared automatically. Operators manage them with dgen_queue, using the Quid from dgen_server:get_quid(Tuid):

Erlang

Quid = dgen_server:get_quid(Tuid),
Entries = dgen_queue:peek_dlq(Tenant, Quid),  %% [{Key, Envelope, Attempts, TsMs}]
dgen_queue:dlq_length(Tenant, Quid),
dgen_queue:requeue_dlq_entry(Tenant, Quid, Key),  %% ok | {error, not_found}
dgen_queue:delete_dlq_entry(Tenant, Key),
dgen_queue:purge_dlq(Tenant, Quid).

Elixir

quid = :dgen_server.get_quid(tuid)
entries = :dgen_queue.peek_dlq(tenant, quid)  # [{key, envelope, attempts, ts_ms}]
:dgen_queue.dlq_length(tenant, quid)
:dgen_queue.requeue_dlq_entry(tenant, quid, key)  # :ok | {:error, :not_found}
:dgen_queue.delete_dlq_entry(tenant, key)
:dgen_queue.purge_dlq(tenant, quid)

requeue_dlq_entry/3 atomically moves the envelope back to the main queue with its attempt count reset to zero, for retry after the cause is fixed.

During handle_locked: the state committed with the lock is already persisted, and the triggering message is already dequeued. On an exception, the lock is cleared before the process exits, and the message is not retried. If the process dies without clearing the lock (SIGKILL, OOM, power loss), the lock blocks the queue until lock_timeout expires. With infinity it blocks forever and needs manual intervention. If the holder is fenced, its message is requeued and retried (§4.4).

During actions: state is already committed and the message consumed. A crashing action stops the remaining actions; nothing is retried.

Supervisor restart: if durable state exists, init/1 is called but the state it returns is ignored. The process resumes from the persisted state and, if configured to, consumes alongside any other consumers.

6. Guarantees

These are normative. Behaviour that violates one of them is a defect.

1. Durability. State and queued messages persist in the backend, independent of any Erlang process, node, or cluster.
2. Delivery. Standard call/cast messages are processed exactly-once under normal operation. If a consumer crashes before its transaction commits, another consumer retries the message. In that case delivery is at-least-once, bounded by dead_letter_threshold when configured (§5).
3. Serializability. Standard messages are committed with strict serializability through the durable queue. A batch reads and writes the state in one transaction.
4. Priority messages trade ordering for immediacy. priority_call/2, priority_cast/2, and handle_info/2 always bypass the queue and the lock, so they are not ordered with respect to the queued stream.
5. Actions run exactly once per committed batch, strictly after that batch's commit, once the state they reference is durable.
6. Lock exclusivity. While a dgen_server is locked, only the lock holder's handle_locked/4, priority messages, and handle_info/2 execute; other standard call/cast messages wait until the lock clears.
7. Callback purity. handle_call/3, handle_cast/2, and handle_info/2 run inside a database transaction and must not perform side effects. Side effects belong in Actions (§4.3) or handle_locked/4 (§4.4), which run outside a transaction.

7. Configuration

Options in the Opts proplist of start/3,4 and start_link/3,4:

8. When dgen_server is (and isn't) a good fit

Good fit when you want the gen_server model but need state and in-flight messages to survive the death of a process, node, or cluster. A database round-trip per message must be acceptable, and side effects must be able to wait for Actions or run in handle_locked/4.

Reconsider when you need in-memory latency. A plain gen_server is faster for state that need not survive a restart. Also reconsider if your callbacks need side effects inline that can't be deferred, or if you can't or don't want to operate a strongly-consistent external database.