Production hardening¶

Thread safety of `SemvecState.update()`¶

SemvecState is a Rust extension type exposed via PyO3. The wheel ships a single _core.abi3.so and the Python layer (SessionManager) wraps each session in a _Session dataclass with no per-instance Python-level lock.

The collection of sessions is protected by a threading.Lock inside SessionManager (lookup, insert, evict). That lock is released before the caller ever touches SemvecState.update(). Concurrent update() calls on the same SemvecState instance therefore depend on whatever locking the Rust core does internally, which is not part of the public contract in 0.6.0.

Operating rule (memory / library backend — the default):

Treat each SemvecState instance as single-writer. Serialize update() calls per session at the application layer.

Lifted by the redis and mongo backends

This single-writer rule applies to the default in-process memory backend (and to direct library use of SemvecState). With SEMVEC_SESSION_BACKEND=redis or =mongo, the backend is the authoritative store and the constraint is lifted — any replica may serve any session. See Stateless scale-out (redis backend).

Nothing stands behind either of them, and that is what matters when you plan backups. Each is the system of record for the sessions it holds: a second store would put two version counters on one state, and a compare-and-set could then succeed against the wrong one. So neither reads DATABASE_URL, and neither can be treated as a cache you may lose.

The consequence differs only in how each is usually configured. Redis will discard data if you let it, so semvec refuses to start against a Redis configured to evict or not to persist, naming the setting — plan it as a database (appendonly yes, maxmemory-policy noeviction, backups). MongoDB persists by default, so there is no equivalent gate. Set SEMVEC_MONGO_URI (no default — a guessed host would look like it works and lose state) and optionally SEMVEC_MONGO_DB, and install pip install 'semvec[mongo]'. A missing driver refuses to start and names the extra; only an unrecognised backend name degrades quietly to memory.

"Unrecognised" has two halves, and only one of them is a typo. A name semvec ships is honoured; a name an installed package advertises through the semvec.backends entry-point group is honoured too, so installing a backend is what makes its name selectable. What degrades is a name neither ships nor advertises — which is a spelling mistake, and a safe default beats a crash. If you install a backend, name it, and get memory anyway, the name in your environment does not match the one in the package's metadata. See Session backends.

Recommended patterns:

# asyncio app
import asyncio
session_locks: dict[str, asyncio.Lock] = {}

async def safe_update(session_id, state, vec, text):
    lock = session_locks.setdefault(session_id, asyncio.Lock())
    async with lock:
        return state.update(vec, text)

# threaded app — pin one state per thread, or wrap in a queue
from concurrent.futures import ThreadPoolExecutor
pool = ThreadPoolExecutor(max_workers=1)  # one worker per session

In the default memory backend the REST layer already enforces this for you: /v1/run and /v1/store route every request for session_id=X through the same in-process _Session, and FastAPI's per-request handler keeps the call serialized within a single worker. Across multiple uvicorn/granian workers you need sticky routing — see the REST reference on SessionManager not being shared across worker processes. (With SEMVEC_SESSION_BACKEND=redis or =mongo this sticky-routing requirement is lifted — see Stateless scale-out.)

update_batch() is the recommended path for bulk ingest: it accepts a list of (embedding, text) pairs and lets the Rust core amortize the state mutations. Do not parallelize multiple update_batch() calls on the same state.

Graceful shutdown¶

SessionManager.shutdown(timeout=5.0) is wired into the FastAPI lifespan and fires on SIGTERM (uvicorn / granian both honor it). The sequence on SIGTERM is:

Stop accepting new connections. uvicorn closes the listening socket.
Drain in-flight requests. Active /v1/run, /v1/store and other handlers finish their current call. uvicorn's default timeout_graceful_shutdown is unbounded — set it explicitly (--timeout-graceful-shutdown 30) to bound the wait.
Cancel the session sweeper (semvec-session-sweeper asyncio task), await its exit.
SessionManager.shutdown(timeout=5.0): drain the shared embedder (5 s budget — beyond that the embedder is dropped without waiting), then self._sessions.clear().
Process exit.

What is lost at step 4:

In-memory SemvecState for every session that wasn't snapshotted. clear() drops the Rust handles. SQLAlchemy session/cluster/member metadata stays in the database; the hot semantic state does not. Restoring after restart requires POST /v1/session/{id}/import with a snapshot taken via GET /v1/session/{id}/export while the process was still up.
Embedder batches in flight that did not complete within 5 s. The sidecar/batched embedder's pending Futures resolve with a shutdown error; the calling handler propagates a 5xx to the client.

What is preserved:

Compliance pack event-store rows — written with the turn into the store the active backend keeps (SQL, Redis, or MongoDB), so they are durable before shutdown begins.
Audit chain (HMAC / RS256 entries written synchronously).
Every SQLAlchemy row for sessions, clusters, regions, observers, members, audit events — when a SQL store is configured; the default memory backend without SEMVEC_STATE_PERSIST opens no database.

Operational rule: if your application cannot tolerate snapshot loss, issue GET /v1/session/{id}/export on a checkpoint cadence (every N turns or every M minutes) and store the result alongside your own application state. The 0.6.0 server does not auto-snapshot.

Systemd example:

[Service]
ExecStart=/opt/semvec/.venv/bin/semvec serve --host 127.0.0.1 --port 8080
KillSignal=SIGTERM
TimeoutStopSec=60s
Restart=on-failure

Kubernetes example:

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: semvec
    lifecycle:
      preStop:
        exec:
          # Stop receiving new traffic, then let SIGTERM drain in-flight.
          command: ["sleep", "10"]

The preStop sleep 10 is a deliberate hack — it lets the service-mesh / Endpoints controller drop the pod from rotation before SIGTERM fires, so no new connection lands on a pod that's already draining.

Liveness vs readiness¶

0.6.0 ships one health endpoint: GET /v1/health (no auth, returns status, active_sessions, version). It is suitable as a liveness probe — it answers if the asyncio loop is alive and the SessionManager is reachable.

It is not a readiness probe. It does not check:

Embedder is loaded (first call lazy-loads it; a fresh worker will return 200 on /v1/health while the next /v1/run blocks for 10–30 s on model download / GPU warm-up).
Database connection is healthy.
Sidecar embedder (SEMVEC_EMBEDDER_URL) is reachable.
License-verification keyset is loaded.

Workarounds until a dedicated readiness endpoint ships:

K8s exec probe — call the API once during readiness, fail if it doesn't return memories:

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      curl -fsS -H "X-API-Key: $SEMVEC_LICENSE_KEY" \
        -X POST http://127.0.0.1:8080/v1/run \
        -H 'Content-Type: application/json' \
        -d '{"message":"readiness probe"}' \
        | grep -q session_id
  initialDelaySeconds: 30
  periodSeconds: 15
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /v1/health
    port: 8080
  periodSeconds: 10
  failureThreshold: 5

Pre-warm in the entrypoint — make the embedder load synchronously before uvicorn starts taking traffic:

.venv/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"
.venv/bin/semvec serve --host 0.0.0.0 --port 8080

Combine both: the entrypoint pre-warm makes the first /v1/run fast, the readiness probe protects against silent embedder failures (out-of-disk on model download, sidecar 503, etc.).

Prometheus label cardinality and tenant leaks¶

/metrics exports three series from a private CollectorRegistry:

Series	Type	Labels
`semvec_requests_total`	Counter	`method`, `endpoint`, `status`
`semvec_request_duration_seconds`	Histogram	`method`, `endpoint`
`semvec_active_sessions`	Gauge	—

The endpoint label is the route template (e.g. /v1/session/{session_id}), not the resolved URL — a deliberate fix to keep cardinality bounded. UUID session/cluster/region IDs never reach the label.

No tenant-scoped series ship. license_subject is not a Prometheus label in 0.6.0. The Counter and Histogram aggregate across all callers. Implication:

A noisy tenant cannot be isolated from /metrics alone — you need application logs (which do carry session_id) plus log-based aggregation.
Conversely, you do not need to strip tenant data from /metrics before exposing it to a shared monitoring stack.

If you add custom Counters/Histograms to a fork or downstream deployment, do not add session_id, cluster_id, region_id, memory_hash, or license_subject as labels. Each is unbounded in cardinality (UUIDs, BLAKE3 hashes) and will blow up the Prometheus backend. Use exemplars or trace-IDs for per-request drilldown instead.

OpenTelemetry hooks¶

Semvec 0.6.0 does not emit OpenTelemetry traces or metrics natively. If you need distributed tracing across your agent → /v1/run → embedder sidecar → database hops, instrument at the FastAPI layer with opentelemetry-instrumentation-fastapi:

pip install opentelemetry-instrumentation-fastapi \
            opentelemetry-instrumentation-sqlalchemy \
            opentelemetry-exporter-otlp

Install

OpenTelemetry instrumentation is opt-in: pip install opentelemetry-instrumentation-fastapi opentelemetry-sdk opentelemetry-exporter-otlp. semvec does not bundle OTel.

# wrapper around semvec.api:create_app
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from semvec.api import create_app

app = create_app()
FastAPIInstrumentor.instrument_app(app)

This captures HTTP-level spans (method, route, status, duration). It does not see inside the Rust core — SemvecState.update() is a single PyO3 call, opaque to Python-level tracing. The same is true for sidecar embedder calls if you do not separately instrument the embedder process.

Connection draining for the embedder sidecar¶

When SEMVEC_EMBEDDER_URL is set, the API process owns a SidecarEmbedderClient (HTTP keep-alive pool) instead of an in-process SentenceTransformer. Shutdown order matters:

Drain the API first (SIGTERM semvec serve workers). The SessionManager.shutdown() lifespan hook closes the embedder client inside the 5 s drain window.
Then drain the sidecar. SIGTERM python -m semvec.embedder. It has its own batch-flush logic; the daemon will reject new HTTP submissions and finish in-flight batches before exiting.

Reversing the order (sidecar first) causes the API workers to surface ConnectionRefusedError on every in-flight /v1/run until they finish draining. Always shut the consumer down before the producer.

If you run sidecar and API as separate Deployments behind a Service, add a preStop sleep on the sidecar Deployment that is longer than terminationGracePeriodSeconds on the API Deployment, so the sidecar outlives any API pod that's still draining.

State persistence & durability¶

The "Graceful shutdown" section above describes the 0.6.0 baseline: in-memory state, lost on clear(), restored only by an explicit export/import cadence you wire yourself. From 0.7.1 the server can do that for you — durable per-session state behind one environment variable.

`SEMVEC_STATE_PERSIST` — durable per-session state (default OFF)¶

Set SEMVEC_STATE_PERSIST=1 and point DATABASE_URL at Postgres to make each session's semantic state survive a worker restart. The mechanics:

Write-behind, not write-through. A flush is not taken on every turn. The server flushes a session's state on a periodic tick and again on SIGTERM (inside the graceful-drain window). This keeps the hot path off the database — turns stay in memory and only the flush pays the write cost.
Lazy reload. State is read back from the store on the first access to a session after a restart, not eagerly on boot. A fresh worker starts cold and hydrates a session the moment a request for it arrives.
Recovery point (RPO). On a graceful shutdown (SIGTERM) the final flush fires, so a session resumes bit-exact. On a hard crash (OOM kill, SIGKILL, node loss) you can lose up to one flush interval of turns — whatever changed since the last tick. Size the flush interval to the loss you can tolerate.

State blobs are stored as compressed binary (~56% smaller than the legacy JSON encoding). The reader is backward-compatible: legacy uncompressed JSON blobs written by older deployments are still read transparently, so an in-place upgrade needs no migration step.

One owner worker per session (memory backend)¶

Applies to the memory backend only

Everything in this SEMVEC_STATE_PERSIST section describes the default in-process memory backend, where state lives in the worker that owns the session and Postgres — when SEMVEC_STATE_PERSIST=1 — is the durable store for it, not a backstop behind a second one. If you want any replica to serve any session with no sticky routing, use the redis backend instead — see Stateless scale-out (redis backend).

In the memory backend, persistence assumes a single writer per session. There is no distributed lock — instead, each flush carries an optimistic version, and the store refuses a write whose version is stale (a compare-and-set that would clobber a newer blob). That refusal is loud (logged, surfaced), not a silent last-writer-wins.

The practical consequence: route every request for a given session_id to the same worker (sticky routing — consistent hashing at the load balancer, or a session-affinity cookie). If two workers both own the same live session and both flush, the CAS rejects the loser rather than corrupting the blob, but you will see churn and lost writes on the rejected side. Sticky routing avoids the contention entirely. This is the same single-writer rule as Thread safety of SemvecState.update(), extended across worker processes.

`SEMVEC_STATE_DB_SHARDS` — Postgres sharding (opt-in, advanced scaling)¶

For deployments where a single Postgres primary is the write-throughput ceiling, SEMVEC_STATE_DB_SHARDS takes a comma-separated list of Postgres DSNs and spreads the state-blob table across them:

export SEMVEC_STATE_PERSIST=1
export SEMVEC_STATE_DB_SHARDS="postgresql://u:p@shard-a/semvec,postgresql://u:p@shard-b/semvec,postgresql://u:p@shard-c/semvec"

Sharded by session_id via rendezvous (HRW) hashing. Each session maps to exactly one shard. Rendezvous hashing is chosen for its minimal reshuffle property: adding a shard moves only ~1/N of the keys (vs. a modulo scheme that remaps almost everything), so you can grow the pool with bounded rebalancing.
Metadata stays on the primary. Only the high-volume state-blob table is sharded. The session / cluster / region / audit metadata tables continue to live on the primary DATABASE_URL, so ownership checks and the compliance surface are unaffected.

This is a write-throughput lever, not a write-amplification fix. Sharding spreads writes across more backends; it does not reduce how much each turn writes. The orthogonal lever for that is blob compression (above), which shrinks each write ~56%. Use both together at scale. (Ships in 0.7.1.)

Persisting state in-process / library use¶

Automatic persistence is a server convenience — it lives in the FastAPI lifespan. A library user driving SemvecState / SemvecSession directly (no FastAPI) persists manually, and the primitive is the same compressed blob the server uses:

# --- on shutdown / checkpoint: serialize and store anywhere ---
blob = state.to_bytes(compress=True)   # compact binary, ~56% smaller
# persist `blob` wherever you like: a file, your own DB column, an
# object store (S3/GCS), etc. — it's opaque bytes.
Path("session-42.semvec").write_bytes(blob)

# --- on startup: load it back ---
blob = Path("session-42.semvec").read_bytes()
state = SemvecState.from_bytes(blob)   # auto-detects compressed vs legacy

from_bytes() auto-detects the encoding, so a blob written by an older build (uncompressed) still loads. See to_bytes / from_bytes in the Core API reference.

Durable cross-frontend dedup in-process is the same recipe plus one shared session: hold a single SemvecSession, feed every frontend's turns through it, and save/load it with to_bytes/from_bytes on the boundaries. The matched_id in each dedup_signal is a durable handle — it survives the to_bytes/from_bytes round-trip, so a correlation stored before the restart still resolves afterward. See Cross-frontend dedup → in-process.

What a restore reproduces — and the one part that is licence-gated¶

From 0.8.2 a restored snapshot continues the source state's evolution: the same memories in the same tiers, the same retrieval results, and the same trajectory on the next update(). The integrity checksum covers all of it, so a blob that verifies is a blob that continues.

One exception, and it depends on your licence. The three diagnostics methods — calculate_fsm(), calculate_metrics() and calculate_advanced_metrics() — are salted per state instance. From 0.8.2 that salt travels inside the snapshot, sealed, so a restore can reproduce the source state's numbers exactly. Reproducing them requires all three of:

an official wheel (pip install semvec) — a wheel you build yourself carries no sealing secret and therefore never seals,
a Pro or Enterprise licence — on Community the salt is not written at all,
the same licence subject that wrote the snapshot.

If any of the three differs, the restore is still complete and valid — every memory, every retrieval result, every checksum — but the calculate_* methods resume from a fresh salt. Their absolute values then differ from what the source state reported. What they are used for keeps working: comparing a value against a threshold, and watching it move, within one restored state.

Why this is worth knowing before you see it. If you alert on calculate_fsm() > 0.7, or diff metrics across a restart, expect a step change at the restore boundary in the cases above — and read it as the documented behaviour rather than as data loss or a defect. Snapshots written before 0.8.2 behave the same way on every tier, because they carry no salt at all.

Semvec says so on stderr when a snapshot carried a sealed salt that this build/subject could not open:

semvec: snapshot carries a sealed seed that this build/subject cannot open
(wrong build secret, different licence subject, or tampering); continuing
with a fresh seed

A snapshot that never carried one — pre-0.8.2, Community, or a self-built wheel — restores silently, because that is the normal documented case and not a mismatch.

Stateless scale-out (redis backend)¶

Everything above this section describes the default memory backend: state lives in the worker that owns a session, so you must route each session_id to one owner worker (sticky routing) and a single writer mutates it at a time.

From 0.8.0 there is a second option. Set SEMVEC_SESSION_BACKEND=redis and the model inverts: Redis becomes the authoritative store for every session, pods are stateless, and any replica can serve any session. The single-writer-per-session rule and the sticky-routing requirement (above) are lifted — put the API behind a plain round-robin load balancer.

This is opt-in. The default stays memory; an unset or unrecognised SEMVEC_SESSION_BACKEND behaves exactly as previous releases. Install the client with pip install "semvec[redis]".

Everything in this section applies equally to SEMVEC_SESSION_BACKEND=mongo (pip install "semvec[mongo]", SEMVEC_MONGO_URI required): MongoDB is then the authoritative store, every mutation is written through synchronously, and pods are stateless in the same way. The differences are configuration, not model — MongoDB persists by default, so there is no startup durability gate equivalent to the Redis one, and audited turns need a replica set (see the compliance pack guide).

How correctness holds without sticky routing¶

Redis is authoritative. The current copy of each session's state lives in Redis, not in any single pod. A pod that has never seen a session loads it from Redis on first access.
Version-validated per-pod cache. Each pod keeps a hot local cache of the sessions it has recently touched (bounded by the existing SEMVEC_MAX_SESSIONS LRU). On every read, the cached entry's version is checked against Redis with one cheap call (no blob transfer). If another pod has advanced the session, the stale local copy is dropped and reloaded — so a cache hit is never served stale.
Write-through CAS. Writes go through an optimistic compare-and-set against the authoritative version. If two pods race on the same session, the loser's write is rejected loudly (and it reloads and retries) rather than clobbering the newer blob. No distributed lock, no last-writer-wins.
Redis is the system of record — there is nothing behind it. This is a change of design, not of wording: earlier releases kept a second copy in Postgres, written off the hot path, and refilled Redis from it when a key went missing. That copy no longer exists. A key lost to eviction, maxmemory pressure or a Redis started without persistence is a lost session, not a cold read.

Two copies were removed rather than kept because they put two version counters on one state, and a compare-and-set can then succeed against the wrong one. The server therefore refuses to start against a Redis configured to evict or not to persist, and says which setting is wrong — see Redis durability below. Plan your Redis as a database: appendonly yes, maxmemory-policy noeviction, and a backup policy. - Fleet-wide rate limiting. The configured QPS/burst ceiling is enforced through Redis across all replicas, so adding pods does not multiply the limit.

Readiness¶

Wire the orchestrator's readiness probe to GET /v1/readyz (distinct from the /v1/health liveness probe). /v1/readyz reports whether the pod's dependencies — Redis, the durable database, the embedder — are actually serviceable, so a pod is pulled out of the Service endpoints during a dependency outage instead of black-holing requests. Keep /v1/health on the liveness probe.

Deployment artifacts¶

Two ready-to-adapt deployments ship in the repo:

deploy/k8s/ — Kubernetes manifests: deployment.yaml, service.yaml, configmap.yaml (non-secret config, including SEMVEC_SESSION_BACKEND=redis and REDIS_URL), hpa.yaml (horizontal autoscaler), and secret.example.yaml (license key + DATABASE_URL). Wire /v1/readyz to the readiness probe and /v1/health to liveness.
deploy/compose/ — a Docker Compose load-test harness: API + Redis + Postgres + embedder + nginx, for exercising the stateless path locally.

A production Dockerfile is at the repo root.

Environment variables¶

Variable	Purpose
`SEMVEC_SESSION_BACKEND`	`memory` (default), `redis`, `mongo`, or a name an installed package contributes. Anything but `memory` selects the stateless path.
`REDIS_URL`	The session store for the redis backend — authoritative, with nothing behind it.
`DATABASE_URL`	The SQL session store, used only by `memory` + `SEMVEC_STATE_PERSIST=1`. Not read by the redis or mongo backends. No default: if you ask for SQL persistence without setting it, startup fails naming the variable rather than creating a file.
`SEMVEC_EMBEDDER_URL`	Sidecar/daemon embedder endpoint (shared by both backends).
`SEMVEC_REDIS_SESSION_TTL_S`	Idle TTL applied to Redis session keys; `0` (default) = no expiry.
`SEMVEC_MAX_SESSIONS`	Bounds the per-pod hot cache (the LRU that also caps the `memory` backend).
`SEMVEC_SESSION_SWEEP_S`	Cadence of the idle-session sweep (default 60 s). The same tick also runs the SQL flush, which does nothing unless the backend is `memory` with `SEMVEC_STATE_PERSIST=1` — a write-through backend has already stored the turn.

Rollback¶

Rollback is a single flag flip: set SEMVEC_SESSION_BACKEND back to memory (or unset it) and redeploy. There is no data migration — the memory backend is byte-for-byte unchanged from previous releases, so reverting is instant and safe.