OpenAI-compatible proxy¶

Semvec exposes an OpenAI-compatible HTTP surface so you can keep using the official openai SDK (or any OpenAI-compatible client) and get persistent, per-conversation memory for free. Semvec sits between your client and your real LLM provider: it injects the relevant memory as context on the way in, forwards a compacted request to the provider, and folds the reply back into memory on the way out.

You change only two things in your client: base_url and api_key.

Quickstart¶

pip install openai

from openai import OpenAI

client = OpenAI(
    api_key="sk-semvec-...",                       # your Semvec proxy key
    base_url="https://YOUR-HOST/openai/v1",         # point at Semvec
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What did we decide about the schema?"}],
    extra_headers={
        "X-Semvec-Session": "design-thread-42",     # which conversation (see below)
        "X-Upstream-Authorization": "Bearer sk-...",  # your provider key (BYOK)
    },
)
print(resp.choices[0].message.content)

That request goes to Semvec, which retrieves what it remembers for design-thread-42, prepends it as context, calls your upstream provider, and stores the assistant's reply so the next turn can recall it.

Conversations: the `X-Semvec-Session` header¶

Your proxy key identifies you (the tenant) — it does not identify a single conversation. A real API key is reused across many unrelated chats, end-users, and parallel agent runs, so Semvec needs a separate signal for "which conversation is this?" so memory from one chat never leaks into another.

Semvec resolves the conversation (thread) from, in order:

The X-Semvec-Session request header (recommended — explicit and unambiguous).
The OpenAI user field in the request body (the standard per-end-user identifier).
A last-resort hash of the first user message (lossy — distinct conversations that open with identical text collide; set #1 or #2 to avoid this).

# Option A — the dedicated header (recommended):
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"X-Semvec-Session": "support-ticket-9921"},
)

# Option B — the standard OpenAI `user` field:
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    user="support-ticket-9921",
)

Use a stable id per conversation: the same id continues the same memory; a new id starts a fresh one.

Upstream credentials: BYOK vs managed¶

Semvec forwards your request to a real LLM provider. There are two ways to supply the provider credential.

BYOK (bring your own key) — default¶

You send your own provider key on each request in a separate header, X-Upstream-Authorization. Your Authorization: Bearer stays the Semvec proxy key. Keeping the two on different headers means neither is ever confused for the other.

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={
        "X-Semvec-Session": "thread-1",
        "X-Upstream-Authorization": "Bearer sk-your-provider-key",
    },
)

The provider key is used only for that one upstream call and is never logged or stored.

Managed¶

Your Semvec operator stores the provider key for your tenant. You then send only the Semvec proxy key — no X-Upstream-Authorization:

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"X-Semvec-Session": "thread-1"},
)

Managed mode centralizes provider billing; BYOK is the default because it keeps Semvec out of the business of custodying your provider secrets.

Prompt compaction: `x-semvec-tokens-saved`¶

Because Semvec already holds the conversation history, it does not forward your whole messages array upstream. It replaces the prior turns with a compact context block and keeps only your latest user turn. That means the upstream prompt is smaller — you pay for fewer prompt tokens — without you trimming history yourself.

Each response carries a header reporting how many prompt tokens that compaction dropped on the turn:

resp = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"X-Semvec-Session": "thread-1"},
)
print(resp.headers["x-semvec-tokens-saved"])   # e.g. "318"
completion = resp.parse()

The upstream usage block is passed through untouched, and every response also carries an x-request-id (echoed from your request if you sent one, otherwise minted) so you can correlate a turn with your own logs.

Streaming¶

Streaming works exactly as with the OpenAI API — pass stream=True and iterate chunks. Semvec forwards the upstream stream to you and folds the assembled reply into memory once the stream completes:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize the thread so far."}],
    stream=True,
    extra_headers={"X-Semvec-Session": "thread-1"},
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

`GET /v1/models`¶

GET /openai/v1/models is supported and returns an OpenAI-shaped model list for the model your proxy key is wired to, so SDK calls and tooling that probe models keep working.

Security note¶

Your proxy key is both a credential and your memory identity

The Semvec proxy key is simultaneously your bearer credential and (via the tenant it resolves to) your memory identity. Two consequences follow:

Rotating the key can drop the session. Treat key rotation as an operational change that must preserve your tenant's memory bindings — check with your operator before rotating.
Sharing the key shares the memory. Anyone holding your proxy key can read and write your Semvec memory (within whatever thread they name). A leaked proxy key is a memory-confidentiality breach, not just a billing one. Guard it with the same care as a provider key.

Under BYOK, your provider key transits Semvec only for the upstream call and is never logged or persisted. Under managed mode, the operator's stored provider key is encrypted at rest.