Re-ID Service#
The Re-ID service packages the existing Re-ID database contract and VDMS and Qdrant adapters as a standalone deployable service. This extraction keeps adapter behavior unchanged and adds a backend-neutral gRPC boundary.
Current Scope#
The service provides health, FindMatches, FindSchemaMetadata, manual
PurgeExpired, IngestStats, authorized CreateCollection, EnrollPoi, and
AppendPoiEmbeddings, and DeletePoi alongside Tracker MQTT ingestion.
The Controller still uses compatible controller.reid* imports for its
in-process path. Controller retirement, POI APIs, gallery management, and
trajectory export are follow-on work.
Baseline gRPC Contract#
The authoritative declaration is reid/proto/reid.proto, service
scenescape.reid.ReIDService. All nine RPCs are unary calls with
google.protobuf.Struct requests and responses. Method names and existing
response fields remain compatible with the initial service. Health.ready
is additive; IngestStats is now explicitly declared as well as implemented.
RPC |
Request |
Response |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The examples use three-element vectors only for brevity. FindMatches requires
a rectangular numeric matrix with 1-1024 rows, 1-4096 dimensions, and finite
float32 vectors. Inner-product normalization additionally requires nonzero
vectors; L2 permits zero vectors. Every row must match the selected gallery’s
schema dimensions, including when the service adapter starts with
dimensions=None. Existing schema dimensions are cached per set for
30 seconds (up to 128 sets); absence and failed lookups are not cached. A schema
whose dimensions cannot be read produces UNAVAILABLE, not an unchecked query.
object_type and an optional set_name must be nonempty
strings of at most 256 characters without NUL bytes. An omitted or null
set_name selects the configured set. Optional k_neighbors is an integer
from 1 to 1000; omitting it preserves the adapter default. Optional constraints
is an object of the existing confidence-bearing metadata filters, for example
{"gender":{"label":"female","confidence":0.95}}. Reserved identity/persist
fields remain ignored by the adapters; constraints cannot override object_type,
set_name, k_neighbors, reid_vectors, or internal call/threshold controls.
Unknown top-level request fields are rejected. CreateCollection accepts a
name, dimensions from 1 to 4096, and metric IP or L2. It creates or verifies
schema only; it does not enroll descriptors. The configured live-gallery,
default live-gallery, and internal schema-marker names are reserved. Compatible
repeats return created:false; mismatches return FAILED_PRECONDITION.
EnrollPoi requires a declared collection, object type, 1-5 valid vectors,
severity, notes, and enrolled_by; the service generates the POI id.
AppendPoiEmbeddings requires an existing POI and enrolled_by, and accepts
1-5 additional vectors. Each successful operation stores an immutable audit
record with writer, timestamp, operation, and vector hashes. A failed batch
returns no POI id or success count; this is not a storage transaction or
durability guarantee. No general-gallery descriptor-write RPC is exposed.
DeletePoi requires the POI writer token, a declared non-live collection, a
canonical lowercase UUID, and nonempty deleted_by attribution. It removes
all searchable descriptors for that POI, including appended embeddings,
while retaining the collection, schema markers, and all other identities.
The stored profile must identify the UUID as a POI; ordinary gallery entries
cannot be deleted through this RPC. A missing or already-deleted POI returns
NOT_FOUND. Successful deletion is verified by an exact UUID lookup before
returning deleted:true; backend failures or unverifiable deletion return
UNAVAILABLE. A timed-out deletion may already have completed: inspect the
collection before retrying. Enroll, append, and delete operations are serialized
within one service process; concurrent external backend writers or multiple
service replicas are not coordinated.
Deletion is irreversible removal from searchable records, not secure erasure.
VDMS v2.12 uses a collection-linked FindDescriptor query with exact UUID and
the native _deletion constraint to remove graph records. It may retain vector
bytes in descriptor index files; this API does not reclaim those files.
Qdrant uses a UUID-filtered point deletion with wait=true and verifies zero
remaining points. Backend backups and storage reclamation remain deployment
responsibilities. A JSON deletion audit event (action, UUID, collection,
deleted_by, audit ID, and timestamp) is emitted to service logs; deleted
descriptor audit properties are removed with their records. Retain the service
logs externally if a durable deletion audit is required.
VDMS can still rank deleted index vectors before graph lookup. Score-only hits
without an identity are discarded. The adapter expands the candidate search
up to 1000 index hits per vector to backfill usable matches, then returns at
most k_neighbors. If deleted hits exhaust that budget, fewer matches can be
returned. Index compaction/reclamation is not implemented by this service.
matches preserves the adapter’s per-vector result lists and similarity metric
semantics. An empty per-vector list means no usable match, not backend failure.
An absent gallery also returns one empty list per query vector. VDMS’s explicit
not-found responses are treated as empty lookups; unknown nonzero statuses,
malformed/incomplete transactions, and transport failures remain UNAVAILABLE.
VDMS schema lookup uses a read-only descriptor-set entity query to avoid the
v2.12 FindDescriptorSet missing-set error, and reads the schema marker for
the metric when available.
Schema metadata remains an array of existence, dimensions, and metric; unknown
backend metadata values may be null. Manual purge uses the existing TTL reclaim
semantics, not arbitrary deletion. Disabled TTL returns result: 0; successful
Qdrant purge returns null because its API does not supply a deleted count.
Health performs a read-only availability probe on every call. It does not
require a populated gallery or create a schema. A reachable backend returns
status: "ok", ready: true, and connected: true; a failed probe returns
status: "degraded", ready: false, and connected: false, still with an OK
gRPC status. A successful Health RPC establishes process liveness; callers must
inspect ready for readiness. IngestStats is local process state and remains
available during a backend outage. Its counts are process-local, not durable
gallery statistics.
VDMS readiness probes use a read-only entity lookup that succeeds on an empty database. After a failed probe, the adapter closes the stale connection and attempts to reconnect, rate-limited to once every five seconds. This supports recovery from startup outages and backend restarts without restarting the service. Reconnect does not create schemas or write markers. Connected VDMS sockets use a five-second I/O timeout.
gRPC status |
Meaning |
|---|---|
|
Valid operation completed, including empty matches, disabled ingest/TTL, absent schema, or degraded Health |
|
Missing, unsupported, or malformed request fields |
|
Collection-creation or POI-writer token is missing, invalid, or not configured |
|
Named collection exists with incompatible or unverifiable schema |
|
POI id does not exist in the named collection |
|
Matching, schema lookup, or purge failed against the backend; no fabricated empty success |
|
Unexpected service failure; private exception details are not returned |
|
RPC is not part of the baseline contract, including gallery-write methods |
Python Client#
reid.client.ReIDClient accepts a caller-owned channel, defaults to a five-second
deadline for every call, and propagates grpc.RpcError unchanged. The caller
owns channel security and closure; the client does not silently open a plaintext
connection. For example, when the endpoint has a TLS terminator:
from pathlib import Path
import os
import grpc
from reid.client import ReIDClient
credentials = grpc.ssl_channel_credentials(Path("/run/secrets/reid-ca.pem").read_bytes())
with grpc.secure_channel("reid.example.com:443", credentials) as channel:
client = ReIDClient(channel, timeout=5.0)
readiness = client.health()
metadata = client.find_schema_metadata()
statistics = client.ingest_stats()
collection_token = os.environ["REID_COLLECTION_CREATE_TOKEN"]
collection = client.create_collection(
"poi_person", 512, "IP", collection_token)
poi_token = os.environ["REID_POI_WRITE_TOKEN"]
enrollment = client.enroll_poi(
"poi_person", "person", [[0.1, 0.2, 0.3]], "high", "red jacket", "alice", poi_token)
Set REID_COLLECTION_CREATE_TOKEN to enable CreateCollection and
REID_POI_WRITE_TOKEN to enable EnrollPoi, AppendPoiEmbeddings, and DeletePoi; otherwise
the corresponding RPC returns PERMISSION_DENIED. Send each as
authorization: Bearer <token>.
The service listener itself remains plaintext on REID_SERVICE_PORT, so keep it
on a trusted private network or use an authenticated TLS proxy. The bearer token
authorizes collection declaration but does not encrypt transport. Restrict
manual purge at the deployment boundary as well.
Contract Verification#
Run the host-only transport tests with the test dependencies installed:
tests/.venv/bin/python -m pytest tests/sscape_tests/reid/test_api.py
make complexity-check SERVICE=reid
The tests send protobuf requests over a real loopback gRPC channel to the server with real VDMS and Qdrant adapters. Only database clients are mocked; success, invalid input, backend errors, outage/recovery, proto/client/handler parity, and absence of general-gallery writes are checked. This is not a live-database or deployment TLS verification.
Identity State and Counting#
The service publishes identity state and completed decisions on
scenescape/data/reid/{scene_id}/{category}. Messages retain track_id, gid,
matched, and similarity, and include reid_state and unique_id_count.
The four states preserve Controller behavior:
pending_collection: gathering features or waiting for a query result.matched: an accepted match has assigned the gid.query_no_match: a query completed without an accepted match, including live-gid collisions.reid_disabled: matching is disabled; tracks retain their local identity.
matched: false alone does not indicate a completed no-match decision. Consumers
must inspect reid_state. Analytics preserves these states in regulated and
region output, and marks undecided Tracker objects as pending_collection.
The cumulative counter is owned by each scene/category matcher and resets when the service restarts. It increments when a track is first registered without a usable embedding or while ReID is disabled, and when a query has no accepted match. A later no-match after an initially missing embedding increments it again, as in the Controller. A no-match result after a track leaves also increments it. It is not current occupancy or a count derived from terminal states.
IngestStats exposes per-matcher unique_id_counts and their aggregate in
counters.unique_id_count. Analytics forwards the applicable matcher count as
the existing unique_detection_count field in regulated scene messages.
Configuration#
The service uses REID_DATABASE (VDMS or QDRANT), REID_HOSTNAME,
REID_PORT, REID_USE_TLS, the existing Re-ID certificate variables, and
REID_DESCRIPTOR_TTL_SECS. REID_COLLECTION_CREATE_TOKEN and
REID_POI_WRITE_TOKEN enable their respective write operations. REID_SERVICE_PORT
selects the gRPC port and defaults to 50051.
The backend remains the Compose service reid; the extracted application is
reid-service so existing backend overrides continue to work.