Switches version; unavailable chapters open its introduction.

Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Benchmarking

A catalogue of contextdb’s benchmark targets: what each measures, which measurement profile runs it, and the rules for writing new ones. Benchmarks are descriptive. The product’s acceptance bars are listed in the performance guide; this page does not define, relax or replace them.

How to run benchmarks

All performance run commands — profiles, preparing clean checkouts, baseline and candidate runs, comparison and publication — are in docs/performance/README.md. They are not repeated here.

Targets

Every target is declared in its crate’s Cargo.toml and uses Criterion. scripts/ci-performance.py assigns targets to profiles and is authoritative if this table and the script disagree. Targets marked all only run only in the all profile.

CrateTargetProfileWhat it measures
contextdb-parserparser_throughputall onlyParser-only microbenchmark.
contextdb-engineengine_throughputvector (filter nv_)Local engine behaviour; its nv_ cases cover named-vector indexes.
contextdb-engineengine_full_throughputstorageLarger write/reopen and mixed workflow paths.
contextdb-enginestorage_scale_hardeningstorageTable-local index maintenance, reopen locality, wide-row commit batch size and file-backed single-row autocommit. Its fixtures assert locality counters before timing, so a failure is a correctness/locality failure first.
contextdb-enginerank_policy_recallvectorRank-policy recall.
contextdb-enginemixed_workflows_prworkflowsBounded mixed graph, relational and vector workflows.
contextdb-enginesubscriptions_prworkflowsBounded subscription fan-out and backpressure workloads.
contextdb-serversync_prworkflowsBounded sync workloads through direct sync APIs, without CLI/process overhead.
contextdb-serverserver_throughputsystemCLI throughput.
contextdb-serverserver_sync_systemsystemSystem sync with authenticated Iroh and file-backed CLI/server processes; needs the release binaries the system profile builds first.
contextdb-serverblob_transferall onlyBlob transfer.
contextdb-engineindexed_scan_filter_pk_inserts, indexed_scan_filter_file_backed_open, indexed_scan_filter_in_memory_pk_inserts, indexed_scan_filter_entity_list, indexed_scan_filter_non_id_pk_insertsall onlyIndexed scan and filter paths.
contextdb-engineacceptance_three_paradigm_query, sql_surface_distinct, graph_adjacency_probe, composite_index_routingall onlyQuery-surface and routing paths; see each source.
contextdb-enginetier1_atomic_visibility_under_contention, tier1_event_bus_throughput, tier1_context_guard_overhead, tier1_scope_label_overhead, tier1_acl_overheadall onlyVisibility, event bus, context, scope-label and ACL overheads.
contextdb-enginetier2_commit_validate_overhead, tier2_fk_revalidation_overhead, tier2_state_machine_overheadall onlyCommit validation, foreign-key revalidation and state-machine overheads.
contextdb-enginecommit_write_volume_scaling, sigterm_drain_in_flight_persistence, trigger_throughput, version_cleanup_scaling (feature test-seams)all onlyCommit volume scaling, shutdown drain persistence, trigger throughput and version cleanup scaling.

Some sources assert internal budgets or counters (for example a millisecond budget inside a bench). Those assertions are real checks, but they are not product acceptance bars.

Design rules

  • Keep setup outside timed loops unless startup cost is the thing being measured.
  • Keep benchmarks bounded enough to avoid Criterion stretching runs into multi-minute samples.
  • Derive system benchmarks from acceptance and integration scenarios rather than synthetic isolated loops.
  • If a benchmark spends its whole run on setup or build noise and never reaches useful Criterion output, treat that as a benchmark-shape problem to fix, not a valid result.