OmniSentient Technical Platform Specification¶

OmniSentient is an AI Developer Platform and Supply-Chain Security Operating System. It is engineered to solve two fundamental software engineering challenges: reducing context errors in repository code understanding through deterministic, AST-grounded retrieval, and automating supply-chain vulnerability remediation under strict Human-In-The-Loop (HITL) governance.

1. System Overview & Architecture¶

1.1 Platform Purpose & Dual-Engine Thesis¶

Modern software repositories contain deeply interconnected architectures across hundreds of files. Standard AI chatbots and general-purpose LLM interfaces lack repository-level ground truth, frequently hallucinating invalid imports, missing architectural constraints, and inventing nonexistent API signatures.

OmniSentient operates on a Dual-Engine Architecture:

1. Code Intelligence Engine (RAG Subsystem): Parses repository source code into structural Abstract Syntax Trees (ASTs), indexes code units as dense vector embeddings, and streams grounded, citation-backed answers via Server-Sent Events (SSE).
2. Emergency Security Remediation (ESR) Engine: Ingests repository events (pushes, pull requests, CI pipeline alerts), correlates dependencies against known CVE databases, computes impact blast radius, and proposes remediation strictly as standard Pull Requests requiring human approval.

Both engines are constrained by the Governance Spine—a server-authoritative policy layer backed by a tamper-evident, hash-chained audit ledger.

┌───────────────────────────────────────────────────────────────────────────┐
│                               OMNISENTIENT                                │
│                                                                           │
│   ┌───────────────────────────┐           ┌───────────────────────────┐   │
│   │  CODE INTELLIGENCE ENGINE │           │     SECURITY ENGINE (ESR) │   │
│   │  • AST Syntax Parsing     │           │  • Webhook Ingestion      │   │
│   │  • Hybrid Retrieval       │           │  • CVE & Blast Radius     │   │
│   │  • Cross-Encoder Reranker │           │  • Automated PR Fixes     │   │
│   │  • Grounded Token Stream  │           │  • Human Review (HITL)    │   │
│   └─────────────┬─────────────┘           └─────────────┬─────────────┘   │
│                 │                                       │                 │
│                 └───────────────────┬───────────────────┘                 │
│                                     │                                     │
│   ┌─────────────────────────────────┴─────────────────────────────────┐   │
│   │                         GOVERNANCE SPINE                          │   │
│   │   Server-Side RBAC · Policy Enforcement · Audit Ledger (SHA-256)  │   │
│   └───────────────────────────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────┘

1.2 Quick Start & Local Setup¶

Prerequisites¶

• Python 3.10+
• Supabase Account & PostgreSQL Database (with pgvector enabled)
• Google Gemini API Key (for embedding generation and language models)
• GitHub Personal Access Token (PAT) or GitHub App installation

Installation Steps¶

1. Clone the repository:
   bash git clone https://github.com/shubham5728/omnisentient.git cd omnisentient

2. Install core and development dependencies:
   bash pip install -r requirements.txt pip install -r requirements-dev.txt

3. Configure environment variables:
   bash cp .env.example .env # Populate SUPABASE_URL, SUPABASE_KEY, GEMINI_API_KEY, # GITHUB_APP_ID, GITHUB_PRIVATE_KEY, and GITHUB_WEBHOOK_SECRET

4. Launch the local platform server:
   bash python run.py
   The platform dashboard will initialize at http://localhost:3000.

2. Code Intelligence Engine (RAG Subsystem)¶

The Code Intelligence Engine extracts, indexes, retrieves, and synthesizes repository knowledge without breaking syntactic boundaries.

2.1 AST Parsing & Structural Boundary Preservation¶

Traditional text chunking splits files by character length or line count, which frequently severs functions in half and destroys parameter signatures. OmniSentient uses Python's native ast module (located in services/ai/chunking/ast_python.py) to parse source code into an Abstract Syntax Tree.

• Class-level and function-level isolation. ClassDef, FunctionDef and AsyncFunctionDef nodes are lifted whole, using each node's lineno and end_lineno so a chunk always begins and ends on a real syntactic boundary.
• Hierarchical context attribution. A method inside a class is stored under the symbol path ParentClass.method_name, alongside its file path, decorator list, and — for classes — its base classes. Retrieval can therefore answer "which class does this belong to" without re-parsing.
• Token budgeting. Each extracted node is measured before emission and compared against the chunker's max_tokens, so an oversized function is handled deliberately rather than silently truncated mid-body.

# src/omnisentient/services/ai/chunking/ast_python.py
class ASTPythonChunker(BaseChunker):
    def chunk(self, file_path: str, content: str) -> List[Chunk]:
        ...

The parser is Python's own ast, so the boundaries it produces are the language's boundaries — not a heuristic approximation of them.

2.2 Retrieval Pipeline¶

Code queries benefit from two complementary search paradigms:
1. Exact Identifier Search (Sparse / BM25): Locating specific variable names, error strings, or route paths (e.g. auth_token_v2, 401_UNAUTHORIZED).
2. Conceptual Architectural Search (Dense): Finding logic related to high-level intentions (e.g. "Where is session expiration handled?").

Current deployed configuration — The live query endpoints (/api/rag/query and /api/rag/query/stream) use dense-only retrieval: code chunks are embedded with models/gemini-embedding-001 and retrieved by cosine similarity from Supabase pgvector.

Designed retrieval architecture — The pipeline (services/ai/retrieval/) is built to support full hybrid fusion:

• Sparse Engine (BM25): scores exact keyword occurrences across tokenized codebase symbols (services/ai/retrieval/bm25.py).
• Dense Engine (Vectors): converts code chunks into vector representations via gemini-embedding-001.
• HybridRetriever: fuses both result sets using Reciprocal Rank Fusion (services/ai/retrieval/hybrid.py).

Hybrid mode is available in the pipeline and will be enabled in a future release.

2.3 Reciprocal Rank Fusion (RRF) & Cross-Encoder Reranking¶

When hybrid retrieval is active, the results from the Dense and Sparse retrievers are fused using Reciprocal Rank Fusion (RRF):

$$RRF_Score(d \in D) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

Where $M$ is the set of retrieval engines (Dense, Sparse), $r_m(d)$ is the rank of document $d$ in engine $m$, and $k$ is a smoothing constant (default: 60). Implementation in services/ai/retrieval/hybrid.py.

An optional reranking pass (services/ai/rerankers/cross_encoder.py) refines candidate ordering after fusion by scoring query–document pairs. Despite the module name, it is not a locally hosted cross-encoder performing cross-attention — it issues the relevance judgement to gemini-1.5-flash. The distinction matters for latency and cost: every rerank is a model call, not a local forward pass.

Current deployment status: Both RRF fusion and Cross-Encoder reranking are implemented but not active in the current deployed endpoints, which use dense-only retrieval (see §2.2).

2.4 Server-Sent Events (SSE) Streaming Protocol¶

LLM completions stream token-by-token using unidirectional Server-Sent Events (SSE) rather than WebSockets, simplifying connection management and firewall traversal.

The non-streaming sibling, POST /api/rag/query, runs the same pipeline and returns one JSON document instead. See §5.1.

3. Emergency Security Remediation (ESR)¶

The ESR engine operates as a supply-chain firewall, triaging dependency vulnerabilities and proposing non-breaking updates.

1  Repository webhook event      Push, PR or CI signal; HMAC-SHA256 verified
              ↓
2  Vulnerability ingestion       Manifest changes matched against known CVEs
              ↓
3  Blast radius analysis         Is the vulnerable symbol reached in your AST
              ↓
4  Remediation proposal          Patch synthesised on an isolated branch
              ↓
5  Pull request dispatch         PR-scoped token; never a direct commit to main
              ↓
6  Human review and approval     Merge authority stays with your branch protection

3.1 Webhook Ingestion & Vulnerability Correlation¶

When a GitHub push, PR, or CI workflow event occurs, the webhook dispatcher validates the HMAC-SHA256 signature (integrations/github/provider.py). If manifest changes (package.json, requirements.txt, Pipfile) or CI failures are detected, the payload is forwarded to the vulnerability correlation engine.

3.2 Blast Radius & Impact Scoping¶

The system maps dependencies across all repositories within the organization to determine the blast radius:

• Identifies downstream internal packages dependent on the affected library.
• Evaluates whether the vulnerable symbol/method is actively called in your repository's AST.

3.3 Human-In-The-Loop (HITL) Pull Request Delivery¶

Absolute Security Constraint: OmniSentient never commits directly to protected branches. Remediations follow a strict lifecycle:
1. Fix authored on an isolated branch (omnisentient/fix-<cve-id>).
2. GitHub Pull Request opened with dependency diffs, security advisories, and rollback instructions.
3. Organization engineers inspect, test, and approve the PR before merge.

4. Governance Spine & Trust Architecture¶

The Governance Spine establishes operational boundaries, multi-party controls, and tamper-evident auditing across all platform actions.

4.1 Tamper-Evident Hash-Chained Audit Ledger¶

Security-relevant records carry a SHA-256 hash that binds each row to its predecessor, so deleting or editing history breaks the chain at verification time. There are two independent chains, written by different layers.

Incident chain — written in the database. A BEFORE INSERT trigger on incidents reads the previous row's hash, stores it as prev_hash, and signs the new row. Because the trigger runs inside PostgreSQL, an application bug cannot skip it.

-- database/schema/INSTITUTIONAL_ENFORCEMENT.sql
NEW.prev_hash := last_hash;
NEW.row_hash  := encode(digest(
    concat(NEW.id, '|', NEW.org_id, '|', NEW.event_code, '|', NEW.detected_at, '|', last_hash),
    'sha256'
), 'hex');

Audit-log chain — written in the application. audit_logs rows carry a monotonic chain_seq alongside the hash:

# src/omnisentient/domain/shared/db.py
raw_payload = f"{next_seq}|{prev_hash}|{created_at}|{org_id}|{actor_canonical}|{action}|{payload_json}"
row_hash = hashlib.sha256(raw_payload.encode('utf-8')).hexdigest()

Metadata is serialised with json.dumps(..., sort_keys=True, separators=(',', ':')) so the payload is canonical and the hash is reproducible.

Reading the ledger. AuditService (services/audit/audit_service.py) queries and formats ledger entries and produces forensic snapshots; it does not compute the per-row chain hashes. Anchor manifests for third-party witnessing live in domain/crypto/anchors.py and are specified in docs/GOVERNANCE_SPINE_V1_SPEC.md.

Status — incident chain installed; audit_logs chain still unsigned.
Checked against the deployed database rather than the schema files.

• The incidents table, the chaining trigger and the immutability policies
  are now applied (supabase/migrations/20260817_incidents_ledger.sql), and
  verify_incident_chain() runs and returns VALID. It has nothing to
  check yet: no incidents have been recorded, so the trigger has not signed
  a row and the chain is unexercised end to end.
• audit_logs remains unsigned. All nineteen rows are null in row_hash,
  prev_hash and chain_seq. The production path calls the
  append_audit_log RPC, which inserts without chaining; the hashing shown
  above runs only on the local fallback branch.

So one chain is ready and untested, the other is writing unsigned rows.
Neither yet supports a claim of tamper-evidence.

4.2 Server-Authoritative Role Hierarchy¶

Authorization is enforced server-side in api/middleware/auth_guard.py, before any handler runs. UI controls reflect server permissions but are never the authorization boundary.

Roles are a linear hierarchy, not a capability matrix. Each endpoint declares a minimum role, and access is granted when the caller's rank meets or exceeds it:

# src/omnisentient/api/middleware/auth_guard.py
ROLE_HIERARCHY = ['auditor', 'viewer', 'developer', 'security_lead', 'admin', 'owner']

def can_access(user_role, required_role='viewer'):
    return role_level(user_role) >= role_level(required_role)

Because the check is a rank comparison, a role also inherits everything declared below it. auditor sits at rank 0, so an endpoint guarded by role='auditor' admits every authenticated role in the org — it is a floor, not an exclusive grant.

Two protections sit alongside the rank check and do not depend on it:

• Viewer mutation block. POST, PATCH, PUT and DELETE are refused outright for the viewer role at org scope, regardless of the endpoint's declared minimum.
• Tenant binding. The organization is read from the verified JWT claim, never from the request. If a caller also passes an explicit ?org=, it must match the session tenant; a mismatch is logged as CROSS_TENANT_ATTEMPT and denied.

4.3 Multi-Tenant Data Isolation¶

Isolation is enforced in the application tier. The platform reaches PostgreSQL with a service-role credential, so Row-Level Security is not the boundary on these paths — the scope is applied before the query is issued, and a guard rejects any that arrives without it.

• Tenant resolved from the session. The organization comes from the verified JWT claim, never from the request. A client-supplied ?org= that disagrees is denied and logged as CROSS_TENANT_ATTEMPT (api/middleware/auth_guard.py).
• Scope required at the data layer. Queries against tenant tables — audit_logs, events, projects, organizations, esr_jobs, dependency_findings and the rest — must carry org_id=eq.<org>. In production a guard in integrations/supabase/client.py rejects any that does not, rather than trusting each call site to remember. Row-Level Security is enabled on these tables as a second layer.
• Repository scoping on retrieval. Code intelligence answers are built from the repository named in the request, which is resolved against the caller's own repositories before retrieval runs.
• Path Traversal Guardrails: Local file reading endpoints strictly canonicalize absolute paths to prevent directory traversal (../) or symlink escaping.

5. API & Protocol Reference¶

OmniSentient provides a REST and streaming API. A machine-readable OpenAPI 3.0.3 specification is available at openapi.yaml.

5.0 Authentication¶

All API endpoints authenticate via a session cookie, not a Bearer token.

After completing GitHub OAuth, the server issues a signed JWT in an httpOnly, Secure, SameSite=Lax cookie named omni_session. The browser attaches this cookie automatically on all same-origin requests. There is no separate Bearer token endpoint for external API access.

# Cookie is set by the server after GitHub OAuth callback:
Set-Cookie: omni_session=<signed_jwt>; HttpOnly; Secure; SameSite=Lax; Path=/

# Subsequent API requests send the cookie automatically (browser):
Cookie: omni_session=<signed_jwt>

For programmatic access outside a browser, you must first authenticate through the OAuth flow and capture the resulting cookie, then include it with each request.

A missing or invalid cookie returns HTTP 401. An expired session redirects browser requests to /login and returns HTTP 401 for API paths.

5.1 Code Intelligence Query Endpoints¶

Two endpoints answer repository questions. They take the same request and differ only in how the answer is delivered — choose by whether you want to render tokens as they arrive.

Both are registered under the /api/rag prefix and both require an authenticated session (see §5.0).

Rate limit: 3 600 requests per hour per authenticated user. Exceeding the limit returns HTTP 429 RATE_LIMITED.

Request¶

POST /api/rag/query/stream
Content-Type: application/json
Cookie: omni_session=<signed_jwt>

{
  "query": "How does the audit ledger verify hash chain integrity?",
  "repo_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "session_id": "<optional-uuid>"
}

A missing or non-JSON body returns INVALID_REQUEST (HTTP 400).

Streaming response¶

The streaming endpoint emits four event types:

HTTP/1.1 200 OK
Content-Type: text/event-stream

event: token
data: {"text": "The"}

event: token
data: {"text": " incident"}

event: done
data: {"citations": [...], "request_id": "...", "latency_ms": 842}

Clients must handle error as a terminal event: the stream closes after it, and no done follows.

5.2 Audit Ledger API¶

GET /api/auditor/incidents
Cookie: omni_session=<signed_jwt>

Returns the calling organization's incident ledger, newest first, with each row's chain hash attached. Restricted to the auditor, admin and owner roles — the rank comparison in §4.2 cannot express this on its own, so the ledger endpoints carry an explicit allowlist.

The endpoint takes no query parameters and is not paginated. The org filter is applied server-side from the verified session (incidents?org_id=eq.<org>); it is not a client-supplied argument.

Response¶

A bare JSON array of incident rows:

[
  {
    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "org_id": "8c1d...",
    "severity": "high",
    "event_code": "DEPENDENCY_CVE_DETECTED",
    "status": "resolved",
    "detected_at": "2026-08-14T18:30:00Z",
    "resolved_at": "2026-08-14T19:02:11Z",
    "prev_hash": "e3b0c442...",
    "row_hash": "9f2b71c8..."
  }
]

An empty array is returned when the query fails as well as when there are genuinely no incidents, so an empty result is not by itself proof of a clean ledger.

Export¶

GET /api/auditor/export/incidents
Cookie: omni_session=<signed_jwt>

Same scope and role, delivered as CSV for forensic archival. The export carries a header block recording export time and the requesting actor, then the columns id, org_id, severity, event_code, status, detected_at, resolved_at, row_hash.

6. Core System Invariants & Verification¶

6.1 Architectural Guarantees¶

Each guarantee below names the code that enforces it, so it can be checked rather than taken on trust.

1 — Remediation reaches your default branch only through review.
The remediation path has no direct-commit step. It creates a branch ref, then opens a pull request against it (POST repos/{repo}/pulls). Merge authority stays with the organization's own branch protection.
Enforced in: integrations/github/client.py, services/platform/esr_service.py

2 — Answers carry their sources.
The generation stream terminates with a done event carrying citations for the chunks the answer was built from. Chunks come from AST boundaries, so a citation points at a real function or class, not an arbitrary text window.
Enforced in: api/routes/rag.py, services/ai/chunking/ast_python.py

3 — Ledger rows are hash-chained at write time.
Incident rows are signed by a PostgreSQL BEFORE INSERT trigger, so the chain is written below the application and an application bug cannot skip it. audit_logs rows carry a separate application-written chain with a monotonic sequence number.
Enforced in: database/schema/INSTITUTIONAL_ENFORCEMENT.sql, domain/shared/db.py
Caveat: the incident chain verifier does not currently reproduce the trigger's hash formula — see §4.1. Writing is sound; checking is not yet.

4 — Tenant scope is taken from the session, never the request.
Org-scoped endpoints resolve the tenant from the verified JWT claim. A client-supplied ?org= that disagrees is denied and logged as CROSS_TENANT_ATTEMPT. Auditor queries apply org_id=eq.<org> server-side.
Enforced in: api/middleware/auth_guard.py, api/routes/audit.py

5 — Webhook payloads are authenticated before they are processed.
GitHub deliveries must carry a valid X-Hub-Signature-256, verified with hmac.compare_digest against an HMAC-SHA256 of the raw body. If no webhook secret is configured, delivery is rejected outside debug mode rather than trusted.
Enforced in: integrations/github/provider.py

GitHub App scope. Installation tokens are checked for the contents, pull_requests and metadata permissions before use (integrations/github/auth.py). Write access is required to open remediation pull requests.

6.2 Test Suite Execution & Verification¶

The suite runs under pytest, with ruff for linting:

# Unit and integration tests
pytest tests/ -v

# Streaming and RAG specific tests
pytest tests/api/test_rag_streaming.py -v

# Style and lint
ruff check src/

Current state — this suite does not pass green. The streaming tests fail
on authentication (HTTP 401 against /api/rag/query/stream), which points at
missing test credentials rather than at the endpoint. A broader run shows
further failures, and ruff check src/ currently reports a large backlog.
Treat these commands as the way to run the suite, not as evidence that the
tree is verified — that claim would need the suite to be green first.