System Architecture

TruePortAI is a Defensible AI Gateway — an enterprise-grade, multi-tenant platform that sits between your applications and commercial LLM providers. It provides real-time governance, complete audit trails, ML-powered violation detection, and full data sovereignty.

C4 Level 1 — Context Diagram

C4Context title TruePortAI — System Context Person(devUser, "Developer / AI Engineer", "Sends AI requests via SDK or HTTP") Person(adminUser, "Platform Admin", "Manages tenants, billing, system config") Person(tenantAdmin, "Tenant Admin", "Manages API keys, policies, team members") System(trueportai, "TruePortAI Platform", "AI Gateway, Governance & Audit Engine") System_Ext(openai, "OpenAI API", "GPT-4o, GPT-3.5-turbo") System_Ext(anthropic, "Anthropic API", "Claude 3.5, Claude 3") System_Ext(google, "Google Gemini API", "Gemini 1.5 Pro / Flash") System_Ext(azure, "Azure OpenAI", "Enterprise GPT deployments") System_Ext(s3, "AWS S3 / Azure Blob", "Tenant-owned raw interaction storage") System_Ext(smtp, "SMTP / SES", "Email notifications and alerts") System_Ext(stripe, "Stripe", "Billing and subscriptions") Rel(devUser, trueportai, "Routes LLM requests through", "HTTPS/REST") Rel(tenantAdmin, trueportai, "Configures keys, policies, views analytics", "HTTPS") Rel(adminUser, trueportai, "Manages tenants, billing, system settings", "HTTPS") Rel(trueportai, openai, "Proxies requests to", "HTTPS") Rel(trueportai, anthropic, "Proxies requests to", "HTTPS") Rel(trueportai, google, "Proxies requests to", "HTTPS") Rel(trueportai, azure, "Proxies requests to", "HTTPS") Rel(trueportai, s3, "Streams raw logs to", "HTTPS/S3 API") Rel(trueportai, smtp, "Sends alerts via", "SMTP/TLS") Rel(trueportai, stripe, "Manages billing via", "HTTPS")

C4 Level 2 — Container Diagram

graph TB subgraph "Client Layer" APP["Client Application<br/>(any OpenAI-compatible SDK)"] BROWSER["Browser<br/>(Angular Dashboard)"] WWW["Public Website<br/>(Next.js)"] end subgraph "CDN / Edge" CF["AWS CloudFront<br/>Global CDN + TLS termination"] end subgraph "trueportai-services | AWS Lambda + Python FastAPI" GW["AI Gateway / Proxy<br/>Route: /trueportai-services/{provider}/..."] TSAPI["TruePortAI REST API<br/>Route: /trueportai-services/api/v1"] GWMW["Gateway Middleware<br/>Auth · RateLimit · AuditLog"] CB["Circuit Breaker<br/>(Redis-backed)"] RL["Rate Limiter<br/>(Redis fixed-window)"] end subgraph "platform-backend | AWS Lambda + Python FastAPI" AUTH["Auth Service<br/>JWT · OTP · Password Reset"] ORGAPI["Organization API<br/>Tenants · Members · Invites"] CFGAPI["Config API<br/>Hierarchical Config Mgmt"] SUBAPI["Subscriptions & Catalog<br/>Plans · Products · Billing"] STOR["Storage Service<br/>S3 · Azure · GCP abstraction"] ADMINAPI["Admin API<br/>Platform-level management"] MAILSVC["Mail Service<br/>SMTP · Templates"] end subgraph "trueportai-analytics | On-Premise / VPC Python" INGST["S3 Event Listener<br/>(Trigger / Polling)"] MLPIPE["ML Pipeline<br/>PII · Bias · Injection · Exfil"] VIOWRT["Violation Writer<br/>MongoDB + Alert Publisher"] end subgraph "Data Layer" MONGODB[("MongoDB Atlas<br/>saas_platform_core<br/>trueport_ai")] REDIS[("Redis<br/>Rate limits · Circuit state")] S3STORE[("AWS S3 / Azure Blob<br/>Raw interaction JSON")] end subgraph "External LLMs" OPENAI["OpenAI API"] ANTHROPIC["Anthropic API"] GOOGLE["Google Gemini API"] AZURE_AI["Azure OpenAI"] end APP --> CF BROWSER --> CF WWW --> CF CF --> GW CF --> TSAPI CF --> AUTH CF --> ORGAPI GW --> GWMW GWMW --> RL GWMW --> CB CB --> OPENAI & ANTHROPIC & GOOGLE & AZURE_AI GWMW --> S3STORE GWMW --> MONGODB TSAPI --> MONGODB AUTH --> MONGODB ORGAPI --> MONGODB CFGAPI --> MONGODB SUBAPI --> MONGODB STOR --> S3STORE MAILSVC --> SMTP_EXT["SMTP / SES"] S3STORE --> INGST INGST --> MLPIPE MLPIPE --> VIOWRT VIOWRT --> MONGODB VIOWRT --> MAILSVC RL --> REDIS CB --> REDIS style GW fill:#7B68EE,color:#fff style MLPIPE fill:#FF6B6B,color:#fff style MONGODB fill:#50C878,color:#fff style REDIS fill:#FF9500,color:#fff style S3STORE fill:#4A90D9,color:#fff

Solution Repositories

Repository

Technology

Deployment

Responsibility

platform-backend

Python 3.11 · FastAPI · Uvicorn

AWS Lambda (Zappa)

Auth, RBAC, Billing, Mail, Config

trueportai-services

Python 3.11 · FastAPI · Uvicorn

AWS Lambda (Zappa)

AI Gateway Proxy, Key Mgmt, Usage Analytics API

trueportai-analytics

Python 3.11 · HuggingFace · Triton

On-Premise / VPC GPU Node

ML violation detection pipeline

trueportai-ui

Angular 17+

AWS S3 + CloudFront

AI Governance dashboard (SPA)

platform-ui

Angular 17+

AWS S3 + CloudFront

Platform admin dashboard (SPA)

trueportai-www

Next.js 14+

AWS S3 + CloudFront (Static Export)

Public website + this documentation

Implementation Stack

Layer

Technology

Purpose

API Runtime

Python 3.11 + FastAPI + Uvicorn

Async REST APIs

ASGI → Lambda

Mangum (AWS Lambda adapter)

Serverless hosting

ODM (Async)

Motor + Beanie

MongoDB async driver

Password Hashing

Argon2-cffi

Credential security

Token Auth

python-jose (HS256 JWT)

Session management

Proxy HTTP

httpx (async)

Upstream LLM forwarding

Rate Limiting

Redis (fixed-window)

Per-key throttling

Circuit Breaker

Redis (state machine)

Provider resilience

Frontend

Angular 17 + TailwindCSS

Platform dashboards

Public Site

Next.js 14 + TailwindCSS

Marketing + docs

Database

MongoDB Atlas (M10+)

Operational data

Storage

AWS S3 / Azure Blob

Audit log archive

CDN

AWS CloudFront

Global edge delivery

ML Inference

NVIDIA Triton Inference Server

GPU-accelerated models

ML Models

HuggingFace Transformers

RoBERTa, DeBERTa, DistilBERT

Key Workflows

1. AI Request Flow

flowchart LR A[Client App] -->|"POST /trueportai-services/openai/v1/chat/completions\nx-api-key: sk-..."| B[CloudFront] B --> C[TruePortAI Gateway] C --> D{API Key\nValid?} D -->|No| E[401 Unauthorized] D -->|Yes| F{Rate Limit\nOK?} F -->|Exceeded| G[429 Too Many Requests] F -->|OK| H{Circuit\nBreaker Open?} H -->|Open| I[503 Service Unavailable] H -->|Closed| J[Forward to Provider] J --> K[LLM Provider API] K --> L[Response Stream] L --> M[Return to Client] L --> N([Async: Log to MongoDB]) L --> O([Async: Stream to S3]) O --> P([Analytics Engine triggers])

2. Violation Detection Pipeline

flowchart TD A[Raw JSON in S3] -->|S3 Event / Polling| B[Analytics Ingest] B --> C[Deserialize Payload] C --> D{Policy Rules\nMatch?} D -->|Yes| E[Instant Violation Record] C --> F[ML Pipeline — Parallel] F --> G[PII/PHI NER\nRoBERTa] F --> H[Bias / Toxicity\nDeBERTa-v3] F --> I[Prompt Injection\nDistilBERT + Heuristics] F --> J[Exfil Guard\nRegex + Entropy] G & H & I & J --> K{Any\nViolations?} K -->|No| L[Update log: clean=true] K -->|Yes| M[Write Violation Records\nto MongoDB] M --> N[Send Alert Email\nvia Platform Mail API] M --> O[Update Dashboard\nMetrics]

3. Authentication & Tenant Selection Flow

flowchart LR A[User Login] --> B[POST /auth/login] B --> C{Credentials\nValid?} C -->|No| D[401] C -->|Yes| E[Global JWT Issued\nsub = user_id] E --> F[GET /tenants\nList user's organizations] F --> G[User Selects Tenant] G --> H[POST /auth/select-tenant] H --> I[Scoped JWT Issued\nsub, tid, role, env] I --> J[Access Platform APIs\nwith scoped token] J --> K{Need env change?} K -->|Yes| L[POST /auth/switch-environment] L --> M[New scoped token\nfor dev / staging / prod]

Multi-Tenancy Architecture

TruePortAI uses a logical multi-tenancy model: all tenants share the same database cluster but are isolated via tenant_id fields and middleware-enforced scoping.

graph TD subgraph "Request enters" REQ[Incoming Request] end subgraph "TenantContextMiddleware" M1{Skip global\n/auth routes?} M2{JWT has\ntid claim?} M3{Subdomain\npresent?} M4{X-Tenant-Slug\nheader?} M5[Resolve Tenant\nfrom DB] M6[Attach tenant_id\nto request.state] M7[403 — Tenant\nUnresolved] end REQ --> M1 M1 -->|Yes| PASS[Pass through] M1 -->|No| M2 M2 -->|Yes| M5 M2 -->|No| M3 M3 -->|Yes| M5 M3 -->|No| M4 M4 -->|Yes| M5 M4 -->|No| M7 M5 --> M6

Tenant context resolution priority:

  1. JWT tid claim (production flows)

  2. HTTP subdomain (e.g., acme.trueportai.com)

  3. X-Tenant-Slug header (local/dev override)

Security Architecture

graph TD subgraph "Authentication" AT1[Email + Password → JWT] AT2[OTP / Magic Code → JWT] AT3[Password Reset → Temp Token → JWT] AT4[OAuth2 → Google / LinkedIn / GitHub] end subgraph "Authorization (RBAC)" AZ1[Platform Admin — System-wide] AZ2[Tenant Owner — Full tenant control] AZ3[Tenant Admin — Manage members & keys] AZ4[Member — Use keys, view own analytics] AZ5[Viewer — Read-only analytics] end subgraph "Gateway Security" GS1[API Key Validation sk-...] GS2[Rate Limiting 60 RPM default] GS3[Circuit Breaker 5-failure threshold] GS4[PII Redaction on response] GS5[Prompt Injection Block] end subgraph "Data Security" DS1[Argon2 password hashing] DS2[Short UUID — no ObjectId exposure] DS3[Tenant-isolated storage paths] DS4[Encrypted storage credentials] DS5[HTTPS everywhere CloudFront TLS] end

Reliability Patterns

Circuit Breaker State Machine

stateDiagram-v2 [*] --> CLOSED : Service starts CLOSED --> OPEN : 5 consecutive failures OPEN --> HALF_OPEN : 30s timeout expires HALF_OPEN --> CLOSED : Request succeeds HALF_OPEN --> OPEN : Request fails CLOSED --> CLOSED : Success resets counter

Provider Failover Strategy

Primary Provider → [Circuit Breaker] → Secondary Provider → [Circuit Breaker] → Error Response

Each provider has an independent circuit breaker key in Redis:

  • circuit_breaker:openai:state

  • circuit_breaker:anthropic:state

  • circuit_breaker:google:state

  • circuit_breaker:azure:state

Rate Limiting Algorithm

Fixed Window per API key per minute:

Redis Key: rate_limit:{api_key}:{YYYY-MM-DD-HH-MM}
Redis Ops: INCR → if count > limit → 429; else EXPIRE 60s