Overview Features How It Works Architecture Security Compliance
    Architecture

    The runtime beneath production inference.

    Gateway, control plane, dashboard, and runtime — the four components that schedule, snapshot, and isolate GPU workloads under the hood.

    <1s
    Cold Start
    90%+
    GPU Utilization
    30+
    Models / Node
    Piloting with infrastructure teams at frontier model labs and inference platforms.
    Key Features

    Built for how inference actually runs.

    Sub-Second Restore

    Restore full GPU memory state from snapshot in under a second. No cold container init on the critical path.

    Density at Scale

    Run dozens of models per node by eliminating idle GPU allocation — the scheduler packs and evicts on demand.

    Infrastructure, Not a Black Box

    Deploy on your own nodes or ours. InferX is the scheduling and isolation layer — it doesn't own your model or your data.

    How It Works

    Snapshot, store, restore.

    01

    Snapshot Creation

    We capture the entire GPU memory state of your model after initialization.

    02

    Optimized Storage

    Snapshots are compressed and stored efficiently in a distributed storage system.

    03

    Instant Restoration

    The exact GPU memory state is instantly restored, eliminating initialization time.

    Snapshot Cold Start Profiling
    Container Start 300–500 ms Snapshot metadata Loading 1–2 sec GPU/CPU Snapshot loading 0.5–3.5 sec 1st Request TTFT 0.5–1.5 sec TTFT 10s ms TTFT 10s ms ... 1st request Pre-Warm Cost: 200–400MB Blob GPU+CPU snapshot size, Memory CS-TTFT < 5 sec Container Start 5–10 sec Image Pull 15–25 sec Model Load to GPU 20–40 sec First Token 1–3 sec Cold start: 40–75+ sec, no snapshot restore
    CS-TTFT = Cold Start to First Token. Ranges vary by model size and snapshot size.
    openai.com/v1/chat/completions
inferx.net/v1/chat/completions
    Same request shape. No code changes to switch the endpoint.
    Platform Architecture

    Five components. One runtime.

    INFERENCE REQUEST / RESPONSE Gateway — receives inference requests and manages the client-facing API. Gateway Scheduler — decides which node and container serve each request, and when to warm or evict. Scheduler GPU NODE Standby container — allocated but not yet serving; restores from snapshot on demand. Inference Container (Standby) Container Runtime Warm container — snapshot restored, actively serving inference requests. Inference Container (Warm) Container Runtime Node Agent — manages container lifecycle and snapshot restoration on the GPU node. Node Agent Snapshot BlobStore — stores GPU/CPU state snapshots for fast restore. Snapshot BlobStore GPU #0 GPU #1 ··· GPU #N

    Gateway

    ⌄

    Routes inference requests to the appropriate runtime and manages the API interface for clients.

    Scheduler

    ⌄

    Optimizes resource allocation and determines when to warm up or scale down inference containers.

    Node Agent

    ⌄

    Manages container lifecycle and handles snapshot restoration on GPU nodes.

    Container Runtime

    ⌄

    Executes inference operations inside isolated containers with optimized GPU access.

    Snapshot BlobStore

    ⌄

    Stores model snapshots containing initialized model state for rapid loading.

    Snapshot Mechanism

    How CPU and GPU state moves through the blob store.

    Source container — capturing CPU and GPU state before it's written to the blob store. Inference GPU Container (Snapshotting) CPU State GPU State GPU GPU Snapshot — persisted CPU + GPU memory state in the blob store, restorable to any node. GPU Snapshot CPU State GPU State InferX Blob Store Destination container — restoring CPU and GPU state read back from the blob store. Inference GPU Container (Restore) CPU State GPU State GPU Full CPU + GPU memory state persisted as one addressable blob — restorable to any node in the pool.
    Secure Container Runtime

    Multi-tenant isolation, without sacrificing performance.

    InferX's secure multi-tenant architecture ensures full isolation between customer workloads while maximizing GPU utilization — through low-level containerization and memory-level enforcement.

    GUEST USER User Inference Process InferX NCCL Call Intercept library GUEST KERNEL Guest Kernel — intercepts NCCL calls from the tenant process before they leave the guest. InferX Container Runtime Guest Kernel HOST USER Host VMM — validates and mediates every NCCL call from the guest before it reaches host hardware. InferX NCCL Call Security Check InferX Container Runtime Host VMM (Virtual Machine Monitor) NCCL Call NCCL Resp Nvidia NCCL library — the underlying collective-communication library, unmodified. Nvidia NCCL library HOST KERNEL Linux Host Kernel — the real kernel and hardware, shared but never directly exposed to tenants. Linux Host Kernel 1. Compute isolation 2. Network isolation 3. GPU a. NCCL (library interface) i. NVLink (intra-node) ii. RDMA (inter-node) GPU NCCL Secure Isolation InferX VM Level Secure Container Runtime
    01

    Compute Isolation

    Complete workload isolation ensures security and prevents cross-contamination between models.

    02

    Network Isolation

    Separate virtual network environments for each customer's inference workloads.

    VPC · TCP/IP
    03

    GPU Isolation

    Secure access to GPU resources via NCCL library interface without compromising performance.

    NVLink (intra-node) · RDMA (inter-node)

    This allows cloud providers and enterprises to safely run multiple LLMs on shared hardware — securely and efficiently.

    Security & Compliance

    Built to sit inside an enterprise security perimeter.

    IN PROGRESS

    SOC 2 Type II

    Formal audit underway, covering the control environment across scheduling, storage, and access.

    ENFORCED

    Zero Data Retention

    Inference payloads are never persisted. Snapshots capture model state, not customer input or output.

    ENFORCED

    Tenant Isolation by Default

    Compute, network, and GPU isolation apply to every workload — no shared-tenancy opt-out.

    See how you can save 30-70% on your inference workloads?

    See how we deployed 60+ models on 2 GPUs →