ARCHITECTURE

    The GPU runtime built for serverless inference.

    InferX virtualizes CPU, CUDA, and multi-GPU communication state to snapshot, restore, migrate, and scale production inference workloads on demand.

    Deploy on your existing Kubernetes clusters and GPU infrastructure with multi-GPU snapshot/restore, secure workload isolation, rapid scale-out, and failure recovery.

    Multi-GPU Snapshot/RestoreGPU VirtualizationFork RestoreSecure NCCL IsolationOCI / CRI Compatible
    Deploy InferXTalk to Us

    GPUs weren't designed for serverless inference.

    Traditional GPU inference startup requires more than loading model weights. Large-model workloads must initialize the container and inference framework, load model state, initialize CUDA, establish multi-GPU communication, and prepare the runtime before serving traffic.

    Request
    Container Start
    Framework Initialization
    Model Weight Loading
    CUDA Initialization
    NCCL / Tensor Parallel Initialization
    Ready
    Making storage faster only solves part of the cold-start problem.

    For distributed models, initialized CPU, GPU, and communication state also have to be reconstructed before the workload can serve traffic.

    CORE TECHNOLOGY

    Virtualize the execution environment, not just the GPU.

    InferX provides a VM-level secure container runtime and GPU virtualization layer. Applications run through standard container interfaces while InferX controls the CPU and GPU execution environment underneath.

    By intercepting CUDA and NCCL interactions, InferX can manage execution state that conventional container orchestration does not control.

    Runtime internals
    • InferX secure container runtime
    • Guest execution environment
    • GPU virtualization
    • CUDA interception
    • NCCL interception
    • KVM guest kernel
    • Host VMM
    • Linux host kernel
    • OCI compatibility
    • CRI compatibility

    Snapshot the running model, not just its weights.

    Once an inference workload is initialized, InferX can capture its execution state so it can be restored without repeating the full startup path.

    Resume instead of reinitialize.

    This allows inactive workloads to return to service without rebuilding the entire inference environment from scratch.

    DISTRIBUTED INFERENCE

    Cold start doesn't stop at one GPU.

    Large models commonly span multiple GPUs using tensor parallelism. Those GPU processes share coordinated execution and communication state through NCCL.

    InferX's GPU virtualization layer is designed to preserve and restore multi-GPU execution state, including NCCL-dependent workloads.

    One snapshot. Four production primitives.

    The same snapshot primitive powers elasticity, mobility, and recovery.

    Stop keeping every model resident on a GPU.

    Traditional serving often requires GPU memory and compute capacity to remain reserved for models even when traffic is low. InferX allows inactive workloads to remain in snapshot form and consume GPU capacity when they are needed.

    Traditional

    Model ADedicated GPU
    Model BDedicated GPU
    Model CDedicated GPU
    Model DDedicated GPU

    Idle models still occupy GPU capacity.

    InferX

    Shared GPU PoolActive workloads use GPUsInactive workloads remain as snapshots

    GPU capacity is assigned when workloads need to run.

    SECURITY

    Isolation doesn't stop at the container boundary.

    Distributed GPU workloads introduce a communication plane beyond normal compute and network isolation. InferX extends the security boundary into GPU communication.

    InferX validates NCCL communication at the runtime boundary before it reaches the host GPU communication domain.

    PLATFORM

    From API request to GPU.

    Gateway

    Routes inference traffic and detects whether a serving instance is available.

    Scheduler

    Selects placement and triggers workload startup when new capacity is required.

    Node Agent

    Manages local GPU resources, model state, snapshots, and workload lifecycle.

    InferX Runtime

    Executes and isolates inference workloads while managing CPU and GPU execution state.

    Snapshot Store

    Stores restorable workload state locally or remotely.

    DEPLOYMENT

    Your Kubernetes. Your GPUs. InferX underneath.

    InferX deploys into existing Kubernetes environments while managing the inference data plane independently from the cluster's general-purpose orchestration.

    Kubernetes remains the infrastructure orchestration layer. InferX manages model state, GPU placement, snapshot lifecycle, runtime execution, and inference-specific scheduling.

    KubernetesPrivate CloudBare MetalGPU Cloud

    Turn existing GPU infrastructure into an on-demand inference platform.

    Deploy InferX across your Kubernetes clusters, private cloud, bare metal, or GPU cloud infrastructure.

    Talk to InferXExplore Deployment