InferX: AI Lambda-Like Inference Function as a Service
InferX is an advanced serverless inference platform engineered for ultra-fast, efficient, and scalable deployment of AI models. It enables serving multiple models within a single environment, achieving cold start times of less than two seconds.
With InferX, you can:
Whether you're running LLMs, vision models, or custom AI pipelines, InferX delivers unmatched speed and efficiency for next-gen AI applications.
Ultra Fast Cold Start
Cold start GPU-based inference in under 2 seconds for large models (12B+).
GPU Slicing
Allocate only a fraction of a GPU (e.g., 1/3 GPU) per model to efficiently run multiple workloads in parallel.
Super High Model Deployment Density
Serve hundreds of models on a single node (e.g. 30 models and 2 GPUs in the demo), maximizing hardware utilization.
80+% GPU Utilization
Based on Just-In-Time scaling and super high deployment density, we can achieve 80% GPU utilization.
Lambda-like AI Serving
Automatically scale AI inference workloads with on-demand execution.
Optimized Performance
Reduce latency, improve cost efficiency, and streamline AI inference at scale.
How It Works
The InferX platform follows a streamlined process to deliver fast, efficient AI inference:
InferX Cold Start Process
How InferX achieves < 2 second cold starts
Snapshot Generation
When a model is uploaded, InferX captures a complete memory snapshot of the fully initialized model, including GPU memory state, CPU memory state, and runtime context.
Pre-warm Phase
Before any inference requests arrive, InferX loads the snapshot into CPU memory or high-speed Blobstore. During this phase, the model consumes zero GPU resources.
Inference Request
When a request arrives, the API Gateway checks for a warm instance. If none exists, it signals the scheduler to perform a cold start using the pre-warmed snapshot.
Ultra-Fast Cold Start (< 2 seconds)
The snapshot is loaded directly into GPU memory at PCIe bandwidth speeds. Unlike normal startups, there's no GPU initialization or model loading from disk, enabling cold starts in under 2 seconds.
Traditional Cold Start
2-10+ min
Loading time from disk to GPU
InferX Cold Start
< 2s
Direct snapshot restoration at PCIe speeds
Upload & Deploy
Users upload their inference docker image, and InferX provides a unique URL. The system generates a snapshot of the container with both CPU and GPU states.
Pre-Warm Process
Before inference requests arrive, InferX loads the GPU snapshot into CPU memory or InferX Blobstore with ultra-high throughput (PCIe bandwidth, tens or hundreds of GB/s per node). During this phase, the model consumes ZERO GPU resources.
Inference Request Processing
When a request arrives, InferX checks for a warm instance. If available, the request is processed immediately with minimal latency.
Ultra-Fast Cold Start
If no warm instance exists, InferX loads the snapshot GPU state to GPU memory from CPU memory or Blobstore. Unlike normal cold starts, there's no GPU initialization, and the start latency is limited only by GPU PCIe bandwidth. This enables cold starts in under 2 seconds, even for 12B+ parameter models.
Platform Architecture
InferX AI Function Platform is an AI-native serverless inference platform that serves AI models as AWS Lambda-like functions.
InferX Platform Architecture
A Lambda-like AI Function-as-a-Service platform
API Gateway
OpenAI-compatible API interface for model inference
Dashboard
User interface for monitoring and management
Scheduler
Resource allocation and container management
Blobstore
High-speed storage for model snapshots
Model Containers
Warm and cold start container instances
GPU Resources
Shared GPU hardware with 90% utilization
Key Performance Indicators:
< 2s
Cold Start Time
90%
GPU Utilization
3500%
Deployment Density
Dashboard
The graphical user interface exposed to users for model management, deployment monitoring, and performance tracking.
API Gateway
The API interface that supports OpenAI-compatible REST APIs for seamless integration with existing applications.
Scheduler
A cluster-level scheduler that assigns warm instances to server requests if they exist, or initiates cold starts when needed, optimizing resource allocation across the platform.
Blobstore
A high-throughput blob storage system specifically designed to store model snapshots and enable rapid transfer of container snapshot data to GPU and CPU memory during restoration.
Advantages Over Traditional Systems
The industry average for GPU utilization in Kubernetes inference platforms is typically only 10-20%. InferX solves the core limitations that cause this inefficiency:
✅ Handling Sporadic Requests
Traditional platforms leave GPU-based inference instances idle when there are no requests. InferX co-deploys multiple models (potentially from multiple tenants) in a shared cluster, and since peak demand times rarely overlap, this dramatically decreases idle time.
✅ Improving High Availability
Traditional platforms must deploy 1-4 idle inference instances for failover and redundancy, especially with 5+ minute cold start latencies. InferX's ultra-fast cold starts eliminate the need for standby instances.
✅ Enabling True Auto-Scaling
For busy inference systems with inconsistent workloads, traditional platforms require idle instances for fast scale-out due to high cold start latency. InferX's sub-2-second cold starts enable true on-demand scaling without pre-provisioning.
Ready to Transform Your AI Inference?
Whether you're running LLMs, vision models, or custom AI pipelines, InferX delivers unmatched speed, efficiency, and cost savings for your next-gen AI applications.