Deployment workflow
After creating your handler function, package it into a Docker image and deploy it to an endpoint:1
Create a Dockerfile
Package your handler function and all its dependencies into a Docker image.
2
Deploy to an endpoint
Push your image and create an endpoint using one of two methods:
- Deploy from Docker Hub: Build locally and push to a container registry.
- Deploy from GitHub: Auto-build and deploy directly from your repository.
Model deployment
To deploy workers with AI/ML models, follow this order of preference:- Use cached models: For models on Hugging Face (public or gated), this is the recommended approach. Cached models provide the fastest cold starts and persist across worker restarts.
- Bake the model into your Docker image: For private models not on Hugging Face, embed them directly in your container image. This increases image size and initialization time, but ensures the model is stored directly on machine disk before the worker starts, so it’s ready to serve requests quickly after billing begins.
- Use network volumes: For development workflows or very large models (500GB+), store models on a network volume. Reading from network volume will occur after a worker starts running and billing begins, and will be slower than reading cached or baked models directly from machine disk. However, worker initialization time will be faster, and changes to the underlying image will not require rebuilding and repushing the image with the model artifact, making it faster to iterate during development.
Worker types
Workers can run in two modes depending on your latency and cost requirements:- Active workers run continuously (24/7) and are always ready to process requests instantly. They eliminate cold starts entirely, making them ideal for latency-sensitive or high-traffic applications.
- Flex workers scale dynamically based on demand, scaling down to zero when there are no requests. They incur cold starts when scaling up but cost nothing when not in use, making them ideal for variable or sporadic workloads.
Worker lifecycle
A worker is always either scaled down or running. Billing follows that split: a worker is billed only while it is running.- Scaled down: Runpod prepares workers ahead of time on a host by pulling your image and downloading cached models. No container is running for you, so nothing is billed.
- Cold start: When a request needs a worker, Runpod starts one and billing begins. Your container starts and your handler loads its model. The cold start ends when the handler is ready to take its first job.
- Processing: The worker runs jobs from the queue back to back.
- Waiting: When the queue is empty, the worker waits for the next job for up to the idle timeout. It is still running and still billed. A new job during this window starts immediately with no cold start.
- Scale down: If no job arrives before the idle timeout expires, the worker stops and billing stops. With FlashBoot, Runpod tries to keep the stopped worker in a state that restarts faster.
The console state Idle means the worker has shut down and is not billed. The idle timeout, is the period a worker stays running after a job finishes, waiting to pick up the next one. During this period, the worker is still billed and shows as Running in the console.
Worker states
The Workers tab shows each worker in one of these states:
A worker marked Outdated in the version column is running an older version of your endpoint. Runpod replaces it with a worker on the latest version. It is billed while it is running, like any other worker.
Image pulls are normally done while a worker is scaled down. If a worker starts before its image is on the host, the pull happens during the cold start and is billed. See compute cost breakdown.
If an endpoint repeatedly produces unhealthy workers, Runpod automatically scales it down; see My endpoint was scaled down unexpectedly.
Max worker limits
Account balance determines your maximum workers (flex + active combined):
Need more capacity? Contact support.