Execution model

​

What runs where

import asyncio
from runpod_flash import Endpoint, GpuType

@Endpoint(name="demo", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090)
def process_on_gpu(data):
    # This runs on Runpod worker
    import torch
    return {"result": "processed"}

async def main():
    # This runs on your machine
    result = await process_on_gpu({"input": "data"})
    print(result)  # This runs on your machine

if __name__ == "__main__":
    asyncio.run(main())  # This runs on your machine

​

Flash apps

• FastAPI server runs locally.
• @Endpoint functions run on Runpod workers.

• Each endpoint configuration becomes a separate Serverless endpoint.
• All endpoints run on Runpod.

​

Execution flow

​

Endpoint naming

@Endpoint(
    name="inference",  # This identifies the endpoint
    gpu=GpuType.NVIDIA_A100_80GB_PCIe,
    workers=3
)
def run_inference(data): ...

• Same name, same config: Reuses the existing endpoint.
• Same name, different config: Updates the endpoint automatically.
• New name: Creates a new endpoint.

​

Worker lifecycle

​

Worker states

​

Scaling behavior

@Endpoint(
    name="demo",
    gpu=GpuGroup.ANY,
    workers=(0, 5),   # (min, max) - Scale to zero with no jobs, up to 5 workers
    idle_timeout=60   # Seconds a running worker waits for a new job before scaling down
)
def process(data): ...

1. First job arrives → Scale to 1 worker (cold start).
2. More jobs arrive while worker busy → Scale up to max workers.
3. Jobs complete → Workers keep running and wait up to idle_timeout seconds for new jobs. They are billed while they wait.
4. No new jobs → Workers scale down to the minimum and billing stops for the ones that stopped.

​

Cold starts and warm starts

​

Cold start

• You’re calling an endpoint for the first time.
• All workers have been scaled down after not processing requests for idle_timeout seconds.
• All running workers are busy processing requests.

1. Runpod provisions a new worker with your configured GPU/CPU.
2. The worker image starts (dependencies are pre-installed during build).
3. Your function executes.

​

Warm start

• Worker completed a previous job and is waiting for more work.
• Worker is within its idle_timeout period.

1. Job is routed immediately to the waiting worker.
2. Your function executes.

​

The relationship between configuration and starts

• workers=(0, n): Workers scale to zero when not processing. Every request after the idle_timeout period triggers a cold start.
• workers=(1, n): At least one worker stays ready. First concurrent request is warm, additional requests may cold start.
• Higher idle_timeout: Workers stay running longer before scaling down, reducing cold starts for sporadic traffic.