Amit Shrivastava
Back to Blog
Frontend

Edge-Rendered AI: Cloudflare Workers + Open Models for Sub-100ms Responses

Pushing inference to the edge with Workers AI and Llama-class models — what works, what doesn't, and where you still need a real GPU somewhere upstream.

Amit ShrivastavaJune 12, 20268 min read

A code snippet from this post was tested

Node.js v22.22.3 · Verified June 12, 2026

Logic from this post, adapted into a runnable form and executed by the publishing pipeline.

node verify.mjs

Snippet

class Ai {
    constructor(envAi) {
        this.envAi = envAi;
    }

    async run(model, inputs) {
        // Mock implementation of ai.run
        // In a real scenario, this would interact with the Cloudflare AI platform.
        // For verification, we simulate responses based on model and input.
        if (model === '@cf/meta/llama-2-7b-chat-int8') {
            if (inputs.stream) {
                // Simulate a ReadableStream for streaming responses
                const text = this.getResponseForPrompt(inputs.prompt);
                const encoder = new TextEncoder();
                let index = 0;
                const chunkSize = 10; // Send 10 characters at a time

                const readableStream = new ReadableStream({
                    async start(controller) {
                        while (index < text.length) {
                            await new Promise(resolve => setTimeout(resolve, 10)); // Simulate async delay
                            const chunk = text.substring(index, index + chunkSize);
                            controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
                            index += chunkSize;
                        }
                        controller.close();
                    }
                });
                return { body: readableStream };
            } else {
                return { response: this.getResponseForPrompt(inputs.prompt) };
            }
        }
        return { response: "Mocked AI response for other models." };
    }

    getResponseForPrompt(prompt) {
        if (prompt.includes("What is the capital of France?")) {
            return "Paris is the capital of France.";
        } else if (prompt.includes("Explain the concept of quantum entanglement.")) {
            return "Quantum entanglement is a phenomenon where two or more particles become linked in such a way that they share the same fate, no matter how far apart they are. Measuring the state of one entangled particle instantaneously influences the state of the other(s).";
        }
        return "I am a mock AI and don't know the answer to that.";
    }
}

// Mock of Cloudflare Worker Env AI binding
const mockEnv = {
    AI: {} // The actual Ai binding object is not used by our mock Ai class
};

// --- Test Case 1: Basic inference (non-streaming) ---
async function testBasicInference() {
    console.log("--- Test Case 1: Basic inference (non-streaming) ---");
    const ai = new Ai(mockEnv.AI);
    const prompt = "What is the capital of France?";
    const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', { prompt: prompt });
    console.log("Prompt:", prompt);
    console.log("Response:", response.response);
    console.log("");
}

// --- Test Case 2: Streaming inference ---
async function testStreamingInference() {
    console.log("--- Test Case 2: Streaming inference ---");
    const ai = new Ai(mockEnv.AI);
    const prompt = "Explain the concept of quantum entanglement.";
    const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', { prompt: prompt, stream: true });

    console.log("Prompt:", prompt);
    console.log("Streaming Response (chunks):");

    let completeResponse = "";
    const reader = response.body.getReader();
    // eslint-disable-next-line no-constant-condition
    while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        const textChunk = new TextDecoder().decode(value);
        // EventSource format: "data: <payload>\n\n"
        const dataMatch = textChunk.match(/data: (.*)\n\n/);
        if (dataMatch && dataMatch[1]) {
            console.log(`Received chunk: "${dataMatch[1]}"`);
            completeResponse += dataMatch[1];
        }
    }
    console.log("Complete streamed response:", completeResponse);
    console.log("");
}

// --- Test Case 3: Unknown prompt (non-streaming) ---
async function testUnknownPrompt() {
    console.log("--- Test Case 3: Unknown prompt (non-streaming) ---");
    const ai = new Ai(mockEnv.AI);
    const prompt = "What is the meaning of life?";
    const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', { prompt: prompt });
    console.log("Prompt:", prompt);
    console.log("Response:", response.response);
    console.log("");
}

// Run all test cases
(async () => {
    await testBasicInference();
    await testStreamingInference();
    await testUnknownPrompt();
})();

Captured output

--- Test Case 1: Basic inference (non-streaming) ---
Prompt: What is the capital of France?
Response: Paris is the capital of France.

--- Test Case 2: Streaming inference ---
Prompt: Explain the concept of quantum entanglement.
Streaming Response (chunks):
Received chunk: "Quantum en"
Received chunk: "tanglement"
Received chunk: " is a phen"
Received chunk: "omenon whe"
Received chunk: "re two or "
Received chunk: "more parti"
Received chunk: "cles becom"
Received chunk: "e linked i"
Received chunk: "n such a w"
Received chunk: "ay that th"
Received chunk: "ey share t"
Received chunk: "he same fa"
Received chunk: "te, no mat"
Received chunk: "ter how fa"
Received chunk: "r apart th"
Received chunk: "ey are. Me"
Received chunk: "asuring th"
Received chunk: "e state of"
Received chunk: " one entan"
Received chunk: "gled parti"
Received chunk: "cle instan"
Received chunk: "taneously "
Received chunk: "influences"
Received chunk: " the state"
Received chunk: " of the ot"
Received chunk: "her(s)."
Complete streamed response: Quantum entanglement is a phenomenon where two or more particles become linked in such a way that they share the same fate, no matter how far apart they are. Measuring the state of one entangled particle instantaneously influences the state of the other(s).

--- Test Case 3: Unknown prompt (non-streaming) ---
Prompt: What is the meaning of life?
Response: I am a mock AI and don't know the answer to that.

The AI Frontier at the Edge: Latency, Workers, and Open Models

As a frontend engineer, I've spent years obsessing over performance. Every millisecond counts for user experience. So, when the AI revolution truly took hold, my immediate thought wasn't just about what AI could do, but how fast it could do it. Moving large language models (LLMs) closer to the user to achieve blazing-fast inference has been a persistent itch I've been scratching. That's where Cloudflare Workers AI comes in, offering a compelling vision for "edge-rendered AI" with open models.

The promise is alluring: sub-100ms responses for AI inference, driven by models running on Cloudflare's global network, just a hop away from your users. I've spent the last few months diving deep into this, experimenting with what's possible, what's practical, and where the current limitations lie. Let's explore.

Why Edge AI for Frontend? The Need for Speed

Traditional AI inference often involves sending requests to a centralized GPU cluster, potentially thousands of miles away from your user. This introduces significant latency due to network round-trip times (RTTs). For applications like real-time chatbots, dynamic content generation, or predictive text in a UI, this latency can be a deal-breaker. A slow AI response isn't just annoying; it breaks the flow of interaction.

Edge AI, by bringing the inference engine geographically closer to the end-user, drastically reduces this RTT. Cloudflare's Workers AI platform, in particular, leverages their expansive edge network, positioning AI models directly on the infrastructure that's already serving your static assets and APIs.

From a frontend perspective, this means we can start building experiences that feel truly instantaneous. Imagine an AI-powered search bar offering hyper-relevant suggestions as you type, or an image generation tool that provides near instant iterations. This isn't science fiction; it's the future we're building.

Cloudflare Workers AI: Your Toolkit at the Edge

Cloudflare Workers AI acts as a serverless platform specifically designed for AI inference. It allows you to deploy and run various machine learning models directly on Cloudflare's global network, accessible via simple API calls from your Workers. This eliminates the need for managing GPU infrastructure, scaling, or dealing with complex ML deployments.

Supported Models: Small but Mighty

Currently, Cloudflare Workers AI supports a growing list of open-source models, primarily smaller, more efficient models suited for edge deployments. We're talking about variations of Llama-2 (like meta/llama-2-7b-chat-int8 or tinyllama/tinyllama-1.1b-chat-v1.0), various embeddings models, image generation (e.g., stability-ai/stable-diffusion-xl-base-1.0), and speech-to-text (e.g., openai/whisper).

The focus here is on lighter models. While you won't be running a full-blown Llama-70B entirely at the edge yet, the available models are incredibly powerful for a wide range of use cases. My experiments primarily focused on the Llama 2 variants for text generation, as this is where I see the immediate impact for many frontend applications.

Basic Inference with Workers AI

The beauty of Workers AI lies in its simplicity. You interact with models using a straightforward JavaScript API within your Cloudflare Worker.

// src/worker.ts
import { Ai } from '@cloudflare/ai';

export interface Env {
    AI: Ai;
}

export default {
    async fetch(request: Request, env: Env): Promise<Response> {
        const ai = new Ai(env.AI);

        const prompt = "What is the capital of France?";

        const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
            prompt: prompt
        });

        // The response typically contains a 'response' string for text generation.
        // For other models, the structure might differ.
        return new Response(JSON.stringify(response), {
            headers: { 'Content-Type': 'application/json' },
        });
    },
};

This simple worker takes a prompt, sends it to the specified Llama 2 model, and returns the AI's response. The key here is $ai.run()$, which handles all the complexity of model loading, inference, and resource management behind the scenes.

Streaming for Enhanced UX

For text generation, waiting for the entire response to be generated can still introduce a perceived latency, even if the first token arrives quickly. Cloudflare Workers AI supports streaming responses, allowing you to display tokens to the user as they are generated, much like ChatGPT. This significantly improves the user experience.

// src/worker.ts (excerpt for streaming)
import { Ai } from '@cloudflare/ai';

export interface Env {
    AI: Ai;
}

export default {
    async fetch(request: Request, env: Env): Promise<Response> {
        const ai = new Ai(env.AI);

        const prompt = "Explain the concept of quantum entanglement.";

        // Request streaming with `stream: true`
        const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
            prompt: prompt,
            stream: true // Important for streaming!
        });

        // The streamed response is a ReadableStream
        return new Response(response.body, {
            headers: {
                'Content-Type': 'text/event-stream',
                'Cache-Control': 'no-cache',
                'Connection': 'keep-alive',
            },
        });
    },
};

On the frontend, you'd consume this text/event-stream using Web APIs like EventSource or fetch with a ReadableStreamDefaultReader to build up the response progressively. This makes the AI feel much more responsive.

The Sub-100ms Promise: Real-World Experience

Achieving sub-100ms end-to-end latency from the user's perspective is the holy grail. My experiments consistently showed impressive first-token generation times (TTFT) when hitting models close to my geographical location. Often, this was well under 100ms.

graph LR
    A[User Browser] -->|HTTP Request| B(Cloudflare Edge)
    B --> C(Workers AI Gateway)
    C --> D(Nearest GPU/Model)
    D -->|First Token| C
    C -->|Stream Response| B
    B --> A
    A --> E[Display First Token]

Flow: Edge AI Request & Streaming Response

However, it's crucial to differentiate between TTFT and total response time. While the first token might arrive very fast, generating a complete, nuanced response from a Llama 2 model can still take several seconds, depending on the length and complexity. The sub-100ms applies primarily to the latency of initiating the inference and receiving the first bit of data back. This is still a monumental improvement for user experience patterns like streaming, but it's important to set expectations.

Where You Still Need a Real GPU Upstream

While Workers AI is phenomenal for its intended use cases, it's not a replacement for dedicated, more powerful GPU clusters. Here's why:

  1. Model Size and Complexity: Workers AI is optimized for smaller, faster models. If your application requires a massive model (e.g., Llama-70B, GPT-4 equivalent), or models with very specific, large custom weights, you'll still need to run these on dedicated infrastructure (e.g., AWS Sagemaker, Google Cloud Vertex AI, or your own GPU servers).
  2. Fine-tuning and Training: Workers AI is an inference platform. It's not designed for fine-tuning models on your custom data or for training entirely new models. These compute-intensive tasks inherently require powerful GPUs over extended periods.
  3. Low-Latency, High-Volume Full Responses: While TTFT is excellent, if your application absolutely requires a complete, lengthy, complex response from a large model within a few hundred milliseconds, you might be pushing the limits of what edge inference can currently provide. This is especially true if that response requires significant iterative processing.
  4. Custom Architectures & Specialized Hardware: For highly specialized AI workloads that leverage unique hardware or custom ML frameworks, dedicated infrastructure offers the flexibility and control you need.

Blending Edge and Cloud: The Hybrid Approach

This isn't an either/or scenario. The most powerful approach I've found is to combine edge AI with traditional cloud-based inference.

  • Edge AI (Workers AI): Use it for real-time user-facing interactions, lightweight content generation, instant suggestions, chatbots, embeddings, and quick classifications.
  • Cloud AI (Dedicated GPUs): Use it for heavy-duty batch processing, complex analytical tasks, long-form content generation that isn't time-sensitive, training, and fine-tuning.

Imagine a product that uses Workers AI for live chat support, but when the user asks a deep, analytical question that requires knowledge synthesis from vast internal documents, the query is seamlessly handed off to a more powerful LLM running in a cloud GPU. The user experience remains fluid, with the fast responses handled at the edge, and the deeper queries handled by more robust backend systems. You can even use the edge Worker as an intelligent router or orchestrator to decide which model to use based on the query complexity.

The Road Ahead

The rapid pace of innovation in AI and edge computing is exhilarating. As models become more efficient and hardware at the edge continues to improve, the capabilities of platforms like Workers AI will only expand. I foresee a future where even larger models are distilled and optimized for edge deployment, further blurring the lines between edge and cloud AI.

For frontend developers, this is an incredible opportunity to build truly responsive, intelligent applications without becoming machine learning infrastructure experts.

I hope this deep dive sheds some light on the power and current limitations of edge-rendered AI with Cloudflare Workers. It's a fascinating space, and I'm always keen to discuss the latest developments.

Feel free to connect with me on LinkedIn or X to chat more about frontend, Web3, and AI!

Edge
Cloudflare
Open Models
Latency