Amit Shrivastava
Back to Blog
DevOps & Tools

Production Prompt Engineering: Versioning, A/B Testing, and Rollbacks

Treating prompts like code: how I version, test, and roll back prompt changes the same way I would a database migration — and the tooling that makes it boring.

Amit ShrivastavaJune 26, 20269 min read

Production Prompt Engineering: Treating AI Prompts Like Code

Hey everyone! Amit here. Over the past 10+ years as a software engineer, I've seen countless technologies evolve. From the early days of web development to the complexities of Web3 and now the incredible potential of AI, one truth remains constant: reliable systems require robust engineering practices. As I've delved deeper into AI and prompt engineering, I've realized that the same rigor we apply to traditional code – version control, testing, deployment pipelines – is just as crucial, if not more so, for managing prompts in production.

This isn't about some theoretical future. This is about what I do today to ensure our AI-powered applications are stable, performant, and continuously improving. I want to share how I version, A/B test, and rollback prompt changes with the same confidence I'd handle a database migration. And honestly? The goal is to make it boring – because boring, in software engineering, often means reliable.

The Problem with "Just Changing a Prompt"

Before we dive into solutions, let's acknowledge the common pitfalls. How many times have you heard or done this: "The model is generating weird outputs for X, let me just tweak the prompt a bit in the UI." Or, "Let's try adding 'always be concise' to the instruction."

This ad-hoc approach, while seemingly fast, quickly leads to chaos:

  • No History: You lose track of what prompted a good or bad change. What was the exact prompt that worked best last week?
  • No Reproducibility: If a bug surfaces, you can't easily revert to a known good state.
  • No Collaboration: Multiple team members tweaking prompts independently is a recipe for disaster.
  • No Testing: Changes go live without validation, potentially degrading performance or introducing regressions.
  • No Measurement: How do you objectively know if your "tweak" actually improved things?

These are the same problems we solved for traditional code decades ago. It's time to apply those lessons to prompts.

Versioning Prompts: Git for Your AI's Brain

The cornerstone of treating prompts like code is version control. Just as git manages our source code, it should manage our prompts.

I store my prompts as plain text files (or JSON/YAML for more complex structured prompts) within my project's repository. Each prompt has a unique identifier and lives in a dedicated directory, say src/prompts/.

Consider a simple prompt for a content summarization task:

// src/prompts/summarizeArticle/v1.txt
Please summarize the following article concisely, focusing on the main arguments and key takeaways.
Article:
{article_content}

When I need to iterate, I create a new version:

// src/prompts/summarizeArticle/v2.txt
You are an expert journalist. Summarize the following article in no more than 3 sentences, capturing the essence and most important facts.
Article:
{article_content}

And in my application code, I'll have a utility to load the current or specified version:

// src/server/utils/promptLoader.ts
import fs from 'fs';
import path from 'path';

const PROMPT_DIR = path.resolve(__dirname, '../../prompts');

export const loadPrompt = (promptName: string, version: string = 'current'): string => {
    let promptPath = path.join(PROMPT_DIR, promptName);
    if (version === 'current') {
        // In a real system, 'current' would be determined by a config or DB entry
        // For simplicity here, let's assume 'v2' is current for now.
        // A more robust solution might read from a 'current.txt' symlink or a config file.
        promptPath = path.join(promptPath, 'v2.txt'); // Or dynamically load highest version
    } else {
        promptPath = path.join(promptPath, `${version}.txt`);
    }

    if (!fs.existsSync(promptPath)) {
        throw new Error(`Prompt not found: ${promptPath}`);
    }
    return fs.readFileSync(promptPath, 'utf-8');
};

// Example usage:
// const summarizationPrompt = loadPrompt('summarizeArticle');
// const specificVersionPrompt = loadPrompt('summarizeArticle', 'v1');

This ensures every prompt change is a git commit, with an associated message, diff, and PR review process.

A/B Testing Prompts: Data-Driven Iteration

Knowing which version of a prompt is better requires data. This is where A/B testing becomes indispensable. We don't just deploy a new prompt and cross our fingers; we deploy it alongside the old one and measure its impact.

The process I use looks something like this:

  1. Define Metrics: What does "better" mean? Lower latency? Higher user satisfaction (explicit feedback)? Fewer hallucinations (human evaluation)? More accurate extractions? This is application-specific.
  2. Instrument Your Application: When a prompt is used, log its version, the input, the output, and any relevant user interaction or downstream metrics.
  3. Implement a Gateway/Router: This component decides which prompt version to serve to a given user or request.

Here's a simplified conceptual example of a prompt gateway:

// src/server/services/promptGateway.ts
import { loadPrompt } from '../utils/promptLoader';
import { logPromptUsage } from '../utils/telemetry'; // Assume this exists

interface PromptConfig {
    name: string;
    variants: {
        version: string;
        weight: number; // e.g., 80 for v1, 20 for v2
    }[];
}

const promptConfigs: Record<string, PromptConfig> = {
    summarizeArticle: {
        name: 'summarizeArticle',
        variants: [
            { version: 'v1', weight: 80 }, // 80% traffic
            { version: 'v2', weight: 20 }, // 20% traffic
        ],
    },
    // ... other prompts
};

export const getExperimentPrompt = (promptName: string, userId?: string): string => {
    const config = promptConfigs[promptName];
    if (!config) {
        throw new Error(`Prompt config not found for ${promptName}`);
    }

    // A more sophisticated implementation would use userId or a cookie for sticky assignments
    // and rely on a proper A/B testing framework.
    const rand = Math.random() * 100;
    let cumulativeWeight = 0;
    let selectedVersion = config.variants[0].version; // Default to first

    for (const variant of config.variants) {
        cumulativeWeight += variant.weight;
        if (rand <= cumulativeWeight) {
            selectedVersion = variant.version;
            break;
        }
    }

    logPromptUsage({ promptName, version: selectedVersion, userId, timestamp: new Date() });
    return loadPrompt(promptName, selectedVersion);
};

// Then in your API handler:
// const prompt = getExperimentPrompt('summarizeArticle', req.user.id);
// const completion = await processWithLLM(prompt, articleContent);

This allows gradually rolling out new prompt versions to a small percentage of users, gathering data, and making an informed decision before a full rollout. If the new version performs better, we update the weights to 100% for the new version and eventually deprecate the old one. If it performs worse, we easily kill the experiment.

Rollbacks: Your Safety Net

Despite all our testing, sometimes things go wrong in production. A new prompt might behave unexpectedly under specific, niche conditions, or introduce a subtle bias. This is where an easy rollback mechanism saves the day.

Because our prompts are versioned in Git and our application code loads specific versions, rolling back is as simple as:

  1. Updating the current pointer: In a small setup, this might be updating a single configuration file that dictates which prompt version is "live." For example, changing a prompt_versions.json file in S3 or a ConfigMap in Kubernetes.
  2. Re-deploying (if necessary): If your promptGateway logic reads from a static config, a redeploy will be required. If it reads from a dynamic source (like a remote config service), it might be instant.
  3. Git Revert: In a scenario where the entire deployment pipeline is tied to Git, reverting the commit that introduced the problematic prompt version and initiating a new deployment is the most robust approach. The prompt files themselves are then rolled back alongside the code.

The key is that the old, stable version of the prompt is always available and easily retrievable.

The Workflow: A Boring, Reliable Pipeline

Here's how I envision and often implement the end-to-end process:

graph TD
    A[Engineer Creates New Prompt Version (e.g., v3)] --> B{Git Commit & Pull Request};
    B --> C{Automated Tests/Linting};
    C --> D{Code Review};
    D -- Approved --> E[Merge to Main Branch];
    E --> F[Trigger CI/CD Pipeline];
    F --> G[Deploy to Staging Environment];
    G --> H[Manual/Automated Evaluation on Staging];
    H -- Pass --> I[Deploy to Production (A/B Test 10% traffic)];
    I --> J[Monitor Production Metrics (Performance, Qual. Feedback)];
    J -- Good Results --> K[Increase A/B Test Weight (e.g., 50% traffic)];
    J -- Bad Results --> L[Rollback to previous version (v2)];
    K -- Good Results --> M[Full Production Rollout (100% v3 traffic)];
    M --> N[Archive Old Prompt Version (v2)];

This diagram illustrates a process that is strikingly similar to how we manage standard code deployments. The only difference is the focus on prompt content and LLM-specific evaluation metrics.

Tooling That Makes It Boring

While the core principles are simple, good tooling accelerates this process:

  • Version Control: git is non-negotiable.
  • Prompt Management Libraries: For complex prompt structures, consider libraries that help parameterize and compose prompts (e.g., LangChain's PromptTemplates, custom internal solutions).
  • A/B Testing Frameworks: For sophisticated rollouts and statistical significance, integrate with existing A/B testing platforms or build a purpose-built prompt serving layer.
  • Observability & Telemetry: Log all prompt inputs, outputs, versions, and relevant metrics. Tools like Grafana, Prometheus, or custom dashboards are crucial for monitoring.
  • Evaluation Frameworks: Automate prompt evaluation where possible. For RAG systems, measure recall, precision. For summarization, use ROUGE scores (though human evaluation is still king for nuanced tasks). LLM-as-a-judge can also help automate initial assessments.
  • Configuration Management: Store active prompt versions and A/B test weights in a centralized, dynamic configuration service (e.g., AWS AppConfig, Consul, or even just a well-managed database table).

Conclusion: Engineering Excellence for AI

Treating prompts as first-class citizens in your engineering workflow isn't just about best practices; it's about building reliable, scalable, and continuously improving AI systems. By applying version control, A/B testing, and robust rollback strategies, you transform prompt engineering from an art into a more predictable and engineering-driven discipline. This approach allows for rapid iteration, data-backed decisions, and ultimately, more robust and valuable AI-powered products.

It makes prompt changes boring in the best possible way: predictable, controlled, and resilient.

_If you're also navigating the fascinating world of production AI and prompt engineering, I'd love to connect and share more insights!_

Connect with me: LinkedIn | X

Prompt Engineering
MLOps
A/B Testing
Deployment