Amit.dev
Back to Blog
AI & Agents

AI Agent Memory: Short-Term, Long-Term, and Episodic

Understanding the different types of memory in AI agents and how to implement them for better context retention.

Amit ShrivastavaApril 17, 202611 min read

The AI Agent's Elephant in the Room: Mastering Memory for Smarter Interactions

As a software engineer who's spent a decade wrestling with frontend complexities, diving deep into Web3's decentralized wonders, and now building intelligent AI agents, one thing has become abundantly clear: an AI without good memory is like a human with amnesia. It might be brilliant at problem-solving in the moment, but it lacks the contextual understanding that truly elevates raw intelligence to meaningful interaction.

If you've played around with large language models (LLMs), you've likely hit the "context window limit" wall. That's the AI's short-term memory, and it's surprisingly finite. For our agents to be truly useful – to engage in ongoing conversations, learn from past experiences, and adapt over time – we need to go beyond this fleeting recollection. We need to equip them with robust memory systems.

In this post, I want to demystify AI agent memory. We'll explore the three main types: short-term, long-term, and episodic, and I'll share practical strategies and code snippets (mostly in TypeScript/JavaScript) to help you implement them in your own AI agent projects.

Short-Term Memory: The Immediate Context

Think of an AI agent's short-term memory as its working memory. It's the information it can access right now to respond to the current prompt. For most LLMs, this is primarily handled by the context window.

How It Works (and Why It's Limited)

When you send a prompt to an LLM, the entire conversation history (or a truncated version of it) is often packed into that context window. The model then processes this entire window to generate its next response.

The limitation here is fundamental: computational cost. Processing longer contexts is exponentially more expensive and slower. This is why even models with massive context windows (like 128k tokens) still have practical limits for real-time interaction. It's also why LLMs can "forget" things mentioned much earlier in a long conversation.

Implementing Short-Term Memory

For most basic applications, simply passing the conversation history is enough.

// Example using a hypothetical LLM API client
interface Message {
  role: 'user' | 'assistant';
  content: string;
}
let conversationHistory: Message[] = [];
async function chatWithAI(newMessage: string): Promise {
  conversationHistory.push({ role: 'user', content: newMessage });
  // Keep history within a reasonable token limit (e.g., last N messages)
  // This is a naive approach; a better one measures actual token count.
  if (conversationHistory.length > 10) {
    conversationHistory = conversationHistory.slice(conversationHistory.length - 10);
  }
  const response = await llmClient.complete({
    model: "gpt-4o",
    messages: conversationHistory,
  });
  const aiMessage = response.choices[0].message.content;
  conversationHistory.push({ role: 'assistant', content: aiMessage });
  return aiMessage;
}
// Usage:
// const firstResponse = await chatWithAI("Hello, who are you?");
// const secondResponse = await chatWithAI("Tell me more about your capabilities.");

For more sophisticated short-term memory, you might use techniques like:

  • Summarization during context window management: Before passing old messages, summarize them to save tokens.
  • Hierarchical context: Store a high-level summary of the entire conversation, and fetch detailed older chunks only if explicitly requested or deemed relevant.

Long-Term Memory: Knowledge That Lasts

Long-term memory is where your AI agent stores information that persists across sessions, learns from experiences, and builds a comprehensive knowledge base. This is crucial for agents that need to remember user preferences, learn new facts, or maintain a consistent persona.

The Role of Vector Databases

The most common and effective way to implement long-term memory in AI agents today is using a vector database. Here's the basic idea:

  1. Embeddings: Every piece of information (text, images, audio, etc.) is converted into a numerical representation called a vector embedding. These embeddings capture the semantic meaning of the data. Similar meanings result in similar vectors.
  2. Storage: These vectors are stored in a specialized database optimized for high-dimensional vector similarity search.
  3. Retrieval: When the AI needs to recall information, the input query (e.g., the user's question) is also converted into an embedding. The vector database then efficiently finds the most "similar" (closest in vector space) stored embeddings.
  4. Context Augmentation: The retrieved information is then fed back into the LLM's short-term context window, augmenting its knowledge for the current response. This is often called RAG (Retrieval Augmented Generation).

Practical Implementation with Vector Databases

Here's a simplified conceptual example using a vector database like Pinecone, Weaviate, or ChromaDB (though langchain/vectorstore abstracts this nicely):

// Conceptual code – specific API calls vary by vector store
import { OpenAIEmbeddings } from '@langchain/openai'; // Or any other embedding model
import { Chroma } from "@langchain/community/vectorstores/chroma"; // Or Pinecone, Weaviate, etc.
import { Document } from '@langchain/core/documents';
interface AgentKnowledge {
  topic: string;
  details: string;
}
// Initialize embedding model and vector store
const embeddings = new OpenAIEmbeddings();
let vectorStore: Chroma; // Will be initialized asynchronously
async function initializeVectorStore() {
  vectorStore = await Chroma.fromExistingCollection(embeddings, {
    collectionName: "agent-knowledge",
  });
  console.log("Vector store initialized.");
}
async function addKnowledge(items: AgentKnowledge[]) {
  const docs = items.map(item => new Document({
    pageContent: item.details,
    metadata: { topic: item.topic }
  }));
  await vectorStore.addDocuments(docs);
  console.log(Added ${items.length} knowledge items.);
}
async function retrieveKnowledge(query: string, k: number = 3): Promise {
  const relevantDocs = await vectorStore.similaritySearch(query, k);
  return relevantDocs.map(doc => doc.pageContent);
}
// --- Usage Example ---
// await initializeVectorStore();
// // Add some long-term knowledge
// await addKnowledge([
//   { topic: "agent_purpose", details: "My purpose is to assist users with software engineering questions." },
//   { topic: "user_preference", details: "User Amit prefers TypeScript for code examples." },
//   { topic: "company_policy", details: "Company policy on sensitive data forbids sharing PII." }
// ]);
// async function augmentedChat(prompt: string): Promise {
//   const relevantInfo = await retrieveKnowledge(prompt);
//   const augmentedPrompt = Based on the following context:\n${relevantInfo.join('\n')}\n\nUser: ${prompt};
//   // Now, send augmentedPrompt to your LLM client along with short-term history
//   // (This part combines with short-term memory logic shown above)
//   return await llmClient.complete({
//     model: "gpt-4o",
//     messages: [...conversationHistory, { role: 'user', content: augmentedPrompt }],
//   });
// }

Key Takeaway for Long-Term Memory: The power of RAG is that you don't need to fine-tune your LLM for specific knowledge. You store external information, and the LLM leverages its reasoning capabilities on that information to formulate responses. This makes your agents adaptable and keeps costs down compared to constant fine-tuning.

Episodic Memory: Learning from Experience

Episodic memory is akin to a human remembering specific events, experiences, and the context in which they occurred. For an AI agent, this means recalling not just what happened, but when, where, and why it happened. This is crucial for:

  • Learning from mistakes: "Last time the user asked X, my advice led to Y problem, so next time I should try Z."
  • Understanding recurring patterns: "Every Tuesday, this user asks about project A, indicating a weekly update cycle."
  • Personalization: "The user seemed frustrated when I used too much technical jargon in our last conversation about cloud architecture."

Episodic memory often leverages similar technologies to long-term memory (vector databases for semantic search), but with an emphasis on the event itself and its rich context.

Capturing and Retrieving Episodes

Instead of just storing facts, you store narratives or observations about interactions.

interface Episode {
  timestamp: Date;
  eventType: 'user_query' | 'agent_action' | 'feedback' | 'observation';
  agentId: string;
  userId: string;
  content: string; // The query, action description, feedback, or observation
  sentiment?: 'positive' | 'negative' | 'neutral'; // Optional sentiment analysis
  relevantDocsRetrieved?: string[]; // What long-term knowledge was used
  outcome?: 'success' | 'failure' | 'neutral'; // Was the interaction successful?
}
// Function to store an episode (might use a vector store or a simple database)
async function recordEpisode(episode: Episode) {
  // Option 1: Store in a simple NoSQL DB (e.g., MongoDB, DynamoDB) for easy querying by timestamp/userId
  // Option 2: Embed 'content' + 'eventType' into a vector and store in vector DB for semantic retrieval
  // For simplicity, let's assume a basic record operation.
  console.log(Recorded episode: ${episode.eventType} at ${episode.timestamp.toISOString()});
  // In a real system, you'd save this to a database.
}
// Function to retrieve relevant episodes
async function retrieveEpisodes(query: string, userId: string, k: number = 3): Promise {
  // This is where a vector database shines.
  // Embed the query and search for similar episodes related to the user.
  // For demonstration, let's simulate.
  console.log(Retrieving episodes for user ${userId} related to: ${query});
  // In a real system, you'd query your vector DB for similar episode content, potentially filtering by user and time.
  const mockEpisodes: Episode[] = [
    {
      timestamp: new Date("2023-11-01T10:00:00Z"),
      eventType: "user_query",
      agentId: "my_agent",
      userId: userId,
      content: "Can you help me debug my React component? It's not rendering.",
      outcome: "neutral"
    },
    {
      timestamp: new Date("2023-11-01T10:15:00Z"),
      eventType: "agent_action",
      agentId: "my_agent",
      userId: userId,
      content: "Suggested checking dev tools console and verifying state updates. User seemed confused.",
      sentiment: "negative",
      outcome: "failure" // My previous advice might have been unclear
    },
    {
      timestamp: new Date("2023-11-08T09:30:00Z"),
      eventType: "user_query",
      agentId: "my_agent",
      userId: userId,
      content: "I'm having a render issue with a complex React component again, but this time it's related to prop drilling.",
      outcome: "neutral"
    }
  ];
  // In reality, filter by userId and then use vector similarity on content and eventType
  return mockEpisodes.filter(ep => ep.userId === userId).slice(0,k);
}
// --- Usage Example ---
// // After an interaction where the agent suggested a solution that failed:
// await recordEpisode({
//   timestamp: new Date(),
//   eventType: "agent_action",
//   agentId: "my_agent",
//   userId: "amit_user_123",
//   content: "Suggested checking network tab for API errors, but user mentioned it was a UI-only bug. My advice was off-topic.",
//   sentiment: "negative",
//   outcome: "failure"
// });
// // Before responding to a new query, retrieve past relevant episodes:
// const recurringIssueEpisodes = await retrieveEpisodes("React rendering issues", "amit_user_123");
// // Then feed these summaries into the LLM's context:
// // "Previous interactions about React rendering suggested I sometimes miss UI-only bugs. Focus on UI debugging steps this time."

By storing rich episodic data, your agent can develop a form of experiential learning. It can reflect on past interactions and proactively adjust its behavior, making it more robust and user-centric over time.

Tying It All Together: A Memory Stack for Your Agent

A truly intelligent agent won't rely on just one type of memory. It will use them in concert:

  1. Incoming User Query:
  2. Short-Term Memory Check: Add the query to the current conversation history.
  3. Episodic Memory Retrieval: Query episodic memory (vector store) for past interactions with this user or similar topics that might influence the current response (e.g., user's past frustrations, successful approaches).
  4. Long-Term Memory Retrieval: Query long-term memory (vector store) for factual knowledge relevant to the query.
  5. Context Assembly: Combine the current conversation, relevant episodes, and factual knowledge into a comprehensive prompt for the LLM. You'll need strategies to prioritize and condense this information to stay within the LLM's context window.
  6. LLM Generation: The LLM uses this rich context to generate its response.
  7. Memory Update:
  • Add the LLM's response to short-term memory.
  • Optionally, generate new embeddings for this interaction and add it to long-term memory (e.g., if new facts were discovered or generated).
  • Record a new episode in episodic memory, including details about the query, response, and any observations about the interaction's success or failure.

Beyond the Basics: Advanced Memory Concepts

  • Memory Tiers and Decay: Implement systems where less relevant or older information fades or is summarized to save space, while critical information is retained.
  • Self-Reflection and Learning: Have your agent periodically analyze its episodic memory to identify patterns, improve its decision-making heuristics, or even generate new facts for its long-term memory.
  • Active Recall: Instead of just passive retrieval, have the agent actively "think" about what knowledge might be relevant before a user even asks.
  • External Tools: Memory systems don't just store information; they can also recall which tools to use for specific tasks based on past experiences.

My Personal Take

Building AI agents that feel genuinely intelligent and helpful requires more than just calling an LLM API. It requires a thoughtful approach to how they remember, learn, and adapt. Implementing these different types of memory has been a game-changer in my own projects, transforming agents from one-off responders into capable, learning companions.

I believe the future of AI agents lies in their ability to contextualize information deeply and learn autonomously from every interaction. By mastering short-term, long-term, and episodic memory, you're not just building a smart bot; you're building a sophisticated learning entity.

I'm always eager to connect with fellow engineers and AI enthusiasts! If you're building intelligent agents or just curious about this space, let's connect. Find me on LinkedIn: https://www.linkedin.com/in/amit-shrivastava or X: https://x.com/amit5214. Let's share insights and build the future of AI together!

AI Agents
Memory
Vector Databases
Architecture