Building a second brain with Karpathy's LLM Wiki
Quick Summary
Andrej Karpathy, co-founder of OpenAI and former Director of AI at Tesla, has built an remarkably intelligent second brain system called LLM Wiki. Instead of using AI just to answer questions or write code faster, Karpathy lets AI autonomously build, maintain, and link a personal research wiki. His wiki has now reached over 100 articles and 400,000 words, all written and updated by AI. Unlike traditional RAG which only retrieves temporarily, LLM Wiki operates on a "compilation" principle: AI compiles raw documents into structured knowledge, automatically creates backlinks, detects contradictions, and continuously updates. The system requires only a raw folder for source documents and a wiki folder for Markdown files, running entirely local with Obsidian, with no complex database or vendor lock-in. LLM Wiki marks a shift from using AI to "ask and answer" toward using AI to "build and manage knowledge long-term," and is considered one of the most powerful and practical Second Brain approaches available today.
Andrej Karpathy, co-founder of OpenAI, former Director of AI at Tesla, and the person who coined the term "vibe coding," shared on X how he uses AI, and the answer isn't writing code faster. It's building a self-maintaining, self-linking, self-updating knowledge system for a second brain, which he calls LLM Wiki. His research wiki on a single topic has reached 100 articles and 400,000 words, and notably, every word was written by AI without him typing a single character.
The problem with how we currently use AI to organize knowledge
Does RAG accumulate knowledge over time the way our brains do?
Most current AI tools process documents using a RAG model: you upload a document, ask a question, the system finds relevant passages, and the AI synthesizes an answer. Google's NotebookLM, ChatGPT with file uploads, and most AI workflows use this approach because it's simple and easy to deploy.
But Karpathy points to a core problem that few people notice: RAG does not accumulate knowledge. Every time you ask a question, the system starts from scratch, reading the documents again, finding relevant passages, assembling an answer. Ask the same question the next day and it repeats the entire process as if nothing happened before. A document from March and a document from October don't connect to each other on their own. Nothing accumulates and nothing is learned from the previous session, which is nothing like how our brains actually work.
Karpathy describes the shift in his own thinking with one short sentence that says a lot: most of the tokens he now consumes are no longer going into manipulating code but into manipulating knowledge.
How does LLM Wiki work?
LLM Wiki is not software, it's an Obsidian thinking architecture
Karpathy's idea is not a new piece of software or library. He published it as an "idea file" to create an Obsidian-like architecture. He created a GitHub Gist designed to be copy-pasted directly into an AI agent like Claude Code or OpenAI Codex, then let the agent build the system according to that architecture together with the user. This means you install nothing. Instead, you describe the architecture to the AI and the AI implements it for you.
Three core architectural layers of the Wiki
The system is organized into three distinct layers, each playing an irreplaceable role:
- Raw source folder (raw/): Where you drop any document, whether PDF, article, transcript, note, or tweet, and AI reads it but never modifies this folder. The design principle here is important: collect first, organize later. You don't need to sort or prepare documents before adding them.
- Wiki folder (wiki/): This holds all the Markdown files that AI creates and maintains. It's where knowledge is compiled, linked, and synthesized. Every document in raw/ gets read by AI and integrated into the wiki, updating existing pages, noting contradictions, and creating backlinks to related concepts.
- Configuration file (CLAUDE.md or equivalent): A ruleset that tells the AI how to organize the wiki, format articles, handle contradictions, and maintain consistency across the entire system.
Karpathy describes the relationship between components with one vivid sentence: "Obsidian is the IDE. The LLM is the programmer. The wiki is the codebase." You don't write the wiki yourself. Instead, you ask questions and explore while AI handles the tedious work of maintaining and updating the knowledge base.
The self-maintaining loop is the real differentiator
Three operations running continuously without intervention
What makes LLM Wiki different from ordinary AI note-taking tools is the active loop that runs after the wiki is built. AI doesn't just summarize documents once and stop. It runs three continuous operations:
- Ingest: When you drop a new document into the source folder, AI reads it, extracts key information, and integrates it into the wiki by updating existing pages, creating new ones where needed, and flagging where new information contradicts old information rather than arbitrarily deleting either.
- Query: You ask in natural language, and because the wiki has already been compiled and structured, AI answers with high accuracy and can cite specific pages rather than assembling an answer from scattered passages the way standard RAG does.
- Lint: AI periodically scans the entire wiki to detect broken links, isolated pages with no connections to the rest, contradictions between pages, and knowledge gaps not yet covered. Karpathy calls this "CI/CD for the knowledge base," meaning the system audits its own quality continuously.
Karpathy explains why this system is more sustainable than human-maintained wikis with one simple but precise observation: "People give up on wikis because the maintenance burden grows faster than the value they deliver. LLMs don't get tired, don't forget to update cross-references, and can edit 15 files in a single run."
Why RAG isn't needed at personal scale
Context windows are now large enough to replace vector databases
The most debated argument in Karpathy's proposal is that RAG is unnecessary at personal scale. His logic is this: a comprehensive second brain covering an entire research domain typically compiles to somewhere between 500,000 and 2 million tokens in Markdown. With the long context windows available in current models, that entire wiki can fit into a single query context without needing any complex vector search infrastructure.
Karpathy reports that at around 100 articles and 400,000 words, the system handles complex questions well without any vector database or RAG infrastructure, because AI builds and maintains its own index and summary files and navigates the full text collection efficiently through that self-built structure.
One important caveat: this limit is real. When a wiki grows past a certain threshold, perhaps a few million tokens, the context window does become a genuine bottleneck, and at that point search tools like qmd (a hybrid BM25/vector search tool for Markdown) will need to be integrated to maintain performance.
How to get started in 15 minutes
The first steps to building your first wiki
Karpathy designed this system so that anyone with Claude Code or an equivalent AI agent tool can deploy it immediately without deep technical knowledge. The basic process has four steps:
- Create a new Obsidian vault. This is simply a folder on your computer where all Markdown files will be stored. Obsidian is just the interface you use to read and navigate.
- Create two subfolders:
raw/for source documents andwiki/for AI to write and maintain. These two folders are all you need to set up manually. - Copy Karpathy's GitHub Gist at GitHub and paste it into Claude Code or whichever AI agent you're using. The Gist is written as a set of instructions for the agent, letting the agent build the detailed implementation together with you rather than you doing everything yourself.
- Drop a few initial documents into
raw/and let the agent begin compiling the wiki. From here everything runs on its own.
The entire system runs locally with just two dependencies: Obsidian for viewing and navigation, and an AI agent for writing and maintenance. This means no vendor lock-in, no data sent to the cloud if you use a local model, and no subscription fees beyond the API costs of whichever model you choose.
LLM Wiki compared to MemPalace, Mem0, and Zep
Four different philosophies for the same problem
Around the same time Karpathy's LLM Wiki gained attention, the AI community was also discussing MemPalace, an open-source memory system built by actress Milla Jovovich and engineer Ben Sigman that scored 96.6% on the LongMemEval benchmark. All four systems, LLM Wiki, MemPalace, Mem0, and Zep, address the problem of AI not remembering context between sessions, but they do so through four very different philosophies suited to four different needs.
The easiest way to understand the differences is through a concrete scenario: you have six months of AI conversations about a research project, covering every decision, every argument, every discarded option. You open a new session and ask: "Why did we choose direction A over B back then?" Each system answers in a completely different way.
- Mem0 works like a secretary who takes meeting notes. It uses AI to read conversations, extract important facts such as preferences and decisions made, and stores them in a vector database. When you ask again, it finds the fact closest to your question and returns it. Fast, easy to integrate, and well suited to commercial chatbots, but the reasoning behind a decision and the chain of logic that led there is usually gone because the AI already decided that part wasn't important.
- Zep goes one step further with a time-aware knowledge graph. It doesn't just remember "you preferred X" but "in January you thought X, in March you switched to Y because of Z." Its strength is understanding change over time and it suits applications that need to track user progress, but Zep still uses AI to decide what enters the graph, so there's still a risk of losing important context, especially complex reasoning the AI judged as unnecessary.
- MemPalace takes the opposite philosophy entirely: store everything, then make it findable. Instead of letting AI decide what's worth remembering, MemPalace stores the full verbatim text of every conversation into ChromaDB and organizes it in a hierarchical structure inspired by the ancient Greek memory palace technique: Wing, Hall, Room, Closet, Drawer. Nothing is filtered out but everything has a clear address for retrieval, and the system runs entirely locally without sending data anywhere.
- Karpathy's LLM Wiki solves a fundamentally different problem from the other three. Instead of remembering conversations, it compiles documents into structured knowledge. You don't feed it chat history but rather articles, transcripts, and research notes, and AI builds a linked, summarized, queryable Markdown wiki. Each new document isn't just stored but integrated into existing knowledge, creating new connections between concepts and enriching what is already known.
Comparison table to choose the right tool for the right need
| Criteria | LLM Wiki | MemPalace | Mem0 | Zep |
|---|---|---|---|---|
| Data source | Research documents, articles, transcripts | AI conversation history | Conversation history | Conversation history |
| Storage method | Structured Markdown, AI compiled | Full verbatim text, spatial hierarchy | Facts extracted by AI | Time-aware knowledge graph |
| Does AI filter information? | Yes, AI decides how to organize | No, everything is stored | Yes, AI selects important facts | Yes, AI selects entities and relations |
| Runs locally? | Yes, only Obsidian and a model needed | Yes, ChromaDB and SQLite on device | No, cloud service | No, cloud service |
| Best suited for | Research, learning, document synthesis | Long-term AI context memory | Chatbots, commercial applications | Apps tracking user progress over time |
| Weaknesses | Doesn't remember conversations, requires initial setup | Storage-heavy, no visual UI yet | Loses complex reasoning | Cloud dependency, still risks losing context |
The most important thing to remember when choosing: LLM Wiki and MemPalace solve two different problems and can be used together rather than choosing one over the other. MemPalace remembers the history of your conversations with AI, meaning it knows what you said, what you decided, and how your thinking changed. LLM Wiki organizes knowledge from the outside world, the articles you read, the videos you watched, the documents you collected. Combining both lets AI understand both who you are and what field you're researching, and together they form a more complete second brain.
The most thought-provoking insight from LLM Wiki
Most of us use AI as a tool for generating temporary answers. Each session starts from scratch and nothing accumulates. Karpathy's LLM Wiki suggests a different direction: using AI as a knowledge compiler, where each new document isn't just stored but integrated into an existing structure, creating new connections and enriching what is already known.
If you're researching a specific domain, whether AI, technology, finance, or anything else, this is worth trying today. Create a folder, drop in five articles you've read recently, and let Claude Code begin building the first wiki. After one week of adding documents consistently, you'll see the difference between an archive and an actual knowledge base.
