sop

Scalable Objects Persistence

---

Project maintained by SharedCode Hosted on GitHub Pages — Theme by mattgraham

Beyond RAG: The Self-Correcting Enterprise AI

How SOP turns user frustration into permanent system intelligence

In the rush to adopt AI, most enterprises are hitting a wall: The “Goldfish Memory” Problem.

You spend hours crafting the perfect prompt to teach your AI assistant how to query your specific “Users” schema. It finally works! You close the window. The next day, your colleague asks the same question, and the AI makes the same mistake.

You haven’t built an asset; you’ve just had a conversation.

At SOP, we believe that every interaction with an AI should upgrade the system permanently. We are proud to introduce our latest architecture: The Self-Correction Loop, a feature that turns your database administration tool into a learning organism.

The Architecture of a “Living” System

Most AI Agents effectively have “Read-Only” access to their own operating instructions. They follow the prompt hardcoded by the developer. If that prompt is wrong or incomplete, the Agent fails—forever—until a developer redeploys the binary.

We flipped this model. In SOP, the Agent’s instructions are not code; they are data.

1. The Registry as “Liquid Truth”

Instead of hardcoding tool definitions (like execute_script) into the Go binary, the SOP Agent now fetches its operating manual from a dedicated B-Tree store called llm_instructions located in the SystemDB.

When the Agent initializes, it performs a millisecond-latency lookup:

“I need to run a script. What are the current best practices for joining tables in this specific environment?”

This allows the instructions to change dynamically without restarting the server.

2. The update_instruction Tool: Giving the AI a Pen

We gave our Agent a new tool: update_instruction. This allows the LLM to rewrite its own prompt.

The Workflow:

1. The Mistake: A user asks, “Get me all active orders.” The AI queries orders where status = "active".
2. The Correction: The system errors (or the user corrects it): “Status is an integer! 1 is active, 0 is inactive.”
3. The Learning: The AI successfully retrieves the data using status = 1.
4. The Commit: Crucially, the AI then calls update_instruction.
   • Input: “When querying the ‘orders’ table, never use strings for status. Always use 1 for Active, 0 for Inactive.”
   • Action: This rule is ACID-transaction committed to the llm_instructions B-Tree.

3. The “Enterprise Brain” (Centralized Knowledge)

By setting a simple environment variable (SYSTEM_DB_PATH), multiple instances of SOP—running on different developer machines or production servers—can share this single SystemDB.

• The Junior Developer Effect: A junior dev spends morning struggling with a complex join query. They eventually correct the AI.
• The Senior Developer Benefit: In the afternoon, a senior dev asks the same question. The AI gets it right on the first try because it’s sharing the same “Brain.”

Why This is a Moat

This feature is difficult to replicate with standard “Chat with PDF” RAG solutions because it requires deep integration between the Reasoning Engine (LLM) and the Storage Engine (SOP).

1. Transactional Integrity: We treat “Knowledge” like financial data. Updates to instructions use the same ACID transactions as bank transfers. We don’t lose knowledge if the server crashes.
2. Native Speed: Because the instructions are stored in SOP B-Trees (Key-Value), the lookup is instant. There is no vector search latency for these core operational rules.
3. Context isolation: The AI knows specifically which tool the instruction applies to, preventing “context pollution” where a rule for Table A mistakenly gets applied to Table B.

4. The Precision Guarantee: JIT Compilation & Zero-LLM Execution

This self-correcting brain is the perfect partner to our JIT Compiled Scripting Engine.

While the LLM is used to understand the intent and navigate the schema (using its refined instructions), the final output is not a vague chat response. It is a precise, deterministic SOP Script.

• Zero-LLM Execution: Once the script is generated and verified, the LLM leaves the room. The script is compiled and executed by our high-performance Go engine. This removes “hallucination” from the execution loop.
• Reusable Automation: A successful “exchange” isn’t just a memory; it can be saved immediately as a named script (e.g., calculate_churn_v2).
• Precise Engineering: Users can trust the platform because the “fuzzy” logic of AI is strictly separated from the “concrete” logic of execution. The AI is just the architect; the SOP Engine is the builder.

This combination—Adaptive Knowledge for the Architect and Deterministic Execution for the Builder—creates a platform that feels like magic but runs like engineering.

Conclusion

We are moving beyond “Chatbots” to “Adaptive Systems.”

With SOP’s new Self-Correcting Intelligence, your documentation is no longer a static wiki that goes out of date. It is a living database, curated by the AI itself, growing smarter with every query, every error, and every correction.

Your database shouldn’t just store your data. It should store the knowledge of how to use it.

Evolution: The “Intelligent Librarian” Update (January 2026)

We realized that “Goldfish Memory” wasn’t the only problem. The other problem was “The Encyclopedia Problem”. If you give an AI all the knowledge at once, it gets confused (and the context window explodes). If you give it nothing, it makes things up (hallucinations).

We’ve solved this with three new mechanisms introduced in the DataAdminAgent architecture.

1. The “Peek-Ahead” Schema Injection

Previously, the AI had to guess field names (e.g., hallucinating total instead of total_amount). Now, before the conversation starts, the Agent performs a millisecond “Peek” operation (storeAccessor.First()). It grabs a real sample record, infers the schema (e.g., id: string, active: boolean), and injects this “Ground Truth” directly into the system prompt.

• Result: Zero hallucinations on field names. The AI knows your data structure before you ask.

2. The “Table of Contents” (Meta-Cognition)

We cannot load every rule into the prompt. Instead, we now inject a dynamic list of available knowledge categories (namespaces).

• Old Way: “I don’t know about ‘Q3 targets’.”
• New Way: The AI sees [finance, sales_targets, hr_policies] in its context. When asked about “Q3 targets”, it sees the sales_targets category and proactively decides: “I should read the ‘sales_targets’ chapter before answering.”

3. The “Decision Protocol”

The AI has been taught a strict flowchart:

1. Context Check: “Do I have the answer right here?” -> React Fast.
2. Ambiguity Check: “Is this term undefined?” -> Consult the Library.
   • It executes manage_knowledge(action='list', namespace='finance') completely autonomously to fetch the rules it needs.

This turns the AI from a passive responder into a proactive researcher, capable of navigating gigabytes of institutional knowledge without overloading the context window.

4. Relational Intelligence (The Graph Awareness)

Beyond just field names (“schema”), the Agent now perceives the relationships between stores.

• The Context: During the “Peek” phase, the Agent also reads the Store Registry’s relational metadata.
• The Injection: It sees: - orders {id: string...} (Relations: [user_id] -> users([id]))
• The Impact: When you ask “Show me orders for Alice”, the Agent instantly knows:
  1. I need to find Alice in users.
  2. I must use user_id to join with orders.
  3. It generates the correct JSON join instruction on the first try, avoiding “guessing” how tables are linked.