sop

Scalable Objects Persistence

---

Project maintained by SharedCode Hosted on GitHub Pages — Theme by mattgraham

Building a “Local Expert” AI: Embedding an Intelligent Copilot into SOP Data Manager

I’m excited to share a major update to the SOP Data Manager, the GUI tool for our Scalable Objects Persistence (SOP) engine. We’ve moved beyond simple CRUD operations and integrated a fully context-aware AI Copilot directly into the workflow.

This isn’t just a chatbot overlay; it’s a ReAct (Reasoning + Acting) Agent deeply integrated with the database backend, designed to act as a “Local Expert” for your data.

Important: The AI Copilot requires an LLM API Key (e.g., Gemini, OpenAI) to function. You must provide your own key in the “Environment Configuration” settings. If no key is supplied, the AI Copilot features will be disabled.

🔌 New: “No-LLM” Production Mode

SOP recognizes that Production Environments often have strict security policies that forbid external API calls (Air-gapped) or require zero external dependencies.

We have introduced a Direct Command Interface that works entirely offline:

• Slash Commands: You can bypass the LLM entirely by using slash commands (e.g., /select database=users).
• Zero Dependencies: This mode requires NO internet connection and NO API Key.
• Full Power: You get access to the exact same backend tools (select, run_script, manage_knowledge) that the Agent uses, but driven manually by you.

This confirms our philosophy: The AI is a helper, but the robust machinery underneath is yours to command.

The Problem: Context Switching

Managing complex NoSQL data often involves jumping between a GUI to view items and a terminal to run queries or scripts. You might see a record, wonder “how many other records have this specific field value?”, and have to switch context to write a script to find out.

The Solution: A Floating, Context-Aware Copilot

We’ve refactored the UI to introduce a persistent, floating AI widget.

• Always Available: It floats above your data grid, draggable and resizable, so you never lose sight of the data you’re analyzing.
• Tool-Equipped: The AI isn’t hallucinating answers. It has access to real backend tools:
  • list_stores(): To understand your database topology.
  • get_schema(): To analyze the structure and index specifications of your B-Trees.
  • search() & select(): To query data directly.
• Visual Feedback: When the AI performs a search, it doesn’t just tell you the results—it can trigger the main data grid to refresh and show you the actual records.

Under the Hood: A Secure RAG Pipeline

We purposely designed a Retrieval-Augmented Generation (RAG) pipeline to address the critical security needs of database management.

1. Local Power, Global Intelligence: The backend is built in Go, keeping the execution logic and data access strictly local.
2. Data Privacy & Filtration: Your production data never leaves the environment. Only the prompt for interpretation and reasoning is sent to the LLM. The actual data retrieval and manipulation happen locally via the SOP library.
3. Policy Enforcement: By decoupling the reasoning (LLM) from the execution (Local Tools), we can enforce strict policies on what the agent can and cannot do, removing the risks associated with giving an AI direct access to corporate data.
4. ReAct Loop: The LLM reasons about which tool to use, but the Go backend intercepts these calls, validates them, and executes them safely using ACID transactions.

Why This Matters

This architecture allows us to create a fully controllable, customizable AI necessary for enterprise databases.

• Agentic Interfaces: We build a robust set of tools and give an AI agent the agency to use them, but within a secure sandbox.
• For Developers: You get a natural language interface to your raw data. “Show me the top 5 users created yesterday” translates automatically to a B-Tree range query.
• For Operations: Troubleshooting becomes conversational. “Why is this store empty?” prompts the AI to check schemas and transaction logs.

Natural Language Programming: The “Lego Blocks” Evolution

We’ve taken this a step further with our Script System, evolving it into a robust scripting engine built on “Atomic Lego Blocks.”

Instead of asking an LLM to generate raw, potentially unsafe code (which is hard to debug and secure), we use a Compiler Approach:

1. Intent Extraction: The LLM analyzes your request (e.g., “Find users older than 25 who haven’t logged in for a month”).
2. Block Assembly: It assembles a script using our pre-built, high-performance atomic functions (the “Lego Blocks”).
   • compare(val1, val2): A universal comparator handling strings, numbers, and dates seamlessly.
   • matchesMap(item, criteria): A MongoDB-style query evaluator supporting operators like $gt, $lt, $in, and $eq.
   • toFloat(val): Robust type conversion for numerical analysis.
   • Scan(store, options): High-performance B-Tree traversal (Range, Prefix, Forward/Backward).
   • JoinRightCursor(left, right, key): Optimized Right Outer Join iterator.
3. Safe Execution: These blocks are compiled into a SOP Script that runs on the bare-metal Go engine.

Why “Lego Blocks”?

• Agility & Control: We can tweak the underlying implementation of a block (e.g., optimizing compare for speed) without changing the AI’s behavior.
• Safety: The AI cannot “hallucinate” dangerous code. It can only arrange the safe blocks we provide.
• Performance: The resulting scripts run at native Go speeds, not interpreted Python/JS speeds.

• Streaming & Efficiency: Results are streamed directly to the UI or REST client. This allows processing SQL Joins on huge B-Trees with minimal memory, as we never load the full dataset into RAM.

• SystemDB: These scripts are stored in a dedicated B-Tree, effectively turning your database into a programmable application server.

Next Steps

We are expanding the toolset to include Vector Search capabilities, allowing you to perform semantic queries (“Find documents related to ‘scalability’”) right from the same floating window.

Check out the code and try running it locally with Ollama!

#Golang #AI #LLM #Database #SOP #OpenSource #AgenticUI #ReAct