sop

Scalable Objects Persistence

---

Project maintained by SharedCode Hosted on GitHub Pages — Theme by mattgraham

Evolution of an AI Agent: From Chatty Assistant to Scalable Data Engine

The Vision: AI as a Functional Runtime

When we started building the SOP AI Agent, the goal was never just to create another chatbot that summarizes text. The vision was far more ambitious: to create a conversational interface for high-scale data management.

We wanted an AI that could:

1. Understand natural language intent (“Show me high-salary employees in Engineering”).
2. Translate that into precise database operations.
3. Execute complex workflows (Scripts) that behave like reliable server-side functions.

The ultimate goal was to have the AI act not just as a wrapper around a database, but as a programmable engine where a “Script” is effectively a stored procedure that can be invoked via a REST API.

The Paradigm: Natural Language Programming

We call this interface Natural Language Programming. The goal is to democratize software development by allowing typical users to author programs using plain English.

SOP functions as a compiler for this new language:

1. Authoring: The user describes a workflow: “Check the inventory levels, and if any item is below 10 units, create a reorder request.”
2. Compilation: The Agent translates these high-level intents into machine-executable scriptlets (our AST).
3. Runtime: These scriptlets are stored as Scripts—effectively turning English instructions into repeatable, scalable software artifacts.

This shifts the role of the AI from a passive “assistant” to an active development platform, where the “code” is natural language and the “binary” is the JSON-based Script definition.

The Hardship: Growing Pains

1. The “Chatty” Trap

In our initial iteration, we fell into a common trap in AI development: anthropomorphism over structure. We designed the backend to “talk” like a human, streaming back logs of its thoughts mixed with data.

This led to Parsing Nightmares (frontend using Regex to hunt for JSON) and Scalability Bottlenecks (buffering massive responses in memory to validate them). We realized that to scale, we had to stop treating the AI as a chat partner and start treating it as a compute engine.

2. The State Management Nightmare

Perhaps the most difficult challenge was Recording vs. Runtime.

• Recording: When a user says “Start recording”, every subsequent action needs to be captured. But users make mistakes. They run queries that fail. They ask clarifying questions. How do you distinguish between “noise” and “intent”?
• Runtime: When replaying that script, the environment is different. The transaction context is different. The variables are different.

We initially tried to share state between the “User Session” and the “Script Runner”. This was a disaster. Scripts would accidentally commit user transactions, or user queries would bleed into script execution scopes. We needed a way to guarantee stability for the end-user recording session while maintaining a pristine environment for the runtime.

The Architecture: Built for Scale

We completely refactored the engine around three core pillars: AST Composability, Session Isolation, and Structured Streaming.

1. The AST & Composability

We moved away from “script recording” (saving text commands) to an Abstract Syntax Tree (AST) approach. We defined a rigid schema for a ScriptStep:

type ScriptStep struct {
    Name        string         // Unique identifier for the step
    Description string         // Human-readable explanation of what this step does
    Type        string         // "command", "ask", "if", "loop", "script", "tool"
    Command     string         // The actual instruction
    Args        map[string]any // Parameters
    Steps       []ScriptStep    // Nested steps (for loops/conditionals)
}

This design unlocked Composability and Self-Documentation.

• Composability: Because a ScriptStep can be of type script, one script can call another.
• Self-Documentation: The Description field allows the script to be read and understood by humans and LLMs alike, facilitating “Code Review” for AI agents.
• We can build small, atomic scripts (find_user, calculate_tax).
• We can compose them into complex workflows (process_payroll calls find_user then calculate_tax).
• The runner (runStepScript) simply pushes a new stack frame and executes the child script, just like a function call in a programming language.

2. Session Isolation

To solve the state management nightmare, we strictly separated the Recording Context from the Runtime Context.

• RunnerSession: Holds the state of the active interaction (recording flags, current transaction).
• Execution Context: When a script runs, it spins up a fresh, isolated context (scriptCtx). It gets its own variable scope and its own transaction boundaries.
• Payload Injection: We pass dependencies (like the target Database) via the context payload, ensuring that a script running against UserDB cannot accidentally touch SystemDB.

3. Structured Streaming (The Heart & Soul)

Finally, to solve the “Chatty” trap and enable scaling, we implemented the JSON Streaming pattern, the heart & soul of SOP’s large data chunking extended to the AI space.

Instead of writing raw strings, the engine emits StepExecutionResult objects. We implemented a JSONStreamer that wraps the HTTP response writer.

type StepExecutionResult struct {
    Type    string `json:"type"`    // "command", "ask", "error"
    Result  string `json:"result"`  // The raw data payload
}

As soon as a step finishes, it is serialized and flushed.

• Low Latency: The client sees progress immediately.
• Low Memory: We process 100k records, stream the result, and forget it. No massive buffers.
• Frontend Decoupling: The backend sends a “Script Trace” (JSON array). The frontend decides how to render it—as a Chat bubble or a CSV table.

The Result: A RESTful Experience

The transformation is profound. Running a complex AI script now feels exactly like calling a standard REST API endpoint.

• Input: /play script=audit_salary
• Output: A clean stream of JSON objects.
• Behavior: Deterministic, machine-readable, and pipeable.

We successfully bridged the gap between the flexibility of Generative AI and the rigidity required for data engineering. The SOP Agent is no longer just “chatting” about data; it is streaming it.