sop

Scalable Objects Persistence

Project maintained by SharedCode Hosted on GitHub Pages — Theme by mattgraham

AI Copilot User Guide

The SOP AI Copilot is a powerful, conversational interface for interacting with your SOP databases. It allows you to query data, perform CRUD operations, and automate complex workflows using natural language.

Core Philosophy: Stateless vs. Stateful

To ensure system stability and prevent “dangling transactions,” the Copilot operates in two distinct modes:

Stateless (Interactive & Recording): Every prompt is an independent unit of work.
- If you ask “Select all users”, the Copilot opens a transaction, reads the data, and immediately closes the transaction.
- This applies even when Recording. Each step you record is executed and committed immediately.
Stateful (Playback): When Playing a Script, the Copilot can maintain a transaction across multiple steps.
- This allows scripts to perform complex, multi-step atomic operations (e.g., “Transfer funds: Debit A, Credit B”).
- If any step fails, the entire script transaction can be rolled back.

1. Natural Language Data Access

You can ask the Copilot to retrieve data using plain English. It translates your intent into optimized B-Tree lookups.

Listing Resources

“Show me all databases” -> Lists available databases.
“What stores are in the ‘users’ database?” -> Lists tables/stores in that DB.

Selecting Data

The select tool is powerful and supports filtering and field selection.

Basic: “Get the first 10 records from the ‘users’ store.”
Filtering: “Find users in the ‘users’ store where the ‘role’ is ‘admin’.” Supports MongoDB-style operators for comparisons: $gt, $gte, $lt, $lte, $ne, $eq. Example: “Select employees where age > 30” (Assistant converts this to {"age": {"$gt": 30}}).
Field Selection: “Show me just the ‘username’ and ‘email’ for all users.”
Scanning: “Scan the ‘logs’ store for entries with ‘error’ in the message.” (Note: Scanning large stores can be slow; prefer key lookups).
Ordering: SOP stores are B-Trees and are naturally ordered by their Keys. Therefore, explicit ORDER BY clauses are not supported (and not needed). You always operate in the native B-Tree sort order.
UI Display Note: When selecting specific fields, the backend returns them in the requested order (e.g., select salary, name returns salary then name). However, the UI Grid always displays Key fields (columns from the Key object) before Value fields (columns from the Value object) for consistency. If you request a Value field followed by a Key field, they will appear as Key then Value in the grid. The raw JSON response (accessible via API) preserves your requested order within the Key and Value objects respectively.
Views (Scripts): You can use a Script as a data source! If you have a script named ‘active_users_view’ that returns a list of users, you can query it like a table: “Select name, email from ‘active_users_view’”. This allows you to create complex “Views” using scripts (even with Joins) and query them simply. Streaming Support: Unlike traditional views that might materialize results, SOP streams script output directly. Field selection is applied “late-bound” as items flow through, ensuring high efficiency even for complex pipelines.

Finding Specific Records

Exact Match: “Find the user with key ‘user_123’.”
Nearest Match: “Find the user closest to ‘user_125’.” (Useful for finding range boundaries).

Querying Rules & Expectations

To ensure accurate results, it’s important to understand how the AI Copilot and API interpret your queries, particularly regarding case sensitivity.

Schema Names (Case-Insensitive)

Field names and JSON keys are Case-Insensitive.

Querying for field: "Name" will successfully match a field named name, NAME, or Name.
This flexibility allows you to ask natural questions without needing to know the exact capitalization of the schema.
Note: While the query engine is flexible, the returned data will preserve the original casing stored in the database.

Data Values (Case-Sensitive)

Actual data values used in comparisons are Case-Sensitive.

Search: Searching for role: "Admin" will NOT match a record where the role is "admin".
Joins: When joining two stores, the values in the joining fields must match exactly (e.g., ID "u123" will not join with "U123").
Best Practice: Ensure your query values match the case of the data stored in the database.

Entity Names (Case-Insensitive)

Names of Databases, Stores, and Scripts are Case-Insensitive.

You can refer to your store as ‘Users’, ‘users’, or ‘USERS’, and the Copilot will find the correct one.
Note: In the rare case where two stores have the same name but different casing (e.g. users and USERS), the system prefers the exact match.

Efficient Query Scenarios

SOP is a high-performance database that uses B-Trees. To get the maximum speed (especially on large datasets), structure your questions to leverage the Index (Key) structure.

How to write fast queries: The “Index” is defined by the Key fields of your store.

Fast: Filtering by the first field(s) of the Key (Prefix Match).
Fast: Joining on the Key fields.
Slow: Filtering by a field that is not at the start of the Key (requires a full scan).

Scenario	Index (Key) Structure	Query Example	Status
Exact Match	`[Region, Dept]`	“Find employees in ‘US’ ‘Sales’”	⚡ Fast
Prefix Match	`[Region, Dept]`	“Find employees in ‘US’”	⚡ Fast
Natural Sort	`[Region, Dept]`	“Find ‘US’ employees, ordered by Dept”	⚡ Fast
Skip Prefix	`[Region, Dept]`	“Find employees in ‘Sales’” (Skipped Region)	🐢 Slow (Full Scan)
Sort Conflict	`[Region, Dept]`	“Find ‘US’ employees, ordered by Dept DESC” (If index is ASC)	🐢 Slower (Buffered)

2. CRUD Operations

You can modify data directly.

Adding Data

“Add a new user to ‘users’ with key ‘u_99’ and name ‘Alice’.”
“Insert this JSON into ‘config’: {…}”

Updating Data

“Update user ‘u_99’ and set ‘status’ to ‘active’.”
“Change the email for ‘u_99’ to ‘alice@example.com’.”

Deleting Data

“Delete the record ‘u_99’ from ‘users’.”

Note: In Stateless Mode, these operations commit immediately. If you need to do multiple updates atomically, consider using a Script.

4. Memory & Learning: The Self-Correcting Copilot

The AI is designed to evolve. It possesses two distinct types of memory, allowing it to adapt to your specific environment and business logic over time.

Short-Term Memory ( The “Session” Context)

This is the Working Memory. It holds the immediate context of your current conversation, active transaction, and recent query results.

Duration: Ephemeral. Cleared when you refresh or start a new session.
Capabilities:
- “Peek-Ahead” Schema Awareness: Before answering, the AI “peeks” at your real data to learn field names and types (e.g., seeing that status is an int, not a string). This prevents halogenations about your schema.
- Conversational Continuity: “Refine that last query”, “Use the ID from the previous result”.
Privacy: Session data is isolated. It doesn’t leak into the global brain unless explicitly saved.

Long-Term Memory (The “System Knowledge”)

This is the Enterprise Brain. It persists facts, business rules, and corrections across all sessions and users. It effectively turns the AI into a “Self-Correcting” system.

Storage: Rules are stored in the llm_instructions B-Tree in the SystemDB. They are transactional and durable.
The “Intelligent Librarian”: The AI doesn’t load everything at once. It maintains a “Table of Contents” (namespaces like finance, hr, schema) and dynamically fetches the relevant rules only when needed.
How to Teach (Self-Correction):
1. Direct Instruction: “Remember that ‘EBITDA’ means ‘Earnings Before Interest…’”
2. Correction: If the AI fails (e.g., queries status="Active"), correct it: “Status is an integer! 1=Active.” The AI will then update its own instructions to ensure it never makes that mistake again.
3. Explicit Rules: “Save this instruction in the ‘finance’ category: Fiscal year starts in April.”

Why this matters: You don’t need to write perfect prompts every time. If the AI gets it wrong, you fix it once, and it stays fixed—for you and your team.

5. Script Management (Automation)

Scripts allow you to record a sequence of actions and replay them later. This is “Natural Language Programming.”

Recording a Script

Start: Type /record my_new_script.
- Default (Compiled): Records the exact actions (tools) you perform. When played back, it executes these actions directly (fast, deterministic).
- Interactive Mode: Type /record my_new_script --ask. Records your prompts. When played back, it asks the AI again (slower, but adapts to new data/context).
Teach: Perform your actions step-by-step.
- “Select users where role is admin.”
- “For each user, check their last login.”
- “If last login is > 30 days, set status to inactive.”
Stop: Type /stop.

The script is saved incrementally after every step. If you crash or disconnect, your progress is safe!

Playing a Script

Run: Type /play my_new_script.
With Arguments: If your script takes parameters (e.g., user_id), the Assistant will prompt you for them or you can provide them in the chat.

Managing Scripts

List: “List all scripts.”
Inspect: “Show me the steps for ‘my_new_script’.”
Refactor: If you just had a great conversation and want to save it as a script after the fact, say: “Refactor that last interaction into a script named ‘audit_workflow’.”

4. Advanced Features

Explicit Transaction Management

For complex, multi-step manual operations without creating a script, you can manually control the transaction:

“Begin a transaction.”
(Perform multiple updates…)
“Commit transaction.”

Swarm Computing (Async)

When defining scripts manually (or asking the Assistant to edit them), you can mark steps as is_async: true. This allows the Assistant to execute multiple heavy tasks in parallel (e.g., “Summarize these 50 documents”).

5. Session Tools

/last-tool: Displays the exact JSON instructions (tool name and arguments) of the last executed action. Useful for debugging or verifying what the Assistant actually did.

Example Session

```text User: /record onboard_user Copilot: Recording started for script ‘onboard_user’.

User: Add a user to ‘users’ with key ‘new_guy’ and status ‘pending’. Copilot: Added user ‘new_guy’. (Step recorded)

User: Add a log entry to ‘audit_logs’ saying “User new_guy created”. Copilot: Added log entry. (Step recorded)

User: /stop Copilot: Recording stopped. Script ‘onboard_user’ saved with 2 steps.

User: /play onboard_user Copilot: Executing ‘onboard_user’…

Added user…
Added log entry…

6. Mobile & Small Device Support (“Pocket Admin”)

The SOP AI Copilot is designed with a Mobile-First responsiveness philosophy, turning your smartphone or tablet into a full-featured “Pocket Admin” console.

Optimized UI

The chat interface automatically adapts to smaller screens:

Data Grids: Collapse gracefully or allow horizontal scrolling without breaking the layout.
Action Chips: Suggested actions and database selectors are touch-friendly.
Visual Feedback: Loading states and success confirmations are designed for quick visual scanning on small screens.

Full-Featured Management

You are not limited to “read-only” views on mobile. You can perform all system operations via chat:

Switch Databases: Tap the database selector or simply say “Switch to the ‘finance’ database” to instantly jump contexts.
Run Complex Queries: “Find all users joined today” works just as well on a phone as on a desktop.
Execute Scripts: Trigger complex backend jobs (e.g., “Run end-of-day reconciliation”) with a single message while commuting.
Data Rescue: Need to fix a record urgently? “Update user ‘123’, set status to ‘active’” allows for emergency interventions from anywhere.

The Copilot essentially acts as a highly capable CLI (Command Line Interface) wrapped in a chat bubble, ensuring you have full control over your infrastructure without needing a laptop.