Scalable Objects Persistence
In the world of distributed systems, we often build architectures that resemble a rigid military hierarchy. We have a “Master” node (the General) and “Worker” nodes (the Soldiers). The General issues commands, tracks every movement, and ensures no two soldiers bump into each other.
This works—until the General gets overwhelmed. As the army grows, the General becomes the bottleneck. The soldiers spend more time waiting for orders than doing work.
Scalable Objects Persistence (SOP) takes a different approach. It is designed not like a military hierarchy, but like a swarm of ants.
Imagine an army of ants foraging for food.
SOP implements this “Swarm Computing” model to achieve linear scalability and remove the bottlenecks typical of Master/Slave database architectures.
In an SOP cluster, every application instance or microservice running the SOP library acts as its own Master. There is no central “SOP Server” that accepts data and writes it to disk.
Ants don’t carry the food to the Queen to ask where to put it. They use pheromones to signal “this path is busy” or “food is here.”
SOP uses Redis exactly like pheromones.
When an SOP transaction wants to modify an item, it leaves a “scent” (a lightweight key) in Redis. Other transactions check for these scents. If the path is clear, they proceed immediately. This interaction is lightning fast (microseconds) and involves no heavy data transfer.
What happens if two ants try to place food in the same storage chamber?
In traditional databases, the system locks the entire chamber (Table or Page Lock), forcing one ant to wait until the other is completely finished and has left the building.
SOP employs In-Flight Data Merging.
They only stop if they are fighting over the exact same crumb (the same Key), in which case the OOA ensures data integrity by asking one to retry.
Because there is no central supervisor to overwhelm, you can keep adding “ants” (nodes) to your cluster indefinitely.
The workers coordinate “in good faith” using lightweight signals, merging their work in-flight, and checking in data massively and in parallel.
This is Swarm Computing. This is SOP.
See the “Concurrent Transactions” examples in our language bindings for practical implementations of Swarm Computing:
To enable seamless concurrent merging on a newly created B-Tree, you must pre-seed the B-Tree with at least one item in a separate, initial transaction. This establishes the root node and structure, preventing race conditions that can occur when multiple transactions attempt to initialize an empty tree simultaneously.
Note: This requirement is simply to have at least one item in the tree. It can be a real application item or a dummy seed item.
While SOP provides the “Storage Swarm” (distributed, lock-free data persistence), our AI Framework introduces the “Compute Swarm”.
We have built a powerful framework where programming logic itself is distributed, autonomous, and network-native.
In a traditional system, code is static and deployed to specific servers. In our Swarm, Scripts act as the genetic code of the operation.
Single vs PerStep), allowing complex, multi-step distributed operations to be treated as a single unit of work.Agents in our framework are not just chatbots; they are autonomous compute units.
We are moving towards a model where users can simply “release” a task into the swarm.
This transforms the system from a passive data store into an active, living Swarm Computer.