v0.1 · substrate live · alpha
swarm-lib
Filesystem-as-orchestrator for agentic workflows.
Three primitives — the Yield Rule, atomic-rename queueing, and status.json checkpointing — that turn LLM-driven agents into interchangeable workers backed by durable state. No broker. No daemon. No external database. Just POSIX and JSON.
What works today
v0.1 substrate is complete and validated end-to-end. The Python lib + bash worker loop are both live; full pipeline (enqueue → claim → handler → done + status update) passes integration smoke tests.
swarm_lib.claims
enqueue · try_claim · complete
POSIX atomic-rename queue primitives
swarm_lib.status
initialize · read · write · append_completed
status.json checkpoint state
swarm-cli
All primitives over subprocess for non-Python consumers
jq-friendly single-line JSON
worker_loop.sh
Generic consumer loop, handler-agnostic
Bash, drives the whole pipeline
Why this exists
Agentic workflows today have three structural problems. swarm-lib gives you three primitives to solve them.
Context starvation
One long conversation accumulates context, hits compaction, mid-flight work dies. The whole chain depends on a single conversation surviving.
Sync tool-call blocking
A planner agent holds an expensive context window open while a subprocess churns for 30 minutes. Tokens burn while the model sits idle waiting for I/O.
Chat-history-as-state
When the conversation dies, the work dies. There's no durable source of truth for "where am I in the work" outside the volatile chat window.
The three primitives
1 / atomic queueing
Atomic-rename task queueing
Producers stage tasks under pending/.
Consumers race for them via os.replace(pending/<id>.json, claimed/<worker_id>/<id>.json).
POSIX guarantees the rename is atomic on the same filesystem — two consumers racing for the same task, exactly one wins, no locks needed. Maildir physics, applied to agent work.
2 / durable handoff
status.json checkpointing
Every workflow keeps its state in a single JSON file. Any agent — a fresh Claude Code session, a local ollama-backed model, even a shell script — picks up where the last one stopped by reading this file. Compactions, crashes, rate limits, multi-day pauses all become indistinguishable from a clean restart. Chat history is volatile; the file is the contract.
3 / generic worker
Generic worker_loop.sh
Polls the queue, claims atomically, invokes a handler with task JSON on stdin, moves the result to done/ or failed/. Workers are interchangeable: any process that reads a JSON task is a participant. Claude Code, ollama agents, n8n flows, shell scripts — same substrate.
Quickstart
Install
git clone https://github.com/dpdanpittman/swarm-lib
cd swarm-lib
pip install -e .
# 'swarm-cli' is now on your PATHFrom Python
from swarm_lib import claims, status
run_dir = "~/swarm-runs/demo-1"
# Producer: write a task
status.initialize(run_dir, run_id="demo-1")
claims.enqueue(
run_dir,
task_id="t.1",
task_type="plan",
payload={"hello": "world"},
)
# Consumer: claim and finish it
task = claims.try_claim(run_dir, worker_id="w.demo")
if task:
# ...do work, write artifacts...
claims.complete(task, success=True)
From bash (via worker_loop.sh)
# Write a handler — receives task JSON on stdin, writes to $SWARM_ARTIFACT_PATH
cat > handler.sh <<'EOF'
#!/usr/bin/env bash
echo "handled $SWARM_TASK_ID by $SWARM_WORKER_ID" > "$SWARM_ARTIFACT_PATH"
EOF
chmod +x handler.sh
# Run a worker — claims, invokes handler, marks done
swarm_lib/worker_loop.sh \
--run-dir ~/swarm-runs/demo-1 \
--worker-id w.demo \
--handler ./handler.sh \
--max-iterations 1For whom
- → Builders of multi-step agent workflows that don't fit in one context window
- → Anyone running Claude Code, Codex, or similar long-running LLM agents at scale
- → Multi-agent systems that need durable handoff between specialized workers
- → Operations that need to survive compactions, rate limits, and process restarts
- → Anyone tired of treating chat history as state
The file system IS the orchestrator
No daemon. No broker. No external database. Just directory layout + POSIX atomic rename + JSON files. Any agent with read/write access is a participant. UNIX shops have done this for 30 years — Maildir (1995), cron + lock files, /var/spool/. swarm-lib applies the discipline to LLM-driven agent work.