VERI Administration Guide

1. Infrastructure Overview

Hardware

Total pooled memory: 576 GB across 5 nodes.

Network Topology

• Thunderbolt 5 bridge: Mac A (10.0.0.1) to Mac B (10.0.0.2) — 120 Gbps
• Ethernet LAN: DGX Sparks on 192.168.5.x and 192.168.6.x
• Tailscale mesh: Mac A + phone + Windows — hostname veagents-mac-studio.tail54223a.ts.net
• Tailscale funnel: Dashboard (18790), Gateway (18789), VGI API (8900)

Key Ports

2. Service Management

All services run as launchd agents in ~/Library/LaunchAgents/. They auto-start at boot and auto-restart on crash.

Common Commands

# List all VERI services
launchctl list | grep "^[0-9]" | grep ai.

# Restart a service
launchctl kickstart -k gui/$(id -u)/ai.openclaw.gateway

# Stop a service
launchctl bootout gui/$(id -u)/ai.openclaw.gateway

# Start a service
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.openclaw.gateway.plist

# View service details
launchctl print gui/$(id -u)/ai.openclaw.gateway

Core Services

After Code Changes

The dashboard server (Node.js) must be restarted to pick up changes to server.js:

cd ~/openclaw-ui && npx vite build
launchctl kickstart -k gui/$(id -u)/ai.openclaw.dashboard

The agent engine (Python) auto-restarts via launchd — just save the file.

3. Agent Engine

The Agent Engine provides a tool-use agent loop. Any model the router selects can power it.

How It Works

Users describe a task in plain English. The engine loops: sends the task to the model, the model picks tools (bash, read/write/edit files, glob, grep), the engine executes them with security checks, feeds results back, and repeats until done.

Endpoints

• Health: GET http://127.0.0.1:18796/v1/agent/health
• Run: POST http://127.0.0.1:18796/v1/agent/run (SSE stream)
• Stop: POST http://127.0.0.1:18796/v1/agent/stop

Dashboard Access

Agent Mode is the default in the Chat view. Users toggle between Agent Mode (tools) and Chat Mode (conversation only). A folder input lets users point the agent at a specific project.

Configuration

Environment variables (all optional — auto-discovered from nodes.json):

Audit Trail

All tool executions log to ~/.openclaw/logs/agent-engine/tool_audit.jsonl:

{"ts": "2026-03-27T20:03:11-0400", "session_id": "abc123", "tool_name": "bash", "args": "{'command': 'ls'}", "duration_ms": 12.9, "blocked": false}

4. Model Fleet

Node Registry

The model fleet is defined in ~/.mlx-bench/nodes.json (operator-managed, mode 600).

Escalation Chain

When a model cannot handle the request (context too large, node down), the proxy escalates:

glm-4.7-flash → qwen3-next:80b → llama-3.3:70b → minimax-m2.5 → glm-4.7-358b

Quarantine

Nodes with 10+ consecutive health failures are auto-quarantined. They are reprobed every 300 seconds and auto-reinstated on recovery.

Memory Pressure

When available memory on Mac A drops below 16 GB, requests route to remote nodes automatically.

5. Tenant Administration

Dashboard Admin Panel

Access via the Dashboard at /admin. Requires admin role.

User management:

• View all users, plans, usage
• Promote/demote admin status
• Manually adjust quotas
• Clone agent workspaces for new tenants

Plans

API Keys

Tenants create API keys via the Dashboard or API. Keys are per-tenant and scoped to their sandbox.

# Create key for a tenant (admin)
curl -X POST http://localhost:18790/api/tenant/keys \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"label": "production"}'

Tenant Isolation

Each tenant gets:

• Isolated Docker sandbox (when available)
• Separate memory namespace
• Private file storage
• Scoped API access
• Per-tenant billing meter

6. Security Operations

Four Defense Layers

Incident Response

Security incidents are:

1. Logged to ~/.openclaw/incidents/<id>.json
2. Broadcast to all agents via VPM mailbox
3. Alerted to the operator via WhatsApp

Critical incidents auto-freeze agent wallets.

Audit Logs

7. Monitoring & Logs

Health Check

# Quick health
curl -s http://127.0.0.1:18790/api/health/deep | python3.14 -m json.tool

# Agent engine health
curl -s http://127.0.0.1:18796/v1/agent/health

# MLX proxy status
curl -s http://127.0.0.1:8801/v1/models

Log Locations

All logs are in ~/.openclaw/logs/ unless noted otherwise:

# Real-time monitoring
tail -f ~/.openclaw/logs/gateway.log
tail -f ~/.openclaw/logs/agent-engine.log
tail -f ~/.mlx-bench/proxy.log

# Agent engine audit trail
tail -f ~/.openclaw/logs/agent-engine/tool_audit.jsonl

Billing Data

• Billing database: ~/.openclaw/data/billing.db
• Interaction store: ~/openclaw-workspace/projects/llm-usage-tracking/data/interaction_store.db
• MLX proxy metrics: ~/.mlx-bench/metrics.jsonl

8. Backup & Recovery

Encrypted backups run automatically. For manual operations:

# Run backup
~/openclaw-workspace/automation/backup-recovery/backup.sh

# Restore
~/openclaw-workspace/automation/backup-recovery/restore.sh <backup-file>

Critical Files to Protect

• ~/.mlx-bench/nodes.json — Model fleet (operator-managed)
• ~/.openclaw/openclaw.json — Platform config
• ~/.openclaw/wallets/master.enc — HD wallet seed (encrypted)
• ~/.openclaw/data/billing.db — Billing data

9. Social Media (Amplify)

The Amplify agent manages social media across 8 platforms. All content is draft-only — nothing posts without human approval.

Connected Platforms

Access via the Amplify tab in the Dashboard. Manage OAuth connections to:

• Twitter/X, LinkedIn, Instagram, TikTok, YouTube, Telegram, Discord, Reddit

Content Approval Flow

1. Agent drafts content based on user request or scheduled task
2. Draft appears in the Approval Queue
3. Admin reviews, edits if needed, approves or rejects
4. Only approved content is published

Backend

• Connectors API: GET /api/productivity/connectors (port 3210)
• Drafts API: GET /api/amplify/drafts, POST /api/amplify/drafts/:id/approve

10. Agent-to-Agent Communication

VACP Service

The VACP (Verifiable Agent Communication Protocol) service runs on port 8940.

Contra Protocol (P2P)

The Contra bridge runs on port 8920 for P2P intent discovery via Waku.

Agent Marketplace

The Agent LinkedIn marketplace (/api/marketplace/*) enables:

• Listing creation (compute, data, services, collaboration)
• Automated matching
• HTLC settlements with dispute resolution

11. Troubleshooting

Service Won't Start

# Check error log
cat ~/.openclaw/logs/<service>.err.log | tail -20

# Check if port is in use
lsof -i :<port>

# Force restart
launchctl kickstart -k gui/$(id -u)/ai.<service-name>

Agent Engine Returns Errors

1. Check health: curl http://127.0.0.1:18796/v1/agent/health
2. Check logs: tail ~/.openclaw/logs/agent-engine.err.log
3. Verify MLX proxy: curl http://127.0.0.1:8801/v1/models
4. Restart: launchctl kickstart -k gui/$(id -u)/ai.agent-engine

Dashboard Shows Old Code

The Node.js server caches server.js in memory. After editing:

cd ~/openclaw-ui && npx vite build
launchctl kickstart -k gui/$(id -u)/ai.openclaw.dashboard

Model Returns Empty Responses

1. Check MLX proxy logs: tail ~/.mlx-bench/proxy.log
2. Verify node is healthy: curl http://<node-ip>:<port>/v1/models
3. Check quarantine state: cat ~/openclaw-workspace/projects/llm-router/registry/quarantine_state.json
4. Check memory pressure: sysctl hw.memsize and vm_stat