Written 2026-09-12 by Claude Fable 5.1 to the standard set in 5. Experiment/11. Detailed Design/specifications/C0-detailed-design-specification.md and shown by the reviewed sample chapter 06-execution-of-a-cell.md. Every line number below was read from the source at commit 6e2ee2c2b; the owning scripts have the same Git blob hashes at that commit as at commit b69ae5977, the commit the flow model 5. Experiment/11. Detailed Design/flow-model/flow_model.v001.json was verified against. The local-model draft operations/detailed-design/drafts/chapter_03.md was the lead for the inventories; section 14.1 lists what it got wrong. The flow model’s chapter 03 subgraph (fourteen nodes, twenty-nine edges of which one is inferred, three loops) is the source of the first state diagram of section 5, and every node identifier is named so the renderer can regenerate it. The model does not cover the event collector or the connectivity probe, which it says are not part of a cell’s life; they are described here from the code alone.
Owning files. 5. Experiment/1. Harness/scripts/dgx_fabric.py (561 lines, 26,034 bytes), the controller client; 5. Experiment/1. Harness/scripts/agent_backends.py lines 664 to 1063 (the shared lease back end, inside a file of 1,165 lines and 55,597 bytes whose remainder chapter 05 owns); 5. Experiment/1. Harness/scripts/collect_fabric_events.py (518 lines, 23,498 bytes), the controller event collector; 5. Experiment/1. Harness/container/dgx_connectivity_probe.py (95 lines, 4,095 bytes), the in-container probe; and the folder 5. Experiment/1. Harness/spark/ (four shell scripts, four model profiles, README.md, FABRIC_TELEMETRY_REQUEST.md and spark.env.example), the setup kit for the machine that serves the model. Files this chapter reads into but does not own: agent_backends.py lines 1064 to 1161 (the two concrete back ends that supply the lifecycle’s hooks, chapter 05), run_cell.py (the cell runner that calls the lifecycle once per pass, chapter 06), preflight_batch.py and lane_coordinator.py (the batch driver’s gate and its lane record, chapter 02), timing_lib.py and aggregate_timings.py (the timing record and its aggregator, chapters 09 and 10), capture_evidence.py (the container proof, chapter 04) and 10. User Interface/server.py (the results interface, chapter 11).
1. What a lease is and why the harness borrows a machine this way
A cell is one measured attempt: one coding agent, one maintenance task, one product build, one repetition seed, run at most five times within one cumulative token budget. On the local routes the model the agent talks to is served by one of two owned machines, the NVIDIA DGX Spark and, since 2026-09-09, the NVIDIA IGX Thor. Each machine runs a controller service called the Fabric, which owns the model server, sorts work into named queues called lanes, and grants time-limited permissions called leases. A lease is what lets one agent send inference requests to the machine for one pass. The harness never talks to a model container directly: the client refuses any controller address that is not the controller’s own port or its HTTPS proxy, because a direct model address would bypass admission and leases (dgx_fabric.py lines 158 to 190, validate_controller_url).
The lifecycle has four steps. An admission is a request for a slot, carrying the lane, the kind of work, the model alias, a priority, the lease length and an idempotency key, which the controller answers with an admission identifier and a state. While the state is queued or blocked_by_mode the client polls. When the state is ready the client claims it and receives the lease, whose record names the exact model served (the repository identifier, weights revision, quantisation, engine and serving container). While the agent runs, a background thread renews the lease with heartbeats. When the agent exits, the lease is released with an outcome, or, if the pass failed before a lease existed, the admission is cancelled so that the controller does not hold scheduler state for a process that gave up.
The controller can refuse a lease for two different reasons, and the design treats them differently (the docstring at agent_backends.py lines 774 to 786). It may be unable to grant one right now, because the operator put the machine into maintenance, the machine is rebooting, the lane has no capacity in the controller’s current mode, or the controller failed on its own side; that is not a failure of the cell, so the back end waits and asks again indefinitely, recording each round, and no tokens are spent while it waits because the agent has not been started. Or the controller may refuse the request itself, because the token is not authorised, the alias does not exist, or the model has no reviewed runtime for the protocol asked for; no amount of waiting changes that, so the pass fails at once. The operator ruling that fixed this is dated 2026-08-29 (dgx_fabric.py line 458).
The lifecycle is invoked once per pass by the cell runner through the back end’s run_pass (run_cell.py line 2536), and everything it observes is written into the cell’s manifest under fabric.passes.<k> after every controller event, through a callback the runner hands in (run_cell.py lines 2522 to 2532). Three other programs use the same client: the batch driver’s preflight gate runs a host-side lifecycle and a container-side one before a batch starts (preflight_batch.py lines 174 to 346); the results interface reads the controller’s status and releases an interrupted cell’s lease when the operator stops a batch (server.py lines 702 to 733 and 2329 to 2352); and the event collector, a separate process, keeps the controller’s own account of what it did in a campaign-wide store (collect_fabric_events.py), so that a cell’s waiting can be explained by a model load that happened on the other machine before any lease existed.
2. The reader’s map of the owning files
2.1 dgx_fabric.py
Nine regions. Lines 1 to 15 are the docstring and imports; the module uses only the standard library. Lines 17 to 38 are FabricError, whose four attributes (http_status, admission_state, reason, kind) let code tell apart the kinds of refusal without reading English out of a string. Lines 40 to 52 are the two allowed lanes, programming and research (48, the ruling of 2026-08-27 at 43 to 46), and FabricConfigurationError (51). Lines 55 to 77 name the two endpoints dgx and igx_thor (63), the environment-variable pair each reads (69 to 72), and the selector variable FIVEB_FABRIC_ENDPOINT (77). Lines 81 to 155 are FabricEndpoint (81), an immutable record of name, controller address and token with a redact method (96); resolve_fabric_endpoint (100 to 130); and fabric_endpoint_provenance (133 to 155), which never reads the token. Lines 158 to 190 are validate_controller_url. Lines 193 to 238 are the default transport over urllib (193 to 210) and the process-wide client generation counter (221 to 238) that keeps two lanes’ event streams distinguishable. Lines 241 to 450 are FabricClient: the constructor (244 to 265), _emit (271), redact (277), _request (280 to 316), status (318), catalog (321), the two inference probes (327 and 351), acquire (381 to 430), heartbeat (432), release (439) and cancel_admission (446). Lines 453 to 521 are the judgement of whether waiting can help: the three constant sets (466, 472, 479) and retryable_acquire_failure (486 to 521). Lines 524 to 561 are HeartbeatTask.
2.2 agent_backends.py lines 664 to 1063
The class _FabricLeaseBackend (664) with its docstring (665 to 670) and class attributes (672 to 675). The constructor (677 to 713) stores the endpoint, the lane, the alias, the priority, the lease length, the container image, the runner seam, the polling interval, the client factory, the container labels, the request preset (699 to 705) and the endpoint name, refusing an unknown one (708 to 712). _qualify_selected_endpoint_model (715 to 738). _admission_requirements (740 to 745). The five subclass hooks (748 to 765): _prepare, _compose_argv, _pass_env, _finish_error, _finish_usage. _acquire_lease (768 to 827). run_pass (829 to 1061): the lifecycle skeleton (835 to 867), the emit closure that copies every controller event into it (869 to 901), the client (903 to 905), the defaults (906 to 912), the guarded body (913 to 989), the stop handler (990 to 1000), the broad handler (1001 to 1004), the finally block (1005 to 1047) and the result (1049 to 1061).
2.3 collect_fabric_events.py
Six regions. Lines 1 to 63 are the docstring, which states why the collector exists, what it collects, where it lands, its two modes, what it costs a running cell (nothing) and the note that the controller does not honour the resume header of Server-Sent Events. Lines 74 to 95 are the store path, the eleven event kinds a cell’s timing can be explained by (80 to 88) and the stop handler. Lines 98 to 191 read the environment (98 to 114) and the store: the boot namespace (117 to 131), known_sequences (134), known_event_payloads (158) and last_model_states (180). Lines 194 to 265 open the stream (194), trim long text (208 to 219), drop repeated model states (222 to 246) and parse the stream’s data: lines (249 to 265). Lines 268 to 393 are collect (268 to 362) and _iter_events (365 to 393). Lines 399 to 465 read the store back: load_events (399), model_load_spans (415) and maintenance_windows (448). Lines 468 to 518 are main.
2.4 dgx_connectivity_probe.py
Lines 1 to 27 read the environment and choose the protocol requirements (23 to 26). Lines 29 to 43 are request, a bare urllib call with the bearer token and, when given, the lease header. Lines 46 to 95 are the script body: one admission, a poll, a claim, one heartbeat, one inference request on the chosen protocol, one JSON line of evidence, and a finally that releases the lease or cancels the admission.
2.5 The spark/ setup kit
README.md (3,376 bytes) names the four scripts in run order and where each runs: 01_bootstrap.sh (checks the machine and generates the bearer token), 02_download_model.sh <profile> (downloads a profile’s weights and records the revision hash in a manifest, the weights pin of the plan’s Appendix N section 6), 03_serve.sh <profile> (starts the vLLM container and waits up to 900 seconds for its health endpoint) and 04_verify_from_devbox.sh <host> (the acid test: health, model listing, one raw Anthropic-shaped request and one tool-using turn of the agent tool). spark.env.example holds the port, the image and the state folder; profiles/ holds four candidate models. FABRIC_TELEMETRY_REQUEST.md (11,166 bytes) is the request to the Fabric’s authors for per-request server-side timing attributable to a lease, which the harness cannot measure from outside (section 3 of that document lists what is missing). The kit predates the Fabric controller: it serves one model directly on port 8000 for the shakedown of Appendix N, and the lease lifecycle of this chapter addresses the controller on port 9710 instead.
3. The inputs
3.1 Command-line arguments
The client and the back end have no parser. collect_fabric_events.py parses at lines 469 to 486: --store (default 7. Monitoring/fabric_events.jsonl), --follow (stay connected instead of stopping once the retained history has been read), --timeout (seconds to wait for the controller, default 30), --idle-seconds (in snapshot mode, stop after this long with nothing new, default 20) and --summary (print what is stored and collect nothing). The probe takes no arguments. The runner’s --fabric-endpoint (run_cell.py 1543 to 1545) is chapter 06’s argument and is read here at 1719 to 1721.
3.2 Environment variables
| Variable | Read at | What it decides |
|---|---|---|
DGX_SPARK_FABRIC_URL, DGX_SPARK_FABRIC_TOKEN | dgx_fabric.py 69 to 72 and 120 to 127 (resolve_fabric_endpoint); collect_fabric_events.py 99 and 109; dgx_connectivity_probe.py 12 to 20; server.py 704 to 706 | The controller address and bearer token of the dgx endpoint, which is the endpoint chosen when nothing says otherwise (117) |
IGX_THOR_FABRIC_URL, IGX_THOR_FABRIC_TOKEN | dgx_fabric.py 71; dgx_connectivity_probe.py 14 | The same pair for the igx_thor endpoint |
FIVEB_FABRIC_ENDPOINT | dgx_fabric.py 77 and 117; run_cell.py 1721; dgx_connectivity_probe.py 11; preflight_batch.py 544 and 308 | Which named endpoint is active when no argument names one; unset means dgx |
DGX_SPARK_FABRIC_LANE | run_cell.py 1724 to 1731 and 2337; preflight_batch.py 188 and 201 to 204; dgx_connectivity_probe.py 51 | The lane the admission asks for; must be programming or research (dgx_fabric.py 48) |
DGX_SPARK_FABRIC_MODEL | run_cell.py 1682; preflight_batch.py 189; dgx_connectivity_probe.py 21 | The model alias, when no argument gives one |
DGX_SPARK_FABRIC_PRIORITY | run_cell.py 2339; preflight_batch.py 256 | The admission priority; default 50 |
DGX_SPARK_FABRIC_LEASE_SECONDS | run_cell.py 2340 to 2341; preflight_batch.py 258 and 265 | The lease length; default 900. The heartbeat interval is derived from it (dgx_fabric.py 535) |
DGX_SPARK_FABRIC_REQUEST_PRESET | run_cell.py 2346 to 2347 | The catalogued request preset the admission pins; unset means the controller’s served default |
DGX_SPARK_FABRIC_POLL_SECONDS | run_cell.py 2383 to 2384 | The polling interval of a queued admission; default 3 (dgx_fabric.py 247) |
DGX_SPARK_FABRIC_PROTOCOL | dgx_connectivity_probe.py 22 | Which inference route the probe exercises, responses or messages; set by the preflight at preflight_batch.py 325 |
FIVEB_LEASE_RETRY_SECONDS | agent_backends.py 113 to 123 (lease_retry_seconds), read at 795 | The wait between lease attempts while the controller cannot grant one; default 60; a zero or negative value falls back to the default |
FIVEB_CELL_CONTAINER | run_cell.py 1705 and 1713 to 1716; preflight_batch.py 310 | The image the agent runs in; required for any Fabric back end |
ANTHROPIC_API_KEY, FIVEB_CELL_CREDENTIALS | run_cell.py 1734 to 1749 | Their presence refuses the dgx_claude back end, so exactly one authentication source exists |
DGX_SPARK_FABRIC_LEASE_ID, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_CUSTOM_HEADERS | set, not read: agent_backends.py 755 to 757 and 1140 to 1145 (_pass_env), applied at 973 and removed at 1006 to 1008 | The lease identifier, the Fabric token and the lease header the container inherits by name for the duration of one pass |
3.3 Files read
| File | Read at | What it decides |
|---|---|---|
cells/<cell>/cell_manifest.json, member fabric.passes.<k> of the prior manifest | run_cell.py 2534 to 2535 (on a resume) | The saved lease of the pass being resumed, handed to acquire as saved so that an admission the controller still holds as leased is recovered (dgx_fabric.py 421 to 425) |
cells/<cell>/transcripts/pass<k>.container_id | agent_backends.py 516 to 542 (stop_container_from_cidfile), at 1014 to 1015 | Which container to stop on an abnormal exit |
cells/<cell>/transcripts/pass<k>.stderr.txt | agent_backends.py 1049 to 1050 | The error text the result carries when the process failed |
7. Monitoring/fabric_events.jsonl | collect_fabric_events.py 134 to 191 (the deduplication sets) and 399 to 412 (load_events); aggregate_timings.py 1073 to 1075 | What has already been stored; which model loads and maintenance windows overlap a cell’s waiting |
| The controller’s catalogue | agent_backends.py 228 to 254 (_catalogue_context_windows, cached per process and per endpoint) and 723 to 738 | The alias’s context window, recorded for the reader and used to derive the agent tool’s output cap and compaction point (chapter 05, agent_backends.py 295 to 358) |
3.4 Network
Every call goes through FabricClient._request (280 to 316) with a bearer token and a 30-second timeout (248), except the two inference probes, which allow 900 seconds (347, 375). The controller’s routes: GET /control/v1/status (318 to 319), GET /control/v1/catalog (321 to 325), POST /control/v1/admissions (399), GET /control/v1/admissions/<id> (411 to 412), POST /control/v1/admissions/<id>/claim (415 to 416), POST /control/v1/leases/<id>/heartbeat (433 to 435), DELETE /control/v1/leases/<id> (440 to 442), DELETE /control/v1/admissions/<id> (447 to 448), POST /v1/responses (327 to 349) and POST /v1/messages (351 to 379) with the X-Spark-Lease header, and GET /control/v1/events as a Server-Sent Events stream (collect_fabric_events.py 194 to 205). The agent inside the container reaches the same controller’s /v1/messages route: the container is given the controller address as its base address, the token by name and the lease header by name (agent_backends.py 478 to 481), and the controller’s host name is pinned into the container’s bridge network because the container cannot resolve the Tailscale name the host can (93 to 110, docker_fabric_host_args). The back end also runs docker run (437 to 514) and docker stop (539 to 542) on the host.
4. The happy path in order
4.1 Resolving the endpoint, run_cell.py lines 1712 to 1732
Before staging, the runner resolves which machine the cell talks to. resolve_fabric_endpoint (dgx_fabric.py 100 to 130) takes the argument, then the selector variable, then dgx (117), refuses a name outside the two (118 to 121), refuses a missing address or token by naming the variable that is missing (123 to 128), and validates the address (129). validate_controller_url (158 to 190) requires an http or https scheme with a host (167 to 169), the controller port 9710 or an HTTPS proxy on 443 (174 to 181, the Tailscale case), and no path, query or fragment (182 to 185), because a path would name a model’s own interface and bypass the leases. The runner then requires a lane in the allowed set (1724 to 1731), and on the dgx_claude route refuses any Anthropic credential in the environment (1734 to 1749, chapter 06). The endpoint’s address goes into the cell manifest’s fabric block (2334 to 2348) with the lane, the requested alias, the priority, the lease length and the request preset.
4.2 Building the back end, run_cell.py lines 2372 to 2393
Once per cell, before the pass loop, the runner builds one back end object with the endpoint’s address and token, the manifest’s lane, priority, lease length and preset, the container image, its own invoke_claude_cli as the runner seam, the polling interval, the container labels and the endpoint name. The constructor (agent_backends.py 677 to 713) stores them and refuses an unknown endpoint name (708 to 712). DgxCodexBackend (1064) and DgxClaudeBackend (1101) differ only in the hooks: what profile to write, how to compose the container command, what environment the pass needs, and how the transcript yields error text and usage (docstring, 665 to 670).
4.3 The per-pass call and the record callback, run_cell.py lines 2520 to 2547
For each pass on a Fabric route the runner defines _record_fabric (2522 to 2532), which stores the lifecycle dictionary under fabric.passes.<k>, promotes the catalogue’s context window into the manifest when the back end learned one, and rewrites cell_manifest.json whole. It reads the prior manifest’s saved lease on a resume (2534 to 2535) and computes the idempotency key (admission_key, 641 to 662): a continuing attempt keeps its earlier key so the controller hands the same admission back, and every other attempt asks under a fresh key, because the controller refuses a reused key with different fields, which is what defeated the resume of run set 039 (the docstring’s account). run_pass is then called with the prompt, the workspace, the cell folder, the pass number, the key, the two transcript paths, the saved lease and the callback (2536 to 2541).
run_pass starts by building the lifecycle skeleton (835 to 867): the key, empty admission and lease identifiers, empty state and reason, empty provenance, the start stamp, empty claim, heartbeat, release and interruption stamps, zero waiting seconds and rounds, empty admission, agent-process and release seconds, and an empty event list. The emit closure (869 to 901) is the client’s event callback: every event the client emits is appended to the list with the moment it was recorded and the controller’s own fields filtered to a safe set (_now_from_event, 646 to 662, which keeps identifiers, states, stamps, the provenance and, since Fabric 0.34.0, the request preset and reasoning policy, and drops everything else including any credential). An admission event sets the admission identifier (874 to 875); an admission_state event sets the state and reason (876 to 878); a claim or recovered_lease event sets the lease identifier, the model provenance and its unpacked fields, the preset, the reasoning policy and the claim stamp (879 to 895); a heartbeat sets the last heartbeat stamp (896 to 897); a release sets the release stamp (898 to 899). After every event the callback rewrites the manifest (900 to 901), so the record on disk is current while the pass is running, which is what the interface’s live view reads (section 4.13).
4.4 Qualifying the alias against the endpoint’s catalogue, lines 918 to 923 (flow-model node backend.qualify_model)
Inside the pass’s error boundary, _qualify_selected_endpoint_model (715 to 738) runs first. On the dgx endpoint it only records the context window from the catalogue or the reviewed fallback table (723 to 724, dgx_context_window 257 to 282). On any other endpoint it reads the live catalogue, refuses an alias absent from it (726 to 731), checks that the alias is served with the back end’s interface in the ready state and with the required capabilities (endpoint_backend_compatibility, 163 to 208), and records the catalogue’s context window or null (736 to 738). The comment at 914 to 917 gives the reason it runs inside the boundary: a refusal is then durably recorded as an unstarted pass rather than escaping without an error artefact. The endpoint name and the context window go into the lifecycle (919, 923), and the pass number and an empty container-id path are remembered for the argument composer (928 to 929). The subclass’s _prepare runs (930): nothing for Claude, and for Codex the rendering of a cell-local profile (1078 to 1082, chapter 05).
4.5 The admission and the waiting loop, lines 944 to 957 and 768 to 827 (nodes backend.waiting_for_lease, fabric.admission)
The back end asks for a lease with the lane, the workload maintenance-cell, the alias, the priority, the lease length, the idempotency key, the metadata that lets the controller’s metrics be filtered (the programme, the campaign, the project letter, the run set, the cell and the pass, 949 to 955), the requirements block with the interface, the capabilities and the preset (_admission_requirements, 740 to 745) and the saved lease. The comment at 940 to 943 states that the key is deliberately the same on every waiting round, so a controller that already holds an admission for the pass hands the same one back.
_acquire_lease (768 to 827) wraps the client’s acquire in an unbounded loop. On an exception it asks retryable_acquire_failure (dgx_fabric.py 486 to 521) whether waiting could win a lease. That function returns a reason for any HTTP status of 500 or above or in the set 408 and 429 (504 to 508), for a connection that could not be made (kind equal to unreachable, 509 to 510), and for a terminal admission state in the retryable set (blocked_by_mode, draining, maintenance, deferred, queue_full, expired, timed_out, 472 to 476) or whose reason text contains one of the retryable words (479 to 484); it returns nothing for a configuration error, for anything that is not a FabricError at all, for any other HTTP status, and for a malformed response (502 to 512). When the reason is nothing the exception is raised unchanged (803 to 804). Otherwise the round is recorded: the attempt count into lease_wait_rounds, and a waiting_for_lease event with the attempt, the interval, the reason and the redacted detail (805 to 816, written to the manifest through the callback). The back end sleeps the interval (819), and the seconds slept are added to lease_wait_seconds in a finally so that a wait cut short by a stop signal still counts (820 to 827). A stop signal is a SystemExit or KeyboardInterrupt, which the except Exception at 801 cannot catch, so it leaves the loop at once (docstring, 788 to 793).
Inside acquire (dgx_fabric.py 381 to 430) the admission request is posted (387 to 399), an admission without an identifier is refused (400 to 402), and the admission event is emitted (403).
4.6 Polling, the claim and the recovery, lines 405 to 430 (nodes fabric.admission_polling, fabric.claim, fabric.recovered_lease, fabric.admission_ended)
The client loops on the admission’s state (405 to 430), emitting an admission_state event on every round (407). While the state is queued or blocked_by_mode it sleeps the polling interval plus a random jitter of up to one second or a third of the interval, whichever is smaller, and reads the admission again (408 to 413). When the state is ready it claims the admission, refuses a lease without an identifier, emits claim and returns the lease (414 to 420). When the state is leased and the runner supplied a saved lease identifier, which happens on a resume of a pass whose earlier attempt was granted an admission the controller was never told to release, the client heartbeats that lease, emits recovered_lease and returns it (421 to 425). Any other state is terminal: the client raises a FabricError naming the state and the controller’s reason (426 to 430), and the back end’s loop decides whether to wait again or fail.
Throughout, _request (280 to 316) turns a connection failure into a FabricError of kind unreachable and a reply it cannot read into one of kind malformed_response (291 to 301), honours a 503 answer’s Retry-After header by sleeping and asking again (302 to 309), and turns any other non-success status into a FabricError carrying the status and the redacted detail (310 to 315). The token never appears in an error: redact (277 to 278) replaces it in every message.
4.7 The admission time, the pass environment, the container command and the first heartbeat, lines 968 to 978 (nodes fabric.heartbeat_started, fabric.heartbeat)
With a lease in hand, admission_seconds is computed from the two stamps the lifecycle already holds, the start and the claim, minus the seconds spent waiting (968 to 970; _stamp_gap, 626 to 643, returns null rather than zero when a stamp is missing). The comment at 958 to 967 explains why a stopwatch is not used: the loop’s own accounting of waiting is verified by tests that drive the clock from a fixed script, and one extra call would shift the script under them.
The pass environment is applied (971 to 974): the lease identifier for both back ends (755 to 757), and for Claude also the Fabric token as ANTHROPIC_AUTH_TOKEN and the lease header as ANTHROPIC_CUSTOM_HEADERS (1140 to 1145). The container command is composed (975 to 976). For Claude, _compose_argv (1120 to 1138) names the container-id proof file for this pass and removes any earlier one (1122 to 1127), remembers it so the abnormal-exit path can stop the exact container (1129 to 1131), and calls claude_fabric_container_argv (437 to 514), which keeps the measured runner’s exact flags minus the setting-sources flag (463 to 471), writes the container identifier to the proof file through Docker’s --cidfile (473 to 476), derives the output cap and the compaction point from the context window (480 to 491, chapter 05), and passes the controller address inline, the two secrets by name, the small-model redirect, the three variables that close the runner’s off-machine calls, and the workspace mount (496 to 514). Secrets never appear in the argument vector (docstring, 450 to 456).
HeartbeatTask (dgx_fabric.py 524 to 561) is then started (977 to 978). start (548 to 554) sends one heartbeat before the agent starts, proving the lease is writable and leaving heartbeat evidence even when a short pass finishes before the first interval (comment, 549 to 551), then starts a daemon thread whose interval is one third of the lease length, bounded to between one and sixty seconds (535). The thread (_run, 540 to 546) renews the lease at that interval until stopped; a failure inside it is stored and stops the thread, and is raised only when stop is called (556 to 561), so a failing heartbeat never interrupts the agent from another thread.
4.8 The agent runs, lines 979 to 989
The runner seam is called with the argument vector, the workspace and the two transcript paths (981 to 982); on the runner side this is chapter 06’s invoke_claude_cli (run_cell.py 319 to 329), and the container itself is chapter 04’s. agent_process_seconds is recorded in a finally so an agent killed by a stop signal still reports the seconds it consumed (983 to 987). The outcome is completed for a zero return code and failed otherwise, with a detail naming the status (988 to 989).
4.9 The end of the pass, with the stop, the release and the cancellation, lines 1005 to 1047 (nodes fabric.heartbeat_stopped, fabric.release, fabric.cancel_admission)
The finally block runs on every exit. The pass environment is removed (1006 to 1008). When the outcome is not completed, the container is stopped by its recorded identifier and a container_stopped_on_abnormal_exit event is recorded when a stop was issued (1009 to 1019; stop_container_from_cidfile 516 to 542 gives the container five seconds and never raises). The heartbeat thread is stopped (1020 to 1026); a failure it stored is raised here, and turns the outcome to failed with process status 75 and the redacted message as the detail. When a lease exists it is released with the outcome and the detail (1027 to 1036), the release seconds are recorded in a finally, and a release that raises also sets status 75 (1031 to 1033). When no lease exists but an admission was created, the admission is cancelled, because a queued or ready admission still owns scheduler state (1037 to 1045); a failed cancellation also sets 75. The lifecycle is recorded one last time (1046 to 1047). The client’s release (439 to 444) sends the outcome and the first 2,000 characters of the detail to the controller and emits release; cancel_admission (446 to 450) emits cancel.
4.10 The result to the runner, lines 1049 to 1061 (node backend.pass_result)
The standard error file is read back (1049 to 1050) and an AgentRunResult (362 to 370) is returned with the process status, the transcript path, the error text (through the subclass hook: for Claude, the tail of standard error when the status is not zero, 1147 to 1155; for Codex, the transcript’s failure events, 608 to 623), the reported usage (nothing for Claude, whose usage the ledger owns, 1157 to 1160; the parsed usage for Codex, 545 to 605), the provider identity dgx_spark_fabric, the model provenance from the lease, the argument vector and the lifecycle. The runner closes the agent span with the status (run_cell.py 2547), adds the pass’s waiting seconds to the cell’s total (2544 to 2545), places the four lifecycle durations on the clock (2564 to 2574, chapter 09), and at finalisation carries the last pass’s model provenance into metrics.json as the measured model’s exact identity (2957 to 2960) and the endpoint name into the cell and run manifests (2947, 3158 to 3161, 3181). A Fabric-level failure before the agent started reaches the runner as the result’s error text, and the invocation-error guard halts the cell as agent_invocation_error, invalid (2731 to 2745, chapter 06).
4.11 The preflight probes, preflight_batch.py lines 174 to 346
Before a batch is launched, probe_dgx_host (174 to 291) runs the whole lifecycle from the host: it resolves the endpoint and the lane and alias (188 to 203), reads the controller’s status and refuses maintenance mode (212 to 217, citing run set 034’s death under a controller restart) and a lane with zero capacity in the current mode (221 to 231), reads the catalogue and qualifies the alias for the protocol’s back end (232 to 246), acquires a lease under the workload preflight-host (248 to 262), heartbeats it, sends one inference request on the protocol (263 to 272), releases in a finally (274 to 276), and returns the evidence: the mode, the alias, the endpoint, the exact model and provenance, the admission and lease identifiers, the release state, the response identifier and the client’s events (277 to 285). probe_dgx_container (292 to 345) runs the same lifecycle from inside the cell image through dgx_connectivity_probe.py, copied into the image at /opt/5b/dgx_connectivity_probe.py (container/Dockerfile line 83), forwarding both endpoints’ variables by name and the selected endpoint and protocol as explicit values (314 to 327), with a 1,200-second limit (329 to 330), and reads the probe’s last line of output as JSON evidence (340 to 344). The probe itself (44 to 95) admits under the workload preflight-container with the validation purpose dgx_maintenance_connectivity, polls every three seconds, claims, heartbeats, sends one request, prints one JSON line, and in its finally releases the lease or cancels the admission (91 to 95). Both probes are gates of preflight_dgx (527 to 569), chapter 02.
4.12 The event collector, collect_fabric_events.py
main (468 to 518) installs the stop handler for both signals (503 to 504) and calls collect (268 to 362). collect reads the store’s known sequence keys, payloads and last model states (284 to 288), opens the stream (295), and for every line: breaks on a stop signal (300 to 301); on a line that carries no event, which is the only chance to notice that the retained history has been read because the controller holds the connection open, breaks in snapshot mode after the idle period (307 to 311); refuses with a RuntimeError a sequence number seen before with different content, because that means the controller restarted without a boot identity and the deduplication key is unsafe (315 to 326); skips a sequence already stored (327 to 331); drops a model state the controller has already reported unchanged (340 to 342, is_repeat_of_state 222 to 246, whose docstring records that such repeats were 15,387 of the first 21,350 events); and otherwise stamps the event with the moment it was received, trims its free text to 500 characters (208 to 219, because a download progress bar made the log 4.4 MB), and appends it with a flush (347 to 349). The store is campaign-wide by design (docstring, 38 to 42). aggregate_timings.py reads it back through load_events and model_load_spans (1073 to 1075; the spans at 415 to 445 pair each loading with its ready) and attributes a model start-up that overlaps a cell’s waiting to that cell (model_loads_during, 466 to 486; the model_load stage is one of the three machine-wait stages of timing_lib.py 1483).
4.13 The interface, server.py
The results interface builds its own client from the vault environment (702 to 709) and caches the controller’s status for ten seconds (712 to 733: reachability, maintenance, standby state, active leases, queue depth and mode). The live view reads the newest pass’s lifecycle from the manifest and reports the lease identifier, whether it was released, the admission state, the last heartbeat, the exact model, and whether the pass is waiting for a lease with its rounds and reason (1545 to 1566); it reads the lease’s expiry from the newest event carrying one (1773 to 1783). When the operator stops a batch, the interface gives the process its own moment to release, then releases the interrupted cell’s unreleased lease itself with the outcome cancelled (2329 to 2352).
5. The state machine of a pass’s lease
The first diagram is the flow model’s chapter 03 subgraph: its fourteen nodes, the four chapter 06 and chapter 04 nodes they connect to, and every edge with its line range and the record condition the model records as its witness. A renderer can regenerate it from the nodes whose chapter is 03 and the edges with an endpoint among them. The one inferred witness is marked.
stateDiagram-v2 state "cell.pass.invoking (06)" as INV state "backend.qualify_model" as QUAL state "backend.waiting_for_lease" as WAIT state "fabric.admission" as ADM state "fabric.admission_polling" as POLL state "fabric.claim" as CLAIM state "fabric.recovered_lease" as RECOV state "fabric.admission_ended" as ENDED state "fabric.cancel_admission" as CANCEL state "fabric.heartbeat_started" as HBS state "fabric.heartbeat" as HB state "container.started (04)" as CST state "container.exited (04)" as CEX state "container.stopped_on_abnormal_exit (04)" as CSTOP state "fabric.heartbeat_stopped" as HBX state "fabric.release" as REL state "backend.pass_result" as RES state "backend.pass_failed_before_start" as FAIL state "cell.pass.capturing (06)" as CAP INV --> QUAL : Fabric route, run_cell 2517..2540 (agent_invocation.attrs.backend in dgx_claude, dgx_codex; manifest fabric.passes.k exists) QUAL --> WAIT : alias served, agent_backends 918..925 (lease.fabric_endpoint set; stage lease_wait exists) QUAL --> FAIL : unhappy, alias not served or interface incompatible, 918 (fabric_endpoint null; admission_id null; returncode 70) WAIT --> WAIT : loop, retryable refusal, 795..826 (waiting_for_lease events; lease_wait.attrs.rounds >= 1) WAIT --> ADM : request accepted, 801 (admission event; admission_id set) WAIT --> FAIL : unhappy, non-retryable refusal, 802..804 (no admission event; returncode 70) ADM --> POLL : admission created, dgx_fabric 405..407 (admission_state events) POLL --> POLL : loop, queued or blocked_by_mode, 408..412 POLL --> CLAIM : state ready, 414..420 (claim event; lease_id and claimed_at set) POLL --> RECOV : state leased with a saved lease, 421..425 (recovered_lease event) POLL --> ENDED : unhappy, any other state, 426..430 (admission_state outside the four; admission_id set) ENDED --> WAIT : loop, retryable state, agent_backends 795..800 (waiting_for_lease reason begins "admission ended as") ENDED --> CANCEL : unhappy, non-retryable state, 1002..1049 (cancel event; lease_id null) CANCEL --> FAIL : unhappy, 1042..1049 (cancel event; returncode 70, or 75 when the cancel raised) CLAIM --> HBS : lease held, 975..978 (heartbeat event; last_heartbeat_at set) RECOV --> HBS : recovered lease held, 975..978 HBS --> HB : thread started, dgx_fabric 552..553 HBS --> CST : agent container launched under the lease, agent_backends 975..983 (container_id k exists; agent_process_seconds set) HB --> HB : loop, interval elapsed, dgx_fabric 541..543 (two or more heartbeat events) CEX --> HBX : outcome completed, 989..1028 (returncode 0) CEX --> HBX : unhappy, outcome failed, container already gone, 989..1028 (returncode not 0; no container_stopped event) CSTOP --> HBX : unhappy, container stopped, 1024..1028 (container_stopped_on_abnormal_exit event) HB --> HBX : stopped after the container exited, 1028..1029 HBX --> REL : no heartbeat failure stored, 1034..1041 (release event; returncode not 75) HBX --> REL : unhappy, a heartbeat failed during the pass, 1030..1033 (release event; returncode 75) [inferred: 75 is shared with a failed release and a failed cancel] REL --> RES : released, 1034..1063 (release event; released_at and release_seconds set) REL --> RES : unhappy, release raised, 1036..1041 (no release event; lease_id set; returncode 75) RES --> CAP : AgentRunResult returned, run_cell 2536..2553 (stage agent_invocation; stage capture_handoff) FAIL --> CAP : unhappy, error result returned, run_cell 2536..2553 (agent_invocation.ok false; stderr k exists)
The second diagram is the admission as the client observes it, whose states are the values the lifecycle’s admission_state field can hold together with the client-side events claim, recovered_lease, release and cancel. The controller may have states this repository has never seen; the client treats every state outside the four it names as terminal (426 to 430), and the retry judgement (472 to 484) names the terminal states and words it will wait through.
stateDiagram-v2 [*] --> queued : POST admissions 399; event admission 403 [*] --> blocked_by_mode : lane at zero capacity in the current mode queued --> queued : poll 408..413 every poll_seconds plus jitter queued --> blocked_by_mode : controller mode changed blocked_by_mode --> queued : mode gives the lane capacity queued --> ready blocked_by_mode --> ready ready --> claimed : POST claim 415..416; event claim 419; lifecycle.claimed_at 895 queued --> leased : a saved admission of a resumed pass leased --> recovered : heartbeat 423; event recovered_lease 424 queued --> terminal : draining, maintenance, deferred, queue_full, expired, timed_out, or any other state 426..430 blocked_by_mode --> terminal terminal --> queued : retryable (dgx_fabric 486..521), wait FIVEB_LEASE_RETRY_SECONDS, same idempotency key 940..957 terminal --> cancelled : not retryable; DELETE admissions 447..448; event cancel 449 claimed --> held : heartbeats 432..437 every min(60, max(1, lease_seconds / 3)) 535 recovered --> held held --> released : DELETE leases 440..442 with outcome completed, failed or cancelled; event release 443; lifecycle.released_at 899 held --> held_on_controller : release raised 1031..1033; the controller keeps the lease until it expires released --> [*] cancelled --> [*] held_on_controller --> [*]
6. The sequence of one pass on a Fabric route
The participants are the runner, the back end, the client, the heartbeat thread, the physical controller, the container holding the agent, and the files. The Codex back end follows the same sequence with its own hooks.
sequenceDiagram participant R as run_cell.run() 2520 participant B as _FabricLeaseBackend.run_pass (agent_backends 829) participant C as FabricClient (dgx_fabric 241) participant H as HeartbeatTask thread (dgx_fabric 525) participant F as Fabric controller participant A as docker run: agent (claude_fabric_container_argv 437) participant X as cell files R->>B: run_pass(prompt, pass k, idempotency_key, saved, record=_record_fabric) 2536 B->>B: lifecycle skeleton 835; emit closure 869 B->>C: fabric_client_factory(url, token, poll_seconds, event_callback=emit) 903 B->>C: catalog() on a non-dgx endpoint 725 C->>F: GET /control/v1/catalog 322 B->>B: qualify alias and interface 715..738; lifecycle.fabric_endpoint 919 B->>X: cell_manifest.json fabric.passes.k (record 900) loop waiting_for_lease (795..827): retryable refusal, sleep FIVEB_LEASE_RETRY_SECONDS B->>C: acquire(lane, maintenance-cell, alias, priority, lease_seconds, key, metadata, requirements, saved) 800 C->>F: POST /control/v1/admissions 399 F-->>C: admission id, state C-->>B: emit admission 403, admission_state 407 B->>X: manifest rewritten (900) loop queued or blocked_by_mode (408..413) C->>F: GET /control/v1/admissions/id 411 C-->>B: emit admission_state 407 end alt state ready C->>F: POST .../claim 415 F-->>C: lease id, model_provenance, expires_at, request_preset, reasoning_policy C-->>B: emit claim 419 (lifecycle.lease_id, provenance, claimed_at 879..895) else state leased with saved lease_id C->>F: POST /control/v1/leases/id/heartbeat 433 C-->>B: emit recovered_lease 424 else other state C-->>B: FabricError(admission_state, reason) 427 end end B->>B: admission_seconds from the two stamps 968 B->>B: os.environ: lease id, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_CUSTOM_HEADERS 971..974 B->>B: argv with --cidfile pass k .container_id 975 (1120..1138) B->>H: HeartbeatTask.start 977 (one heartbeat first, 552) H->>F: POST .../heartbeat every min(60, lease/3) s 541..543 B->>A: runner(argv, workspace, stream path, stderr path) 981 (run_cell.invoke_claude_cli 319) A-->>X: transcripts/pass k .container_id (Docker), pass k .stream.jsonl, pass k .stderr.txt A-->>F: POST /v1/messages with X-Spark-Lease, through the lease A-->>B: exit status; agent_process_seconds 986 B->>B: finally 1005: environment removed 1006..1008 opt outcome not completed B->>A: docker stop --time 5 by cidfile 1014 (516..543); event container_stopped_on_abnormal_exit 1016 end B->>H: stop 1022 (join 10 s, 558; stored failure raised 559..561 gives status 75) alt lease held B->>C: release(lease id, outcome, detail) 1030 C->>F: DELETE /control/v1/leases/id 440 C-->>B: emit release 443 (released_at 899); release_seconds 1035 else admission but no lease B->>C: cancel_admission 1042 C->>F: DELETE /control/v1/admissions/id 447 C-->>B: emit cancel 449 end B->>X: manifest rewritten one last time 1046 B-->>R: AgentRunResult(process_status, error_text, reported_usage, model_provenance, argv, lifecycle) 1051 R->>R: close agent span 2547; lease_wait_total 2544; four timing lines 2564..2574 (chapter 09)
7. The records, with their writers and readers
Every record this chapter’s code writes, and the members of other chapters’ records it fills. The lease lifecycle is the canonical record of this chapter; the manifest that holds it is chapter 06’s. Every record here is single agent only; chapter 14 will add the role where a record would split.
| Record | Writer | Fields or content | Readers |
|---|---|---|---|
cells/<cell>/cell_manifest.json, member fabric.passes.<k> (flow-model key lease) | _FabricLeaseBackend.run_pass 835 to 867 (skeleton), emit 869 to 901 (every controller event), _acquire_lease 805 to 827 (waiting rounds), 919 and 923 (endpoint and window), 968 to 970 (admission seconds), 986 (agent seconds), 999 (interruption), 1016 to 1019 (container stop), 1035 (release seconds); written to disk by the runner’s callback _record_fabric (run_cell.py 2522 to 2532) on every event and once more at 1046 | idempotency_key, admission_id, lease_id, admission_state, reason, model_provenance (with model_id, served_model, revision, runtime_version, engine, quantization, container_digest also unpacked to top level), exact_model, exact_model_id, request_preset, reasoning_policy, started_at, claimed_at, last_heartbeat_at, released_at, interrupted_at, lease_wait_seconds, lease_wait_rounds, admission_seconds, agent_process_seconds, release_seconds, fabric_endpoint, catalog_context_window_tokens, events (each with event in admission, admission_state, claim, recovered_lease, heartbeat, release, cancel, waiting_for_lease, container_stopped_on_abnormal_exit, recorded_at, and the controller’s id, state, reason, expires_at, created_at, last_heartbeat, updated_at, ready_expires_at, inference_base_url, exact_model, model_alias, model_provenance, request_preset, reasoning_policy, or for a waiting round attempt, retry_seconds, reason, detail) | run_cell.py 2534 to 2535 (the saved lease on a resume), 2544 to 2545 and 2564 to 2574 (the durations onto the clock), 2957 to 2960 (provenance into metrics.json); aggregate_timings.reconstruct 366 to 395 (the lifecycle as reconstructed activities for a cell without a timing file); server.py 1545 to 1566 (live view), 1773 to 1783 (lease expiry), 2337 to 2341 (the unreleased lease on a stop); gen_run_set_report.py through metrics.json; capture_evidence.collect_container_proof (the manifest’s fabric block, chapter 04) |
cells/<cell>/cell_manifest.json, member fabric (chapter 06, record table) | run_cell.py 2334 to 2348, written at 2352; context_window_tokens promoted at 2525 to 2530 | endpoint, url, lane, requested_alias, priority, lease_seconds, request_preset, passes | run_cell.check_cell_manifest_conflict 1617 to 1656 (refuses a resume onto a different endpoint, a Fabric cell resumed as non-Fabric, or a manifest with no endpoint whose route cannot be established); run_batch.py 427 and lane_coordinator.py 238 to 243 (the batch specification’s fabric block, a different file with the same member name); aggregate_timings.py 366; server.py 1545, 1832, 2339 |
cells/<cell>/transcripts/pass<k>.container_id (container_id) | Docker, through the --cidfile argument composed at agent_backends.py 473 to 476 and named at 1122 to 1127; removed before each pass (1127) | The started container’s identifier | stop_container_from_cidfile 516 to 542 at 1014; capture_evidence.collect_container_proof (chapter 04) |
cells/<cell>/transcripts/pass<k>.stderr.txt and pass<k>.stream.jsonl on a failure before the agent started (stderr, stream) | run_pass 1003 to 1004: the transcript is created empty and the redacted error is written as the standard error | The FabricError or configuration error text, token redacted, at most 1,000 characters | run_pass 1049 to 1050 and the _finish_error hook (1147 to 1155); run_cell._invocation_error 1463 and the guard at 2731 to 2745 |
cells/<cell>/metrics.json members fabric_endpoint, model_provenance, lease_wait_seconds (chapter 06, record table) | run_cell.py 2947, 2957 to 2960, 2969 | The endpoint name; the last pass’s provenance object; the cell’s total waiting seconds | aggregate_metrics.compute_cell 608 to 609 (exact_model_id, model_revision columns) and 632 (seconds_to_success, from which the runner already subtracted the waiting at 2932 to 2935); paired_analysis.py 182; extract_cell_metrics.py 640; gen_run_set_report.py 789 and 1556 to 1560; lane_coordinator.py 376 to 385 (summed into the lane record’s contention.fabric_lease_wait_seconds); timing_lib.py 1483 (the lease_wait and lease_admission stages are machine wait) |
<run set>/run_manifest.json members fabric_endpoints, cost_route, external_model_charge_usd, validation_purpose, cells.<cell>.fabric_endpoint (chapter 06, record table) | run_cell._finalize_manifests 3158 to 3167 and 3181 | The endpoints a run set touched (a list, because a run set that mixes them lists both), local_dgx, zero, and the Codex route’s exploratory marker | gen_run_set_report.py; server.py; attest_batch_outcome.py |
cells/<cell>/timings.jsonl lines lease_wait, lease_admission, agent_process, lease_release (chapter 09, record table) | run_cell.py 2564 to 2574 through StageTimer.record | The four lifecycle durations with measured false, and rounds on the first | chapter 09 |
7. Monitoring/fabric_events.jsonl (the flow model has no key for it) | collect_fabric_events.collect 347 to 349, appended and flushed per event | Per line, the controller’s event with sequence, occurred_at (the controller’s clock), kind, subject_id, data (free text trimmed to 500 characters), and the collector’s recorded_at and _fabric_event_namespace (the controller address and boot identity, 129 to 131) | collect_fabric_events.known_sequences 134, known_event_payloads 158, last_model_states 180, load_events 399, model_load_spans 415, maintenance_windows 448, and main --summary 490 to 500; aggregate_timings.collect 1073 to 1075 and model_loads_during 466 to 486 |
| The preflight evidence (chapter 02, record table) | probe_dgx_host 277 to 287 and probe_dgx_container 340 to 346, returned into preflight_dgx’s gates (562 to 569) | Mode, alias, endpoint, exact model, provenance, admission and lease identifiers, release state, response identifier, the client’s events | chapter 02 |
cells/<cell>/codex_home/config.toml (chapter 05, record table) | DgxCodexBackend._prepare 1078 to 1082 through render_codex_profile 396 | The Codex profile pointing at the controller | chapter 05 |
The lineage from the controller’s answers to the records and the campaign tables is drawn below. Every arrow is a writer or reader relationship from the table above.
flowchart LR F[Fabric controller<br/>admission, claim, heartbeat, release, events stream] C[FabricClient._emit 271<br/>admission 403, admission_state 407, claim 419,<br/>recovered_lease 424, heartbeat 436, release 443, cancel 449] F --> C E[run_pass emit 869..901<br/>lifecycle fields; waiting rounds 805..816] C --> E CB[run_cell._record_fabric 2522] E --> CB CM[cells/cell/cell_manifest.json<br/>fabric.passes.k] CB --> CM CID[transcripts/pass k .container_id<br/>Docker via --cidfile 473..476] E -.-> CID ERR[transcripts/pass k .stderr.txt<br/>on failure before start 1004] E -.-> ERR RES[AgentRunResult 1051] E --> RES TI[timings.jsonl<br/>lease_wait, lease_admission, agent_process, lease_release<br/>run_cell 2564..2574] RES --> TI M[metrics.json<br/>fabric_endpoint 2947, model_provenance 2957, lease_wait_seconds 2969] RES --> M CM --> M RM[run_manifest.json<br/>fabric_endpoints, cost_route 3158..3181] M --> RM COL[collect_fabric_events.collect 268] F --> COL EV[(7. Monitoring/fabric_events.jsonl 348)] COL --> EV AT[aggregate_timings.py<br/>reconstruct 366..395; model_loads_during 466; collect 1073] CM --> AT EV --> AT TI --> AT CT[(6. Metrics/cell_timings.csv, stage_timings.csv,<br/>pass_timing.csv, timing_summary.json)] AT --> CT AG[aggregate_metrics.compute_cell 398<br/>exact_model_id, model_revision 608..609; seconds_to_success 632] M --> AG CFM[(6. Metrics/cell_factor_matrix.csv, time_matrix.csv)] AG --> CFM LC[lane_coordinator.py 376..385<br/>contention.fabric_lease_wait_seconds] M --> LC RP[gen_run_set_report.py 789, 1556] M --> RP SV[server.py<br/>_controller_status 712; live view 1545; _lease_expiry 1773; stop release 2329] F --> SV CM --> SV SV --> UI[the results interface] PF[preflight_batch.py<br/>probe_dgx_host 174; probe_dgx_container 292] F --> PF PF --> PFE[the preflight evidence, chapter 02]
8. The loops and the waits
Five loops, four of them unbounded by design. _request (dgx_fabric.py 286 to 316) repeats a request that answered 503 after sleeping the Retry-After header’s seconds, at least one and sixty when the header is absent or unreadable, with no bound; every other request either returns or raises once, with a 30-second timeout (248) or 900 seconds for the two inference probes (347, 375). acquire’s polling loop (405 to 413) sleeps poll_seconds plus a random jitter of up to one second or a third of the interval, whichever is smaller, while the admission is queued or blocked_by_mode, with no bound; the runner sets the interval from the environment with a default of three seconds (run_cell.py 2383 to 2384). _acquire_lease (agent_backends.py 797 to 827) repeats a retryable refusal after sleeping FIVEB_LEASE_RETRY_SECONDS, default sixty, with no bound, and the only exit besides success is a non-retryable refusal or a stop signal. HeartbeatTask._run (dgx_fabric.py 541 to 546) wakes every one third of the lease length, bounded to between one and sixty seconds (535), until the stop event is set; stop joins the thread with a ten-second limit (558), and a thread that outlives the join is left as a daemon. The collector’s _iter_events (collect_fabric_events.py 365 to 393) reads lines until the stream ends or a stop signal arrives; in snapshot mode collect breaks after twenty quiet seconds (307 to 311), and the stream itself has the 30-second connection timeout (269). The container stop gives the container five seconds and the command sixty (539 to 540). The preflight’s container probe has a 1,200-second limit (preflight_batch.py 329 to 330), and the probe inside it polls every three seconds with no bound (dgx_connectivity_probe.py 61 to 63). The agent’s own run has no timeout here; chapter 06 bounds it through the token budget and the pass count.
9. The guards and refusals
| Guard | Where | What it refuses or records |
|---|---|---|
| Only two endpoint names | dgx_fabric.py 118 to 121; agent_backends.py 708 to 712 and 176 to 178 | An unknown name is a configuration error, never silently dgx |
| A named endpoint must have both halves of its credential | dgx_fabric.py 123 to 128 | The missing variable is named in the message |
| The controller address must be a controller root | validate_controller_url 158 to 190 | Wrong scheme or no host; a port other than 9710 unless HTTPS on 443; any path, query or fragment |
| The token is never printed | FabricEndpoint.redact 96; FabricClient.redact 277; every error site at agent_backends.py 805, 1002, 1026, 1033, 1045; preflight_batch.py 291 and 336 to 339 | The token substring is replaced by <redacted> before any message is written |
| Provenance never carries the token | fabric_endpoint_provenance 133 to 155 | The function never reads the token field |
| Only two lanes | dgx_fabric.py 48; run_cell.py 1727 to 1731; preflight_batch.py 201 to 204 | A lane outside programming and research is refused before staging |
| One authentication source on the measured local route | run_cell.py 1734 to 1749 | An Anthropic key or credentials mount present with dgx_claude refuses the cell |
| The alias must be served with the back end’s interface | _qualify_selected_endpoint_model 715 to 738; endpoint_backend_compatibility 163 to 208 | On a non-dgx endpoint: absent alias, interface not ready, or missing capabilities; the dgx route keeps its reviewed behaviour and records the window only |
| A context window is never copied between machines | _catalogue_context_windows 228 to 254 (keyed by endpoint and alias); 920 to 923 | A non-dgx endpoint’s window is the catalogue’s value or null, never a guess |
| Waiting only when waiting can help | retryable_acquire_failure 486 to 521; _acquire_lease 802 to 804 | Statuses 401, 403, 404, 422 and any other below 500 outside 408 and 429, a malformed reply, a configuration error and any non-Fabric exception are raised at once |
| A stop signal ends a wait at once | _acquire_lease 801 (except Exception); docstring 788 to 793 | SystemExit and KeyboardInterrupt are not caught by the loop |
| The idempotency key is stable across waiting rounds | 940 to 957; run_cell.admission_key 641 to 662 | The controller hands back the admission it already holds rather than one per round |
| An admission or lease without an identifier is refused | dgx_fabric.py 400 to 402, 417 to 418 | FabricError, non-retryable (no status, no kind, no state) |
| A stop is not a failure | run_pass 990 to 1000 | The outcome becomes cancelled, the interruption is stamped, and the exception is re-raised so the process exits |
| No orphaned container after an abnormal end | 1009 to 1019; stop_container_from_cidfile 516 to 542 | The container is stopped by its recorded identifier before the lease is released; a missing file or daemon leaves nothing to do |
| A heartbeat failure cannot interrupt the agent | HeartbeatTask._run 544 to 546; stop 559 to 561 | The failure is stored and raised only at stop |
| No abandoned admission | 1037 to 1045 | An admission that never became a lease is cancelled |
| The pass environment never outlives the pass | 1006 to 1008 | The lease identifier and the two Anthropic-named variables are removed in the finally |
| Secrets never enter the argument vector | claude_fabric_container_argv 450 to 456 and 497 to 499 | The token and the lease header are forwarded by name |
| The preflight refuses a controller in maintenance or a lane with no capacity | preflight_batch.py 211 to 231 | A batch admitted then would queue against stopped models |
| The collector never double-counts and never stores a lie | collect 315 to 331; is_repeat_of_state 222 to 246; trim 208 to 219 | A stored sequence is skipped; a reused sequence with different content stops the collector; ten-second repeats and long progress bars are dropped or cut |
| The collector asks for the repository token only | controller_token 108 to 114 | The message says never the administrator token |
| A resume must stay on one endpoint | run_cell.check_cell_manifest_conflict 1617 to 1656 | A different endpoint, a Fabric cell resumed as non-Fabric, or an unverifiable route is refused without writing |
10. Every unhappy path, in four parts
Each row states what the step is supposed to do and why it works that way, what goes wrong with the trigger and the code path, what is written with the status, and what it costs downstream for the driver, the scorer, the aggregator and the interface.
| Supposed to do, and why it works that way | What goes wrong: trigger and code path | What is written; status and exit | What it costs downstream |
|---|---|---|---|
| Resolve the endpoint from the argument, the selector or the default, failing closed so a cell never silently talks to the wrong machine | Unknown name, missing address or token, or an address that is not a controller root; resolve_fabric_endpoint 118 to 129 and validate_controller_url 167 to 185; caught by the runner at 1722 to 1723 | Nothing; SystemExit before staging | The driver sees a non-zero return with no metrics record; the cell has no disposition (chapter 06, section 10) |
| Qualify the alias against the endpoint’s live catalogue before any admission, so a stale configuration or another machine’s data is never trusted | The alias is absent, its interface is not ready, or a capability is missing; 726 to 735 raise a configuration error inside the pass boundary; the broad handler at 1001 to 1004 catches it | The transcript is created empty and the redacted error is written as pass<k>.stderr.txt; the lifecycle carries no endpoint, no admission; the result’s process status is 70 and its error text is the standard error | The runner’s invocation-error guard halts the cell as agent_invocation_error, invalid (2731 to 2745); the driver does not score it and aborts the batch after two in a row (run_batch.py 1034 to 1041); the aggregator refuses the cell for want of a ledger. The catalogue was already checked by the preflight (232 to 246), so this path means the catalogue changed between preflight and pass |
| Wait out a controller that cannot grant a lease right now, because that is not the cell’s failure and no tokens are spent while waiting | Maintenance, a reboot, a lane with no capacity, a 5xx, a 408 or 429, or an unreachable controller; _acquire_lease 801 to 827 | A waiting_for_lease event per round with the reason and the redacted detail, lease_wait_rounds and lease_wait_seconds in the lifecycle, rewritten to the manifest each round; the pass continues when a lease is won | The live view says the cell is waiting, with the rounds and the reason (server.py 1554 to 1566); the runner subtracts the waited seconds from seconds_to_success (2932 to 2935); a batch can wait for hours with no upper bound, which is the ruling of 2026-08-29 (dgx_fabric.py 458 to 461) |
| Fail fast when the controller refuses the request itself, because waiting cannot change it | HTTP 401, 403, 404 or 422, a malformed reply, or an admission that ended in a non-retryable state; retryable_acquire_failure returns nothing (502 to 521); the exception is re-raised at 804 and caught at 1001 to 1004 | As the qualification row: empty transcript, error in pass<k>.stderr.txt, status 70; when an admission was created it is cancelled (1037 to 1045), and a cancellation that raises makes the status 75 | The invocation-error halt and its consequences, as above |
| Claim a ready admission and hold the lease the controller returned | The controller returns an admission or a lease without an identifier; 400 to 402, 417 to 418 | FabricError with no status and no kind, so it is not retryable; the broad handler records it as above | As above; no campaign cell has been seen to take this path |
| Recover a saved lease on a resume rather than taking out a second one | The admission is leased but the runner supplied no saved lease identifier, or the saved identifier’s heartbeat fails; 421 to 430 and 433 to 435 | The admission ends in the client’s terminal branch (leased is not in the retryable set) or the heartbeat’s error propagates; the broad handler records it; status 70 | The invocation-error halt; the controller keeps the earlier lease until it expires. Whether the controller ever answers leased to a fresh admission was not checked (section 14.3) |
| Renew the lease every third of its length so that the agent’s requests keep being served | The controller becomes unreachable or refuses a heartbeat during the pass; _run 542 to 546 stores the failure and stops renewing | The agent keeps running until the lease expires on the controller and its requests start failing; at the end stop raises the stored failure (559 to 561), the outcome becomes failed, the status 75, and the lease is released with that detail (1020 to 1036) | The transcript ends on an API error and the runner halts the cell as agent_invocation_error (2731 to 2745). The flow model marks this edge’s witness inferred because status 75 is shared with a failed release and a failed cancellation (open item 12) |
| Release the lease with the pass outcome so the machine is free for the next cell | The controller cannot be reached at release time; release raises inside the finally at 1029 to 1033 | release_seconds is still recorded (1034 to 1036); no release event; released_at stays null; status 75 | The controller holds the lease until it expires (at most the lease length after the last heartbeat); the next cell’s admission may queue behind it; the interface reports the lease as unreleased (1557), and a stop from the interface tries the release again (2329 to 2352) |
| Cancel an admission that never became a lease | The cancellation raises; 1041 to 1045 | Status 75; no cancel event | The controller holds scheduler state for a process that gave up until the admission expires; the flow model records the same 75 ambiguity |
| Stop a running cell cleanly on the operator’s signal, releasing what the controller holds | A stop signal arrives while waiting or while the agent runs; the runner raises SystemExit from its handler (run_cell.py 3230 to 3252), which unwinds through 990 to 1000 | The outcome is cancelled, interrupted_at is stamped, the container is stopped by its identifier, the heartbeat is stopped, the lease is released or the admission cancelled, the lifecycle is written; then the exception is re-raised and metrics.json is never written | The cell has no disposition until the batch is resumed (chapter 07); before 2026-08-29 the SystemExit was swallowed by the broad handler and the pass was recorded as failed (comment, 991 to 996) |
| Stop the exact container an abnormal exit left behind, so no compute survives the pass | The identifier file is missing or empty, the daemon is gone, or the stop times out; stop_container_from_cidfile 529 to 542 returns false | No container_stopped_on_abnormal_exit event; nothing else | A container may keep computing until its own process ends; the operator finds it by the 5b-cell=1 label, which is what happened at the 2026-08-29 stop of run set 039 (comment, 517 to 527) |
| Record every controller event into the manifest as it happens, so the live view is current and a crash leaves the last state on disk | The manifest write inside the callback raises; run_cell._record_fabric 2530 to 2532 is not wrapped, and it runs inside the client’s event emission | The exception propagates out of acquire through the broad handler at 1001 to 1004; the lifecycle is recorded once more at 1046, which raises again if the disk is still unwritable | The pass fails with status 70 and an error that names the file system, not the controller; the runner halts the cell as an invocation error. No campaign cell has been seen to take this path |
| Prove the route before a batch spends anything, from the host and from inside the image | The controller is in maintenance, the lane has no capacity, the alias is absent or unqualified, or any client call raises; probe_dgx_host 212 to 246 and 288 to 290; probe_dgx_container 310 to 344 | The gate’s ok false with the redacted detail; the probe’s own admission or lease is released or cancelled in a finally (274 to 276; probe 91 to 95) | The batch is not launched (chapter 02); no cell is affected |
| Keep the controller’s event history so that a cell’s waiting can be explained | The store already holds the sequence with different content; collect 322 to 326 | RuntimeError naming the namespace and sequence; the events before it are already flushed | The collector stops; the operator must decide whether the controller lost its boot identity. Also: the stream dies mid-read (_iter_events 388 to 393 keeps what arrived and prints one line); or the environment lacks the address or token (98 to 114, SystemExit with a message naming the variable) |
| Run the collector alongside every batch, so the store covers the campaign | No timer, unit or launcher in the repository starts it (a search of the harness, the interface and the plan for the script’s name finds only its README lines and the aggregator’s import), and systemctl --user lists no unit naming it | The store’s last event is sequence 8065 of the dgx controller, recorded 2026-08-30 13:02 UTC (the file’s last line; 6,944 lines) | Every cell run since 2026-08-30, including every IGX Thor cell, has no model-load attribution: aggregate_timings.model_loads_during (466 to 486) finds no span, so a wait caused by a cold model start is left unexplained in stage_timings.csv and the interface’s Time view. The record is not wrong, only incomplete, and the docstring’s promise (13 to 16) is unmet for eleven days of the campaign |
11. The metrics this chapter produces and where each goes
The lifecycle feeds four things. First, lease_wait_seconds: summed per cell by the runner (2544 to 2545, and carried across a resume at 2432), written to metrics.json (2969), subtracted from seconds_to_success before that figure is written (2932 to 2935), so the primary time metric on time_matrix.csv and cell_factor_matrix.csv (aggregate_metrics.py 632) excludes waiting for the machine; also summed per lane into the coordinator’s record (lane_coordinator.py 376 to 385) and printed per cell in the run set report’s timing table (gen_run_set_report.py 789). Second, the four durations placed on the timing clock (2564 to 2574): lease_wait and lease_admission are machine wait in timing_lib.pass_timing_rows (1483, 1581 to 1583) and become machine_wait_seconds on pass_timing.csv and the matrix; agent_process and lease_release are reported figures in breakdown_seconds. For a cell without a timing file, aggregate_timings.reconstruct (366 to 395) rebuilds the same activities from the lifecycle’s stamps. Third, the model provenance: metrics.json’s model_provenance (2957 to 2960) becomes exact_model_id and model_revision on the matrix (aggregate_metrics.py 608 to 609), which is what lets analysis split by the exact weights rather than the mutable alias (the ruling of 2026-08-28 quoted at 2948 to 2956); paired_analysis.py 182 reads the same field to pair cells by model. Fourth, the endpoint: fabric_endpoint on metrics.json and the run manifest (2947, 3158 to 3161, 3181), read by the report and the interface. The request preset and reasoning policy the claim returned are in the lifecycle only (893 to 894); no campaign table carries them today, so a comparison across presets has to read the manifests.
The event store feeds one attribution: a model_load span that overlaps a cell’s waiting is placed inside that cell’s timeline by aggregate_timings.place_derived_rows and model_loads_during (466 to 486, 1007), and counts as machine wait. Since the store stops on 2026-08-30 (section 10, last row), the attribution is empty for every later cell.
12. The tests that exercise the mechanism
Seven files under 5. Experiment/1. Harness/scripts/tests/ cover this chapter. test_dgx_fabric.py (seven tests, lines 13 to 156) covers the event deduplication key, the refusal of direct model addresses and the acceptance of the HTTPS proxy, the full queued, blocked, ready, claim, heartbeat, inference and release sequence with its headers, the stable idempotency key across a restart, the Messages probe’s headers, and that the token never appears in an error. test_lease_wait.py (fifteen tests, 51 to 389) covers the retry judgement in both directions, the unreachable and HTTP classifications, the retry interval and its override, a maintenance window followed by a normal pass, the separate recording of waited seconds, the fast failure of a refused request, and the three stop-signal cases: during the wait, during the wait with an admission left behind (cancelled), and while the agent runs (not recorded as a failure). test_fabric_endpoints.py (fifteen tests, 14 to 143) covers endpoint resolution, the selector, the closed failures naming the variable, the Thor address validation, the catalogue qualification on both endpoints, and that provenance and redaction never carry the token. test_fabric_endpoint_wiring.py (eighteen tests, 36 to 519) covers the specification’s endpoint reaching the environment, the host-name pin into the container, the Thor probes, the preflight’s reading of the endpoint, the runner end to end against Thor and unchanged against the default, the three resume refusals across endpoints, and the per-endpoint cache. test_agent_backends.py (twenty-seven tests, 31 to 588, with three more at 2286 to 2439 in test_run_cell.py) covers the container command’s secrets and labels, release after success, failure, timeout and interruption, cancellation when the claim fails, the request preset on admission and claim, the Thor qualification before any admission, the pass environment set and removed, the Fabric failure reported and released, and the output cap arithmetic. test_collect_fabric_events.py (twenty-two tests, 60 to 326) covers storage, deduplication, the boot namespace, the legacy sequence reuse refusal, a dying stream, a quiet stream, the ten-second repeats, the trimmed progress bar, the load spans and maintenance windows, the clock ordering, and the token message. test_preflight_batch.py covers the two DGX gates and the Messages protocol (78 to 108). The interface’s test_provider_catalog.py covers the catalogue roster.
Against section 10: the endpoint refusals, the qualification refusal, the waiting loop, the fast failure, the heartbeat failure reported at release, the stop during the wait and during the agent, the cancellation of a left-behind admission, the sequence reuse and the dying stream are each covered by a named test above. No test covers an admission or lease returned without an identifier, the leased answer without a saved lease, a release that raises, a cancellation that raises, a container stop that fails, or a manifest write failing inside the record callback; and nothing in the repository tests or schedules the collector’s follow mode.
13. The dated incidents that shaped the code
The lane rule dates from the operator ruling of 2026-08-27 that measured maintenance may run in the controller’s auto mode and its research mode, so the repository token is authorised for both lanes and the harness accepts either (dgx_fabric.py 40 to 48; run_cell.py 1727 to 1731; the probe at 48 to 50); the same day’s rulings made the pinned Claude runner, not Codex, the measured agent on the local route (agent_backends.py 1105 to 1112; harness_implementation_spec.md section 12). The wait-rather-than-die rule dates from 2026-08-29 (dgx_fabric.py 455 to 460) and the whole judgement of which refusals are worth waiting through is kept in one place so a reader can check it without tracing call sites (462 to 484). The same day’s stop of run set 039 shaped three things: the SystemExit a stop raises used to fall through to the broad handler and record the pass as failed (990 to 996); the interrupted pass’s container kept computing invisibly until an operator found it by label, which is why the container is stopped by its recorded identifier (1009 to 1013; 517 to 527); and the resume died eighteen seconds in on an idempotency key already used with different fields, which is why a fresh attempt asks under a fresh key (run_cell.py 641 to 662). The renaming of the workload and the structured metadata for the controller’s metrics is dated 2026-08-29 (931 to 938). The container-id proof file dates from the transparency audit of 2026-08-29 (473 to 475). The collector is dated 2026-08-30 (line 2) and records the measurement behind it: the controller unloads a model ten minutes after its last request and loading it back takes about a hundred seconds across six cold starts (5 to 12), that three quarters of the first 21,350 events were unchanged repeats (222 to 235), and that progress bars made the log 4.4 MB (208 to 214). Run set 034’s first attempt was killed by a controller restart into maintenance mode, which is why the preflight refuses that mode (preflight_batch.py 212 to 216). The per-client generation counter exists because the lane coordinator (LP-16) runs two endpoint lanes concurrently and a resumed lane’s fresh sequence must not be conflated with its predecessor’s (221 to 238). The Tailscale HTTPS proxy on port 443 was admitted as a controller root when the interface began reaching the controller over it (174 to 181). Fabric 0.34.0 of 2026-09-05 added the request preset the admission pins and the reasoning policy the claim reports, recorded because the runner cannot choose its own reasoning setting (654 to 659; 699 to 705; the two tests of 2026-09-05 at test_agent_backends.py 222 and 323, tracker item 2.36). The IGX Thor was added on 2026-09-09 as a second named endpoint that inherits nothing from the DGX: its model, interface and context are read from its own catalogue (55 to 63; 715 to 722; the cache keyed per endpoint at 228 to 234). The output-token cap and compaction point derived from the catalogue’s window are chapter 05’s, but the incident that produced them is the local route’s: a cell died one token over a 32,768-token context before it had read a file (324 to 341).
14. The weakest claim, what was not checked, and the token line
The weakest claim
The weakest claim is the last row of section 10, that nothing runs the collector. It rests on a search of the repository for the script’s name in shell, Python, unit and Markdown files, and on systemctl --user list-units showing no unit whose name contains fabric; a unit under a different name, a cron entry, or a process started by hand on another machine would not have been found, and the store’s last line is the only positive evidence. The second weakest is the interface’s line numbers in sections 4.13 and 7: server.py changed between the flow model’s commit and this one, so its lines here differ from the flow model’s record entries, which the currency check will flag until the model is regenerated.
14.3 What was not checked
Whether the controller ever answers leased to a fresh admission, and what states beyond the four the controller has actually returned in the campaign; finding out means reading every manifest’s admission_state and events, which is a results-integrity script under the B4 contract. Whether HeartbeatTask.stop’s ten-second join can leave a renewing thread alive past the release, and what the controller does with a heartbeat that arrives after a release. Whether the Codex back end’s hooks (1064 to 1099) behave as chapter 05 will describe; they were read only for the lifecycle’s use of them. The setup kit’s four scripts were read by head and digest, not whole, because they serve the shakedown of Appendix N and not the lease lifecycle. FABRIC_TELEMETRY_REQUEST.md was read by heading only.
14.4 Token line
The session that wrote this chapter had consumed 455,732 tokens of context by the time writing began, measured as the difference of the remaining-token counter (15,000,000 at the start of the session, 14,544,268 at the last reading before the file was written); of that, 236,446 had been spent before chapter 09 was written and the remainder on this chapter’s materials: the flow model’s chapter 03 subgraph, the draft, the digest sections for the five owning items, the whole of dgx_fabric.py, collect_fabric_events.py and the probe, the back end’s four hundred lines in numbered form, and the call sites in the runner, the preflight, the coordinator, the aggregators and the interface. No harness script was run and no model tokens were spent by the harness.