Pipeline Scripts Reference

18 Python and Go infrastructure scripts that power the CodeSleuth AI pipeline. These scripts handle environment bootstrapping, file write safety, parallel execution, intelligence, database safety, pipeline operations, and post-build verification.

Environment & Bootstrap

bootstrap.py

Validates toolchain, installs dependencies, and generates .env.example. Run at Agent 3 activation before any build tasks proceed.

spec_validate.py

Validates TASK-GRAPH.md consistency before the !build gate opens. Catches structural issues before the Builder begins execution.

python scripts/spec_validate.py --task-graph artifacts/build/TASK-GRAPH.md

Exit 0 = pass. Exit 1 = fail with specific error messages per failing check.

File Write Pipeline

Every file write by the Builder passes through a 5-step pipeline: secret scan → staged write → hot compilation → SAST scan → write log. Each step is a separate script.

staged_write.py

Stage a file write with a terminal diff and an Apply / Edit / Reject prompt. The human sees exactly what will change before any file is touched.

secret_scan.py

Pre-write secret scanner. Runs before every file write and hard-blocks the write if any secret pattern is found. Repair context is returned to the Builder.

write_log.py

Records every approved write to WRITE_LOG.jsonl. This log is consumed by commit_by_task.py to group commits by task.

python scripts/write_log.py record --task TASK-NNN --file path/to/file.py

sast_scan.py

Post-write SAST scan. Runs non-blocking after each approved write. Uses gosec, bandit, or semgrep when available, with an inline regex fallback. Findings accumulate in SAST_FINDINGS.jsonl for the Security agent to review.

check_coverage.py

Enforces a minimum coverage threshold after test contracts pass. A result below the threshold blocks DONE marking for the task.

Parallel Execution

parallel_runner.go

Go goroutine batch executor. Runs multiple tasks simultaneously using goroutines and automatically detects write conflicts before execution begins.

Intelligence

semantic_search.py

Embedding-based RAG search over the codebase. Agent 3 queries this before each architecture-class task to find relevant prior code and avoid duplication.

python scripts/semantic_search.py search --query "validate user input" --index artifacts/build/SEMANTIC_INDEX.json

Uses Ollama nomic-embed-text when available; falls back to TF-IDF if Ollama is not running.

Database

migration_safety.py

Schema delta detection, bidirectional SQL generation, and data-loss risk assessment. HIGH-risk migrations block the pipeline until !migration-approve is issued.

Pipeline Operations

pipeline_replay.py

Restore the codebase to the git checkpoint created after a specific task, then reset all later tasks to PENDING. Requires commit_by_task.py to have run, so each task has a corresponding git commit.

python scripts/pipeline_replay.py --task TASK-NNN --project-dir .

Invoked by the !replay TASK-NNN Orchestrator command. See Command Reference.

session_fork.py

Fork the pipeline into two parallel branches for side-by-side comparison of alternative implementations or architectures.

Invoked by !fork and !merge Orchestrator commands.

register_project.py

Registers or updates a project entry in config/context.json → registered_projects. Called by Agents 3–6 at each stage completion to keep the registry current, and by Agent 7 after every maintenance cycle. Enables the always-resident pipeline — registered projects auto-resume on the next Start agent 0 without requiring re-activation.

pipeline_improve.py

Post-run telemetry analyzer. After a pipeline completes, analyzes timing, failure rates, and retry patterns to generate PROP-NNN improvement proposals.

Invoked by !pipeline-health and !pipeline-promote commands.

token_tracker.py

Session token usage tracking and !budget reporting. Reads per-model pricing from config/pricing.json.

Pricing data from config/pricing.json supports multiple Claude models with separate input/output/cache rates.

Post-Build

container_verify.py

Build the Docker image and run the full test suite inside the container. Auto-generates a Dockerfile.test if none exists. A passing test suite that fails inside the container indicates a genuine environment mismatch.

Invoked by the Verifier agent in Phase 12B. See Verifier.

contract_test.py

Start the application, call every endpoint declared in INTERFACES.md with minimal requests, and validate status codes and response fields. Provides runtime API contract verification beyond static type checking.

python scripts/contract_test.py --interfaces artifacts/build/INTERFACES.md --base-url http://localhost:8080

Invoked by the Verifier agent in Phase 6D.

commit_by_task.py

Read WRITE_LOG.jsonl, group all writes by task ID, create one conventional git commit per task, and generate PR_BODY.md with a structured pull-request description.

python scripts/commit_by_task.py --project-dir .

Also required by pipeline_replay.py — each task needs a commit reference for checkpoint restoration.