patchbay

New in 0.2.4: patchbay audit. Free CLI to scan an MCP server for OAuth 2.1 gaps. Read the November 2026 sweep (25 public MCP servers scanned, 2 of 8 reachable-and-public ones expose mutating tools without auth) or try it:

npx -y @patchbay-nebius/agent audit https://mcp.example.com/mcp

21 live checks + 1 retired, against the MCP 2026-07-28 authorization spec and the OAuth 2.1 / RFC 8252 / 7591 / 8707 / 9068 / 9207 / 9728 stack it builds on. Context-sensitive severity, --pr-draft for CI, --auth for walled servers. Jump to the audit section for the full flag list.

Hosted scheduled scans + alerts coming Q1 2027 for teams. CLI stays free and MIT forever. Interest check · email waitlist.

New in 0.4.0: shareable scan URLs + badge. Add --publish to your audit and get back a public report URL plus an embeddable score badge:

[![patchbay](https://patchbay-mcp.dev/api/badge/https%3A%2F%2Fmcp.example.com%2Fmcp.svg)](https://patchbay-mcp.dev/report/<id>)

New in 0.5.0: AUTH-006 (RFC 8252 §7.3 loopback redirect check) + GitHub Action. Drop this in your CI to fail builds on MCP OAuth gaps:

- uses: GodSmasher/patchbay-audit@v1
  with:
    url: https://your-mcp-server.example.com/mcp
    fail-on: high

Separate repo: GodSmasher/patchbay-audit · SARIF output into code scanning supported.

---

Works with

 

Describe an integration in one sentence. Get a tested connector (TypeScript or Python) that already passed its own tests.

# TypeScript (vitest + injected fetch)
patchbay build "When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag"

# Python 3.12 (pytest + injected async transport)
patchbay build --lang=python "When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag"

90.2% green on TS Core 20 · 73.3% green on Python Core 20 · 70.0% on Python Brutal 20. Measured across 192 live Nebius Sandbox runs with per-run JSON committed to the repo. Fine-tune live on Nebius AI Cloud (Qwen3-30B-A3B on 173 of patchbay's own red→green repair pairs, served from an H100 NVLink GPU behind an OpenAI-compatible vLLM endpoint). Dual-language toggle on the live site at https://patchbay-mcp.dev, public uptime at https://patchbay.openstatus.dev, open-source training dataset on 🤗 patchbay-nebius/connector-repair-dataset.

The numbers

Every row is a live Nebius Sandbox run with the per-attempt files, test report and sandbox operation id committed under bench/runs/. Not cherry-picked cases. Full tables and per-prompt breakdowns in bench/RESULTS-variance.md.

Set	Shape	Runs	Green	First-try	Cost / build
TS Core 20	common integrations (Stripe→Mailchimp, Shopify→Airtable, …)	92	90.2%	52.2%	$0.07
TS Brutal 10	adversarial (OAuth refresh, WebSocket, HMAC, cursor paging)	13	69.2%	23.1%	$0.18
Python Core 20	same 20 prompts as TS Core, Python+pytest sandbox	60	73.3%	31.7%	$0.07 / green build
Python Brutal 20	adversarial (SOAP, GraphQL federation, long-poll, dedup)	40	70.0%	—	$0.16
TS × Nemotron Super ablation	strong-step downgraded 550B → 120B, same prompts	60	46.7%	8.3%	$0.06

Reading these: the common-case Python gap (17 points vs TS Core) closes on hard cases. Python Brutal matches TS Brutal within one point. The Super ablation shows Ultra isn't overkill: dropping to the mid-tier halves the green rate because the saved per-call cost gets eaten by extra repair attempts.

How it works

patchbay runs five steps:

1. Plan. A fast model turns the sentence into source/target apps, trigger, action, docs queries, field mapping. Nemotron 3.5 Lightning.
2. Research. Tavily search + extract, biased toward each vendor's own domain, feeding a mid-tier model that produces a normalized API spec (endpoints, auth, example payloads). Nemotron 3 Super.
3. Generate. A strong model writes the connector, its tests and a README. TypeScript (src/connector.ts + vitest) by default; --lang=python writes connector.py + pytest. Nemotron 3 Ultra, 32k tokens, temperature 0.1.
4. Verify. A Nebius Token Factory sandbox pulls the right base image once and installs the test deps (vitest + typescript, or pytest + pytest-asyncio + pytest-json-report) into a checkpointed image. Every attempt forks that checkpoint with the generated files and networking disabled. For TS the gate is tsc --noEmit then vitest run; for Python it is python -m compileall then pytest -q --json-report.
5. Repair. When a run is red, the strong model gets the file blocks and the failed test output, then tries again. Up to PATCHBAY_MAX_ATTEMPTS times (default 3). Optional parallel mode generates N variants at once, verifies them in parallel sandbox forks, keeps the first green.

Every endpoint in the spec comes with an example payload taken from the documentation; the generated test harness runs the connector against that payload (the "contract" check), so a run only counts as green when the generated code actually works with the shape the real API returns.

Dual-language, first-class

TypeScript and Python share one plan + research step — only the generate prompt, the sandbox image and the test harness swap. The hosted demo at patchbay-mcp.dev has a TypeScript · Python · Both toggle; "Both" fires two live Nebius pipelines side-by-side so you watch the same prompt land in both languages at once.

The Python target ships fixtures.py with the same shape as src/fixtures.ts: SOURCE_EXAMPLE, TARGET_ENDPOINTS, contract_fetch(), fake_fetch(), json_response(). Connectors export async def handle(payload, config) with an injected async transport in config['fetch'], so tests stay hermetic (no network, no real httpx client). URL matcher handles placeholder segments ({id}, :id, <id>) and region subdomains. See bench/RESULTS-variance.md for the per-prompt breakdown.

Carve-outs: PATCHBAY_RUNNER=local only runs TS. The hosted site's recorded replays are TS-only recordings, so without live keys the Python toggle shows the TS replay with a note.

Audit MCP servers and connectors

patchbay audit <url-or-directory> scans an MCP server or a connector repo for OAuth 2.1 and secret-hygiene gaps before you ship it. ~40% of open-source MCP servers have no auth at all, only ~8.5% use OAuth. patchbay audit is the drop-in check.

npx -y @patchbay-nebius/agent audit https://mcp.example.com/mcp
npx -y @patchbay-nebius/agent audit ./my-mcp-server --json
npx -y @patchbay-nebius/agent audit ./my-mcp-server --sarif > results.sarif

21 live checks + 1 retired:

ID	Severity	What it catches
AUTH-001	critical	tools/list succeeds without any credentials (downgraded to info for public-by-design or platform-IAM hosts)
AUTH-002	critical	Static API key or bearer accepted, no /.well-known/oauth-* discovery served
AUTH-003	high	OAuth metadata without PKCE S256 in code_challenge_methods_supported. Per MCP 2026-07-28, MCP clients MUST refuse to proceed in this case.
AUTH-004	high retired	Checked resource_indicators_supported metadata flag. Retired in 0.3.4 after verification that no RFC or MCP spec defines this field. Use AUDIENCE-001 (opt-in probe) for actual audience validation.
AUTH-005	high / info	High when neither CIMD nor DCR advertised. Info when only DCR is advertised (deprecated per MCP 2026-07-28).
AUTH-006	high	Authorization server rejects loopback redirect URIs (RFC 8252 §7.3). Opt-in via --probe-dcr-loopback.
AUTH-007	medium	Invalid bearer token returns a status other than HTTP 401 (MCP 2026-07-28 mandates 401).
AUDIENCE-001	critical	Server accepts bearer tokens minted for a sibling resource on the same AS (RFC 8707). Opt-in via --probe-audience.
AUDIENCE-002	critical	Server accepts a bearer token whose JWT payload has no aud claim at all (catches the if (aud && aud !== me) reject early-exit bug). Opt-in via --probe-missing-aud.
RESOURCE-001	high	Missing or malformed Protected Resource Metadata (RFC 9728). MCP MUST since 2025-06-18.
SCOPE-001	high	All tools share one scope or no scopes declared at all
SCOPE-002	medium	Write/destructive tools share scope with read tools
SECRET-001	critical	Hardcoded sk-…, ghp_…, AKIA…, xoxb-… or hex secrets in source
SECRET-002	medium	Credential read from env but no refresh-token flow found
TRANSPORT-001	high	Remote server reachable over plain HTTP
TRANSPORT-002	medium	Server accepts a spoofed Origin header (DNS rebinding)
REDIRECT-001	medium	Token endpoint treats localhost and 127.0.0.1 inconsistently
HEADERS-001	medium	Weak transport-security response headers (wildcard CORS with credentials, missing HSTS on HTTPS, missing frame-options, missing nosniff).
POISONING-001	high / medium	High for hidden Unicode in tool descriptions (zero-width, BOM, bidi, tag chars). Medium for imperative role-marker patterns (<SYSTEM>, ignore-previous directives, role prefixes).
ISS-001	low	OAuth metadata does not advertise authorization_response_iss_parameter_supported: true (RFC 9207, SHOULD per MCP 2026-07-28).
MANIFEST-001	varies	Tool manifest changed since last pinned scan (--trust to re-pin)
MANIFEST-002	varies	Classifies the manifest delta into breaking (removed tool, removed param, type narrowed, enum shrunk, made-required), semantic-drift (description changed), and additive buckets. One finding per non-empty bucket.
PROBE-001	info	Probe inconclusive, cannot judge auth posture

See patchbay-phase2-plan.md for the roadmap (--with-auth connector templates, --pr-draft, GitHub Action).

Outputs: pretty terminal (default, with CTA), --json, --sarif for GitHub Actions. --ci exits non-zero on critical/high so the audit gates your pipeline. --no-cta keeps CI logs clean.

Pure-core design. packages/audit-core has zero I/O inside the check functions: probing collects a TargetSnapshot, checks are (snapshot) => Finding[]. Trivial to unit-test, trivial for the repair model to consume later.

Training our own repair model on our own benchmark runs

Every patchbay benchmark run is a record of the agent failing and then fixing itself. 173 of those (red draft + failing test report) → green draft moments across the committed benchmark passes make a domain-matched training set — and Nebius Token Factory ships an OpenAI-compatible fine-tune endpoint that eats exactly this format. So we fine-tuned Qwen/Qwen3-30B-A3B-Instruct-2507 on patchbay's own repair history. The deployment path uses a second Nebius product: trained checkpoints come out via /fine_tuning/jobs/{id}/checkpoints, land on a Nebius AI Cloud GPU (H100 NVLink preemptible) behind a vLLM OpenAI-compatible endpoint, and PATCHBAY_REPAIR_LLM_BASE_URL points patchbay's repair step at the self-hosted model. Three Nebius products in one closed loop: Token Factory Inference (live agent) → Token Factory Post-training (fine-tune) → AI Cloud GPU (serve the fine-tune).

• bench/training-{train,eval}.jsonl — 98 train + 11 eval real red→green repair pairs, extracted from committed benchmark runs. Also mirrored on HuggingFace as patchbay-nebius/connector-repair-dataset with a train/eval split.
• scripts/finetune.ts — upload / start / watch Nebius fine-tune jobs (takes --context=32768 to fit our 20k-token repair pairs).
• scripts/finetune-checkpoint.ts — poll a job to completion, fetch the checkpoint URL, print the exact AI Cloud deploy next steps.
• scripts/vllm-deploy/ — Dockerfile + run.sh to serve the fine-tuned weights on an H100 / H200.
• scripts/bench-finetuned-ablation.ts — side-by-side fine-tuned vs Ultra-baseline over the TS Core 20.
• scripts/eval-repair.ts — format, file-coverage and line-overlap metrics against the held-out eval split.
• Full plan + status (updated as the job progresses): docs/finetune.md.

Evidence export

Every live build offers a one-click Evidence .zip download on the site. Trust but verify — the whole run is auditable without re-executing anything:

connector/          final files (ready to drop into a repo)
trajectory/
  attempt-1/
    <files>
    report.json     the test report for that attempt
  attempt-2/ …      every repair attempt the agent made
prompt.txt          the one-sentence prompt
plan.json           source app, target app, field mapping
spec.json           researched API spec (endpoints, auth, example payloads)
usage.json          per-call model usage + cost + total
manifest.json       issuedAt, finalResult, sandbox operation IDs, SHA-256 of every file, verify instructions
README.md           what each file is + how to recompute hashes

The sandboxOperationIds in manifest.json are Nebius Token Factory operation IDs — your project's operation history retains the exact command patchbay ran, its exit code, and its stdout for each one. Recompute hashes locally with shasum -a 256; mismatch means somebody edited the artifact after export.

Three ways to call it

CLI – @patchbay-nebius/agent on npm

npx -y @patchbay-nebius/agent build "When a Stripe payment succeeds, add the customer to a Mailchimp audience"
# or
npm install -g @patchbay-nebius/agent
patchbay build   [--record] [--lang=python] "<one-sentence integration>"
patchbay research [--lang=python]            "<one-sentence integration>"
patchbay verify  [--lang=python] <directory>
patchbay scan    <directory-or-github-url> [--build-top=<n>] [--lang=python]
patchbay audit   <url-or-directory> [--json] [--sarif] [--ci] [--no-cta]

patchbay scan <dir-or-github-url> — autonomous repo scanner. Walks any repository (local directory OR a GitHub URL, which patchbay shallow-clones for you) and reports the integration opportunities patchbay could build: TypeScript Express/Next/Fastify webhook handlers that only contain TODO or throw not implemented, Python FastAPI/Flask routes with pass bodies or raise NotImplementedError, .env.example entries pointing at vendor APIs with no implementation, TODO/FIXME comments mentioning Stripe/Mailchimp/Shopify/… and vendor SDKs that are imported but barely used. Each opportunity ships with a ready-to-run prompt and a priority score; --build-top=N runs the full pipeline on the top N automatically.

npx -y @patchbay-nebius/agent scan https://github.com/<some>/<repo>

MCP server – @patchbay-nebius/mcp on npm, stdio transport.

Four tools — scan_repo (any local path or GitHub URL, returns ranked opportunities with ready-to-run prompts; TS + Python aware), build_connector (full pipeline, lang: "ts" | "py"), research_api (plan + Tavily spec, no code), verify_code (sandbox verify + repair existing files, auto-detects language). Live progress + logging notifications.

claude mcp add patchbay -- npx -y @patchbay-nebius/mcp

Full setup for Claude Code, Codex and Cursor in docs/mcp-setup.md.

Web app – apps/web, Next.js 14 on port 3400. Live demo at https://patchbay-mcp.dev. The same UI streams a live Nebius run or, past the 6/hour per-IP rate limit, the recorded replay. Shareable scan URLs (?scan=owner/name) and shareable run URLs. Public uptime at https://patchbay.openstatus.dev.

Slack – /patchbay <one-sentence integration> as a slash command. Setup in docs/slack-setup.md; the endpoint lives at apps/web/app/api/slack/command/route.ts on the same Vercel deploy.

GitHub PR – comment /patchbay build "<integration>" on any PR and the workflow in .github/workflows/patchbay-pr.yml runs the full pipeline, attaches the generated connector as an artifact and replies with the test score. Needs the three Nebius / Tavily secrets configured in the repo.

Why not just ask Claude Code / Cursor / Codex yourself?

Same connector, same requirements. Rough per-connector cost:

Where	Cost	Factor
patchbay (Nemotron on Nebius)	~$0.07	1×
Claude Code with Haiku 5.5	~$0.10	1.4×
Claude Code with Sonnet 5.5	~$0.35	5×
Claude Code with Opus 4.7	~$1.90	27×

$0.07 vs ~$1.90 for the same connector. The tests run before you see the file.

The agent in your IDE is a generalist. For integration connectors — read the docs, map fields, write the thing, prove it works — that's work with hard structure, and a specialized pipeline on cheap models beats a general-purpose agent on expensive ones. Same reason you don't ask Opus to format JSON.

patchbay sits inside your editor as an MCP server (claude mcp add patchbay -- npx -y @patchbay-nebius/mcp), so you keep using Claude Code / Cursor / Codex for everything else — architecture, refactors, bug hunts — and offload connector work to patchbay. Your Claude usage bill goes down for the task Claude is slowest and most expensive at.

The generated connector is plain TypeScript — or Python 3.12 — in your repo, no runtime dependency on patchbay. Read it, change it, ship it.

Selfhost

git clone https://github.com/GodSmasher/patchbay
cd patchbay
cp .env.example .env.local            # fill in NEBIUS + TAVILY keys
docker compose up --build
# open http://localhost:3400

The Dockerfile is a three-stage build (deps → build → runtime) producing a ~90 MB image around Next.js's standalone server. docker-compose.yml wires the env vars through from .env.local; without live keys the web app streams the recorded replay instead.

Build in public

What broke this week and how it got fixed is in docs/build-log.md. Short version: nested-contract regex (+2 benchmark points), parallel-repair fiasco (reverted), monotonic repair (never regress past the best attempt), timestamp prompt (don't reject the documented example), esbuild preflight (first-try rate 33% → 52%), Vercel monorepo pain (three fixes, one deploy), URL FIDELITY + idempotent-upsert worked examples for Python (closed 2 of the hardest TS→Python green-rate gaps).

Dev

npm install
cp .env.example .env.local            # fill in NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY

npm test                              # vitest, replay mode, no keys needed
npm run typecheck                     # tsc across workspaces
npm run check:nebius                  # auth probe for inference + sandboxes + Tavily
npm run patchbay -- build "..."       # live run (needs keys)
npm run patchbay -- help              # all commands and flags
npm run bench                         # the 20-prompt benchmark
npm run dev                           # Next.js app on :3400
npm run mcp                           # start the MCP server on stdio

Environment

See .env.example. Key knobs:

Variable	Default	What it does
PATCHBAY_MOCK	true	Replay mode. Set to false for live runs.
PATCHBAY_LANG	ts	ts (TypeScript+vitest) or python (Python 3.12+pytest).
PATCHBAY_RUNNER	nebius	nebius (isolated sandbox) or local (dev fallback, TS only).
PATCHBAY_MAX_ATTEMPTS	3	Repair-loop cap.
PATCHBAY_PARALLEL_REPAIRS	1	N parallel repair branches per attempt (max 4). Best-of-N.
PATCHBAY_MODEL_FAST / MID / STRONG	nvidia/Nemotron 3.5 Lightning / 3 Super / 3 Ultra	Model tiers. Any OpenAI-compatible base URL works.
PATCHBAY_LLM_BASE_URL / PATCHBAY_LLM_PROVIDER	(unset) / nebius	Point patchbay at any OpenAI-compatible endpoint — including a self-hosted vLLM serving our fine-tuned Qwen3-30B.

The generated connector has zero patchbay dependency – it's a plain TypeScript or Python file your project can drop in anywhere.

Repo layout

packages/
  agent/        # the pipeline (plan, research, generate, verify, repair) + CLI + scanner
  mcp/          # MCP server exposing scan_repo / build_connector / research_api / verify_code
  audit-core/   # OAuth 2.1 / scope / secret checks for MCP servers (patchbay audit)
apps/
  web/     # Next.js demo
bench/     # the 20-prompt benchmark + results + training data
docs/      # setup guides + build log + finetune plan
scripts/   # finetune launcher, checkpoint fetch, vllm-deploy, ablation bench
runs/      # recorded live runs (gitignored)

Deploy

The web demo is a Next.js app with one SSE route (/api/run). An in-memory IP rate limit caps live runs at PATCHBAY_MAX_RUNS_PER_HOUR (default 6) per hour and silently falls back to the recorded replay when exceeded, so a hug-of-death doesn't burn through credits. For Vercel: apps/web/vercel.json pins the route's function to a 300s timeout (needs a Pro plan; on Hobby, add ?replay=1 or set PATCHBAY_MOCK=true to keep every call under 10s).

---

Built for the Nebius × NVIDIA Global AI Hackathon. Track: Coding & Agentic Engineering.