Copied to clipboard!
ReadMe Realist CI Gatekeeper
GitHub App + LLM Guardrails

Never Let Code Changes
Outpace Your Documentation

ReadMe Realist automatically detects structural code deltas (CLI flags, environment variables, dependencies, routes) during Pull Requests and alerts developers before outdated docs break production.

100%
Automated CI Gatekeeping
< 200ms
Early-Exit Skip Engine
Idempotent
Single Threaded Comments
Fail-Open
Zero Pipeline Friction

Core Value Proposition

The Documentation Drift Dilemma

The Problem: Documentation Drift

Software evolves rapidly during Pull Requests. Developers add environment variables, update CLI parameters, or change API routes—but updating documentation is easily forgotten.

  • New REDIS_URL env variable added to code, but absent in README.
  • Broken developer onboarding & misconfigured staging deployments.
  • Manual PR code reviews miss subtle documentation omissions.
  • Traditional CI linters test code syntax, not documentation truth.

The Solution: ReadMe Realist

An intelligent GitHub App that intercepts PR webhooks, parses structural diffs, extracts key signals, and evaluates changes against repository markdown using LLMs.

  • Structural Signal Extractor: Isolates CLI args, env vars, and routes.
  • LLM Semantic Evaluator: Gemini/Anthropic compares diffs against docs.
  • Automated PR Patches: Suggests exact markdown edits directly in PR comments.
  • Skip Engine: Ignores formatting & doc-only changes to minimize API cost.

Visual Blueprint

End-to-End System Dataflow

Mermaid.js Rendered Live Architecture Map
flowchart TD %% Styling Classes classDef client fill:#1e293b,stroke:#334155,stroke-width:2px,color:#fff; classDef gateway fill:#0369a1,stroke:#0284c7,stroke-width:2px,color:#fff; classDef worker fill:#6d28d9,stroke:#7c3aed,stroke-width:2px,color:#fff; classDef service fill:#c2410c,stroke:#ea580c,stroke-width:2px,color:#fff; classDef external fill:#15803d,stroke:#16a34a,stroke-width:2px,color:#fff; classDef error fill:#991b1b,stroke:#dc2626,stroke-width:2px,color:#fff; subgraph GitHub_Event ["GitHub Platform & Trigger"] A([Developer opens / updates PR]) -->|Webhook Event Delivery| B[GitHub Event Dispatcher] end subgraph API_Gateway ["Gateway & Signature Verification"] B -->|POST /webhooks/github + X-Hub-Signature-256| C[Webhook Route Handler] C --> D{Verify HMAC Signature} D -->|Invalid Signature| E[HTTP 401 Unauthorized] D -->|Ping Event| F[HTTP 200 Pong] D -->|Non-PR Event| G[HTTP 200 Ignored] D -->|Valid PR Event| H[Parse Pull Request Payload] H -->|Draft / Unsupported Action| I[HTTP 200 Ignored] H -->|Valid PR Context| J[Background Worker Queue] J -->|HTTP 202 Accepted| B end subgraph Pipeline_Processing ["Review Pipeline Orchestration"] J -->|Dispatch Async Task| K[ReviewPipeline Orchestrator] K --> L[Start Check Run in_progress] L --> M[Fetch PR Unified Diff] M --> N[Code Delta Parser & Signal Extractor] N --> O{Skip Decision Engine} O -->|No Files / Noise-only / Docs-only| P[Publish Skipped Status] O -->|Substantive Code Changes| Q[Fetch Documentation Files] Q --> R{Docs Exist?} R -->|No Matching Docs| S[Publish Skipped / Neutral Check] R -->|Docs Retrieved| T[LLM Semantic Evaluator] end subgraph LLM_Service ["LLM Evaluation Layer"] T -->|Prompt: Diff + Signals + Docs| U[Google Gemini / LLM Backend] U -->|Structured Verdict JSON| V{Drift Detected?} end subgraph Feedback_Publishing ["Feedback Orchestrator & Persistence"] V -->|UP_TO_DATE| W[Mark Check Run Success & Upsert Resolved Comment] V -->|NEEDS_UPDATE| X[Post/Update PR Comment with Suggested Edits & Check Run] P --> Y[Finalize Check Run completed] S --> Y W --> Y X --> Y E --> Z([End Lifecycle]) F --> Z G --> Z I --> Z Y --> Z end class A,B client; class C,D,H,J gateway; class K,L,M,N,O,Q,R,T worker; class U service; class W,X,P,S,Y external; class E,F,G,I error;

Interactive Playground

Live PR Drift Evaluator Simulator

Simulate incoming code deltas and observe how ReadMe Realist parses signals and creates GitHub feedback.

Select Pre-configured PR Scenario:
Simulated Payload
Existing Content
Pipeline Step Inspector Ready
// System ready. Click 'Evaluate PR' to simulate webhook execution.
GitHub PR Feedback Comment Preview Idempotent Upsert Marker
No evaluation run yet. Select a scenario and launch the simulator above.

Step-by-Step Contract

Data Transformation Lifecycle

Detailed breakdown of the 9 execution stages from GitHub webhook triggers to PR comment upserts.

Step ID Source Node Target Node Data Contract / Payload Transformation & Logic
01 GitHub PR Event Webhook Receiver JSON + X-Hub-Signature-256 Delivers webhook delivery with HMAC-SHA256 signature headers.
02 Webhook Receiver Signature Validator Raw Bytes + Secret Key Validates cryptographic signature before parsing payload to prevent tampering.
03 Webhook Receiver Background Worker PullRequestContext Object Sanitizes payload into domain model and enqueues review task asynchronously.
04 Pipeline Orchestrator GitHub REST API Authenticated JWT Creates initial GitHub Check Run in in_progress state and downloads PR diff.
05 Diff Downloader Code Delta Parser Unified Diff String Filters out binary/lockfile noise and extracts structural signals (CLI, env, deps, routes).
06 Delta Parser Doc Fetcher / GitHub File Globs (README.md, docs/**) Scans repo for target documentation files matching configured glob patterns.
07 Pipeline Orchestrator LLM Evaluator DiffAnalysis + DocBundle Formats context and prompt instructions requesting structured DriftVerdict JSON.
08 LLM Backend Feedback Orchestrator DriftVerdict JSON Parses model response into verdict structure with proposed markdown patches.
09 Feedback Orchestrator GitHub PR & Checks Markdown Comment + Check Run Upserts single persistent PR review comment (via marker tag) and marks check completion.

Production Engineering

Guardrails & Failure Resilience

Early-Exit Skip Engine

PRs with no code changes, formatting/whitespace-only updates, or doc-only edits immediately skip LLM inference, reducing evaluation latency and saving API token costs.

Fail-Open Safeguards

If external LLM services experience rate limits or network timeout errors, the pipeline gracefully posts a neutral check run without blocking developer PR merges.

Idempotent PR Comments

Feedback comments are tagged with an invisible HTML marker (<!-- readme-realist:v1 -->) and upserted in-place to prevent comment spamming across repeated pushes.

Quick Start & Installation

Deployment & Configuration Guide

Follow these steps to deploy ReadMe Realist locally, via Docker, or directly as a GitHub App.

Step 1: Configure a GitHub App

  1. Navigate to GitHub Settings > Developer settings > GitHub Apps > New GitHub App.
  2. Set Webhook URL to your endpoint (e.g. https://your-domain.com/webhooks/github or Smee URL).
  3. Generate a secure Webhook Secret:
    python -c "import secrets; print(secrets.token_hex(32))"
  4. Grant Repository Permissions:
    • Checks: Read & Write (for check runs)
    • Contents: Read-only (for diffs & docs)
    • Pull Requests: Read & Write (for review comments)
  5. Subscribe to event: Pull request.
  6. Generate a Private Key (.pem) and save to ./secrets/github-app.private-key.pem.