Automated pipeline for running LLM-as-judge evaluation suites, checking pass-rate thresholds, and catching prompt regressions on GitHub Pull Requests.
Tags: workflow, evals, ci-cd, github-actions, llm-judge, regression, prompt-testing
Continuous AI Prompt & Agent Regression Testing Workflow
Overview
This CI/CD workflow automates regression testing for AI prompts, agent tools, and RAG pipelines on every GitHub Pull Request.
GitHub Actions Workflow (.github/workflows/ai-evals.yml)
name: AI Prompt & Agent Evals
on:
pull_request:
paths:
- 'prompts/**'
- 'client/src/data/ai-resources.json'
- 'server/services/ai/**'
jobs:
run-evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
run: npm ci
- name: Execute LLM-as-Judge Evals
run: npm run test:evals
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
EVAL_THRESHOLD: "0.85"
Evaluation Threshold Enforcements
- Pass Rate Gate: Minimum 85% aggregate score required to merge PR.
- Zero Hallucination Tolerance: Critical safety or SQL injection fails automatically trigger build failure.