Continuous AI Prompt & Agent Regression Testing Workflow

AI-generated GitHub Actions example for running project-specific evaluation commands and considering thresholds. The underlying dataset and evaluation service are not supplied.

Tags: workflow, evals, ci-cd, github-actions, llm-judge, regression, prompt-testing

Category: devops

Listing attribution: SimplyUtils

Before using this resource

Attribution and license labels come from the listing metadata. Check the source and its license before redistributing or adapting files. This listing has no documented license; do not assume permission to redistribute it.

Read instructions before adding them to an agent or project. Test a small, non-confidential example and inspect the result. A downloaded instruction file may refer to scripts, templates, or other files that are not included here.

SimplyUtils supplied this template, which was generated by an AI agent using an LLM. It is an example to inspect and adapt, not a verified package or a record of expert testing. Check named dependencies, commands, permissions and generated results before use. Redistribution terms for this template have not been verified; licenses for third-party software are separate.

Source

Repository

Continuous AI Prompt & Agent Regression Testing Workflow

Overview

This CI/CD workflow automates regression testing for AI prompts, agent tools, and RAG pipelines on every GitHub Pull Request.

GitHub Actions Workflow (.github/workflows/ai-evals.yml)

name: AI Prompt & Agent Evals
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'client/src/data/ai-resources.json'
      - 'server/services/ai/**'

jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Install dependencies
        run: npm ci
      - name: Execute LLM-as-Judge Evals
        run: npm run test:evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          EVAL_THRESHOLD: "0.85"

Evaluation Threshold Enforcements

  • Pass Rate Gate: Minimum 85% aggregate score required to merge PR.
  • Zero Hallucination Tolerance: Critical safety or SQL injection fails automatically trigger build failure.