MAHMOUD ABDELNABY
Career · CLI SCRIPT · 2

Marketing Prompt Regression Testing Suite across LLM Version Upgrades

Stop prompt drift after LLM API updates. Automate regression testing, JSON schema validation, and tone checks for marketing prompts.

Problem it addresses

Build an open-source prompt regression testing framework for marketing teams (prompt_eval_runner.py): 1) Evaluates prompt outputs against strict assertions (Valid JSON schema, Max word count, Presence of required CTA, Absence of forbidden AI clichés like 'delve' or 'game-changer'), 2) Calculates semantic tone similarity against golden reference answers, 3) Terminal pass/fail test reports suitable for CI/CD.

Who it is for

PRO operating in high-growth e-commerce, digital publishing, and direct-response marketing across Saudi Arabia, UAE, and Egypt.

What is inside

Key files listed in the README repository architecture:

  • eval_suites/ad_copy_json_suite.json
  • eval_suites/seo_meta_suite.json
  • LICENSE
  • README.md
  • reports/sample_eval_report.html
  • src/prompt_eval_runner.py
  • tests/test_runner.py

How to evaluate and use it

  1. Read the README first and confirm that the documented problem matches your use case.
  2. Inspect the listed files to find the data, tools, templates, and tests relevant to your task.
  3. Follow the repository’s Quick Start and check prerequisites before running commands.
  4. Compare outputs with documented examples and review limitations before using them in a decision or client workflow.

Quick Start from the README: python -m unittest discover -s tests · python src/prompt_eval_runner.py --help

Scope and limitations

  • Deterministic Evaluation Fixtures: Runs assertions against LLM outputs across version upgrades to detect regression in tone, dialect, or factual accuracy.
  • Multi-Model Parity: Evaluates prompt performance across GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro to identify breaking prompt behaviors.
  • Hallucination Score Bounds: Quantifies factual consistency and flags forbidden promotional superlative claims.

This page summarizes the available documentation; it does not establish performance or business results. Verify data and prerequisites before use.

Related repositories