cw-hans/agency-agents
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add promptfoo eval harness for agent quality scoring (#371)

Browse Source 
Adds promptfoo eval harness for agent quality scoring. LLM-as-judge system scoring task completion, instruction adherence, identity consistency, deliverable quality, and safety. Includes tests.
This commit is contained in:
Russell Jones
2026-04-10 22:54:31 -04:00
committed by GitHub
parent 1e73b5be0d
commit b456845e85
11 changed files with 796 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
evals/package.json Normal file
View File

@@ -0,0 +1,24 @@
{
"name": "agency-agents-evals",
"version": "0.1.0",
"private": true,
"description": "Evaluation harness for agency-agents specialist prompts",
"scripts": {
"eval": "promptfoo eval",
"eval:view": "promptfoo view",
"eval:cache-clear": "promptfoo cache clear",
"extract": "ts-node scripts/extract-metrics.ts",
"test": "vitest run",
"test:watch": "vitest"
},
"dependencies": {
"gray-matter": "^4.0.3",
"promptfoo": "^0.121.3"
},
"devDependencies": {
"@types/node": "^22.0.0",
"ts-node": "^10.9.0",
"typescript": "^5.7.0",
"vitest": "^3.0.0"
}
}