AgentPostmortem/Evalgate

New scorer refusal: refusal detector for safety evals 8 days ago

help wanted good first issue

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer tool-call: tool-call format for agentic evals 8 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer word-count: output length bounds 8 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

New scorer json-nonempty: fail schema-valid-but-empty outputs 8 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

String-form tags silently dropped, --tags skips the case with no warning 11 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

llm-judge passes on unclamped scores, a 0-100 reply always passes 11 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Rubric criterion with invalid pattern regex throws and kills the whole run 11 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

compareRuns accepts negative/NaN tolerance 14 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Scorer weight: Infinity passes validation and poisons aggregation 14 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

contains scorer with empty-string needle always passes 14 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Suite validation accepts empty prompts and bogus message roles 14 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Invalid numeric CLI flags silently fall back instead of erroring 14 days ago

help wanted good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

temperature and maxTokens are plumbed through the provider but cannot be set by a suite 27 days ago

enhancement good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

An invalid pattern in a JSON schema throws and kills the whole run 27 days ago

bug good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

exact-match always collapses internal whitespace, and trim: false does not turn it off about 1 month ago

good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Docs: --concurrency, --junit and --degrade are missing from evalgate --help about 1 month ago

documentation good first issue

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

evalgate list hardcodes the provider names instead of asking the registry about 1 month ago

enhancement good first issue claimed

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Cost is one hardcoded rate per provider, applied to every model and both token directions about 1 month ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Provider calls have no timeout and no retry, so a 429 fails the build about 1 month ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

One provider or scorer error discards the entire run about 1 month ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Deleting a failing case makes the gate green: removed cases are never regressions about 1 month ago

bug enhancement help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Bug: a run with zero cases reports passed and exits 0, so the gate silently disables itself about 1 month ago

bug help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

--tags and --concurrency are ignored by baseline and compare about 1 month ago

bug help wanted

AgentPostmortem/Evalgate

0

Prompt and agent regression CI. The build fails when your prompt gets dumber. GitHub Action with PR delta comments.

TypeScript

#ai#ci#evals#github-action#llm#prompt-engineering#typescript

Open Issues Need Help