TMHSDigital/plumbline

Issue templates: a nonexistent label, and nowhere to report a site bug about 1 hour ago

docs good first issue

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

CLI help and small doc inconsistencies about 1 hour ago

docs good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

.env.example and the docs promise more than the adapters do about 1 hour ago

docs good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

PLAN.md lists work that is already done about 1 hour ago

docs good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

The load note says every row is asked as a choice, which is not true for typesafe_wire about 1 hour ago

bug docs good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

Small site bugs: misreported example errors, nBins of 0, parity precision about 1 hour ago

bug good first issue site

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

CI's traceback guard can never fail about 1 hour ago

bug good first issue ci

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

load_jevbench skips the duplicate-id check about 1 hour ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

CLI options accept values that silently misbehave or crash with a traceback about 1 hour ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

per_correct_usd divides the priced-row cost by the correct rows across all rows about 1 hour ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

The cascade never considers escalating every case about 1 hour ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

The multiclass Brier floor raises IndexError when a distribution sums to slightly under 1 about 1 hour ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

The README's hosted-vendor command is refused before it sends anything about 1 hour ago

bug docs good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

Report surfaces a shared failure reason instead of filing it in the artifact 1 day ago

bug good first issue

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

Add an adapter config for a second Jev-compatible endpoint 1 day ago

docs good first issue adapter

TMHSDigital/plumbline

0

Measure whether a decision model's probabilities hold up on your own labeled data. Not a leaderboard.

Python

#benchmarking#calibration#classification#cli#confidence-calibration#evaluation#expected-calibration-error#llm#machine-learning#model-evaluation#python#uncertainty-quantification

Open Issues Need Help