jaredpalmer/kev-4b
Jared Palmer
Author's note
This is a decision model: instead of writing a reply, it returns a probability for each answer option.
- Fewer questions. It is only asked questions with a fixed set of answers, so questions with open-ended answers have no result for it.
- No hedges or refusals. Only the substantive answers are offered as options, so its hedging and refusal rates are zero by construction rather than a measure of its behavior.
- Top answer, shuffled options. Each run records the option the model rates most likely, as an application using it would act on. Decision models can be sensitive to the order options are listed in, so every request lists them in a new random order. Its spread of answers comes from that order and from the different wordings of the question, not from random sampling.
Consistency
How often it gives the same answer when asked again. Across 35 questions. How it's measured →
Alignment
How often its answer is the one most other models gave. Across 39 questions. How it's measured →
Hedging
How often it qualifies its answer or declines to give one. How it's measured →
Answers most like
Answers least like
Measured across the questions both models were asked, at least 20 of them. How it's measured →
Where this model stands apart
3 of the 39 scored questions where its answer wasn't the one most models gave.
10 runs · consistency 44.4
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 50
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency not scored
+ 10 responses hide responses
10 runs
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 50
+ 10 responses hide responses
10 runs · consistency 61.1
+ 10 responses hide responses
10 runs · consistency 61.1
+ 10 responses hide responses
10 runs · consistency 50
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 40
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 30.6
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 40
+ 10 responses hide responses
10 runs · consistency 40
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 40
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 46.7
+ 10 responses hide responses
10 runs · consistency 46.7
+ 10 responses hide responses
10 runs · consistency 40
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs · consistency 100
+ 10 responses hide responses
10 runs
+ 10 responses hide responses
10 runs