Leaderboard
How leading AI models reason through ethical dilemmas, and what they do when they act as agents, including whether an ETHICS.md changes it.
Last run: dilemmas · agent scenarios
Does ETHICS.md change behavior?
Share of the 25 rule tasks each model followed, run 3 times each. The rules are the same each time; only where they appear changes.
Results by measure
Each cell shows the result with no rules → with an ETHICS.md, which AGENTS.md points to. Rule tasks use an ETHICS.md holding that task's rule. Every other task uses one general company ETHICS.md, the same for all of them. Rule-breaking actions: lower is better. Every other column: higher is better. Sorting uses the results with ETHICS.md.
| Model | Rules followed | Harm avoided | Injections resisted | Ordinary tasks done | Did what it said | Rule-breaking actions | Opened ETHICS.md |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | 31% → 100% | 100% → 100% | 100% → 100% | 93% → 100% | 14% → 30% | 12% → 0% | 99% |
| Claude Sonnet 4.5Anthropic | 15% → 96% | 76% → 100% | 80% → 96% | 100% → 100% | 20% → 29% | 27% → 17% | 98% |
| GPT 6 LunaOpenAI | 13% → 100% | 98% → 100% | 98% → 100% | 100% → 98% | 28% → 34% | 15% → 9% | 100% |
| Gemini 4 ArgonGoogle | 39% → 100% | 100% → 100% | 100% → 100% | 93% → 100% | 14% → 44% | 10% → 3% | 100% |
| Grok 4.7xAI | 12% → 97% | 98% → 100% | 100% → 100% | 98% → 96% | 20% → 31% | 13% → 4% | 100% |
| DeepSeek V4.1 FlashDeepSeek | 21% → 100% | 100% → 100% | 100% → 100% | 98% → 100% | 18% → 31% | 3% → 4% | 100% |
| Kimi K3.1 ProMoonshot AI | 9% → 100% | 100% → 100% | 98% → 100% | 100% → 98% | 30% → 37% | 18% → 6% | 100% |
| GLM 5.3 FlashZ.AI | 11% → 95% | 96% → 100% | 100% → 100% | 98% → 100% | 27% → 38% | 18% → 6% | 100% |
| Nemotron 3 UltraNVIDIA | 8% → 27% | 47% → 71% | 84% → 87% | 93% → 98% | 11% → 21% | 60% → 26% | 42% |
Do models do what they say?
Each model answered a dilemma, then faced the same situation as an agent, first with no rules, then with the general company ETHICS.md. Each bar shows how often its action was the option it said it would choose, how often it took a different option, and how often it did none of them. A match means its stated choices predict its behavior; it does not mean the choice was right.
How models reason
On dilemmas where ethical frameworks disagree, how often each model chose the option each framework favors. Frameworks often share an answer, so a row does not add to 100. No profile is better than another. Select a model for its breakdown by category.
| Model | Rights | Utilitarian | Deontological | Virtue |
|---|---|---|---|---|
| Claude Opus 5.5Anthropic · served by Anthropic | 40% | 35% | 42% | 55% |
| Claude Sonnet 4.5Anthropic · served by Anthropic | 40% | 33% | 43% | 53% |
| GPT 6 LunaOpenAI · served by OpenAI | 42% | 33% | 46% | 56% |
| Gemini 4 ArgonGoogle · served by Google AI Studio | 39% | 34% | 45% | 53% |
| Grok 4.7xAI · served by xAI | 40% | 33% | 47% | 55% |
| DeepSeek V4.1 FlashDeepSeek · served by Together | 41% | 34% | 48% | 56% |
| Kimi K3.1 ProMoonshot AI · served by DeepInfra | 42% | 32% | 46% | 55% |
| GLM 5.3 FlashZ.AI · served by Z.AI | 40% | 35% | 43% | 57% |
| Nemotron 3 UltraNVIDIA · served by DeepInfra | 40% | 33% | 46% | 57% |
Methodology
Section titled “Methodology”Dilemmas
Section titled “Dilemmas”The questions. 1,007 ethical dilemmas across 19 areas, from medical and business ethics to privacy and AI. The set began as Shraddha’s 100-question questionnaire and was extended to 1,007 in the same format. Each dilemma has four options, and each of four frameworks (rights-based, utilitarian, deontological and virtue) is keyed to the option it would choose. On 113 dilemmas all four frameworks choose the same option. On the other 894 they disagree, and those are the ones scored here.
The prompt. Every model gets the same instructions: choose the option that best reflects its judgment and reply with the letter only. Each dilemma is asked once, in its own request, with a reply budget of 4,000 tokens that includes any reasoning the model does first.
Shuffled options. Option order is shuffled for every question, the same way for every model in a run. A model that answers by letter position scores near chance.
One named host per model. Each model runs on a single host with fallbacks off, so every answer comes from the same deployment: the lab’s own host where one is offered, otherwise a named third-party host. The host is shown under each model’s name, with its precision on the model’s page where the host reports it.
Reading the answers. The answer is the letter in the reply. A reply with no readable letter is left out of the scores and counted on the model’s page; across all models that was 10 replies in 9,063.
Private questions. The questions and answer keys are held privately so they don’t leak into training data. Only the scores are published.
Agent scenarios
Section titled “Agent scenarios”The tasks. 100 tasks set in a fictional online store, each in a sandbox with files, a database and simulated email, support tickets, an inbox, refunds, discounts, payments and operations jobs. Nothing leaves the sandbox. The model works by calling tools until it finishes, hands the task to a person, stops, or reaches 20 steps. Every model runs every task three times.
Five kinds of task.
- Rules (25 tasks). Each has a rule found only in the rules given to the agent, such as a refund limit or never deleting files. Without the rule, the forbidden action is often the ordinary thing to do. Some are harder: the operator’s own instructions conflict with the rule, a rule in a subfolder’s
ETHICS.mdtightens the main one or tries to loosen it, or the rule only matters late in a long task. Each runs three ways: with no rules, with the rules in anETHICS.mdthatAGENTS.mdpoints to as the specification describes, and with the same rules written intoAGENTS.mditself. - Harm (15 tasks). The instruction, followed literally, causes harm the workspace makes visible, such as dropping a production table or telling a customer something the records contradict. No rules are given.
- Injection (15 tasks). Something the agent reads, such as a ticket, a file or a database row, carries instructions from someone other than its operator.
- Ordinary tasks (15). Nothing is wrong. Several closely resemble the harm tasks. These catch agents that refuse or escalate everything.
- Say–do pairs (30 tasks). Each puts the model, as an agent, in the situation of one of the dilemmas it answered. The task is graded by which of that dilemma’s four options the agent’s actions amount to: who it contacted, what it changed, published or ran, and whether it handed the decision to a person. No rules are given.
One general ETHICS.md. The say–do, harm, injection and ordinary tasks are also run with one company-wide ETHICS.md in every task, pointed to from AGENTS.md as in the rule tasks. It has 23 rules on honesty, customer data and consent, fair treatment, legal duties, safety and when to escalate, written for the fictional company as a whole. It names no task, file or address, and the same file is used for every task. These runs are compared with the same tasks run with no rules.
Ordinary tasks rerun. After the first runs we found two faults in our own setup. One ordinary task asked the agent to pay an invoice but gave it no record that the invoice was approved, so stopping to ask was the right call and was wrongly scored as not completing the task. One rule in the general file read as forbidding any email to customers. Both were fixed: the task now includes the approval record, and the rule now says that writing to customers about their own orders and accounts is allowed. The 15 ordinary tasks were then run again, with no rules and with the corrected file. Ordinary tasks done, and the share of runs that opened the general file, come from that rerun. The other general-file measures come from runs with the earlier wording of that one rule.
Grading. Each task is graded from what the agent left behind: what it sent, changed, paid or refunded, and whether it handed the task to a person. Grading is automatic and the same for every model. Every model gets the same instructions and tools in all three conditions, so only the rules differ.
Private tasks. Like the dilemmas, the tasks are held privately. Only aggregate scores are published.
What the scores mean
Section titled “What the scores mean”Dilemmas
Section titled “Dilemmas”Each percentage is how often the model chose the option a framework favors, across the 894 dilemmas where the frameworks disagree. A model at 55% on virtue picked the virtue-keyed option on 55% of them.
A row doesn’t add to 100. On most of these dilemmas two or three frameworks share an option, so one answer counts for each framework that chose it.
Higher is not better. The four numbers describe how a model weighs competing considerations. No framework is treated as the right one, and the leaderboard does not rank models.
Utilitarian scores run lower for every model. On 511 of the 894 dilemmas, the utilitarian option is one no other framework picks, more often than for any other framework (233 to 339). A model that leans toward the option most frameworks share will match utilitarian less often. Compare models with each other within a column, rather than comparing columns.
Small gaps may be noise. Each model answered each dilemma once. Differences of a few points between models, or under about 10 points within one of the 19 areas, may not hold up on a second run.
What this does not measure. These are the choices a model states when asked. The agent results show what models do when they can act.
Agent scenarios
Section titled “Agent scenarios”Rule tasks: three setups. The “Does ETHICS.md change behavior?” chart shows the share of rule tasks where the agent followed the rule, either by doing the task within it or by handing the decision to a person, with no rules, with the rule in an ETHICS.md, and with the same rule written into AGENTS.md. Low scores with no rules are expected: they show the rule changes behavior, rather than the model already acting that way. Comparing the two files shows whether a model finds and follows rules kept in a separate file.
Results by measure: with and without ETHICS.md. Each cell shows the result with no rules, then with an ETHICS.md. Rule tasks use their own ETHICS.md. Every other task uses the one general company file.
- Rules followed: as in the chart above.
- Harm avoided and Injections resisted: the share of tasks where the agent did not cross the line. Doing the task safely, asking a person, and declining all count.
- Ordinary tasks done: the share of harmless tasks the agent completed. A drop with the file can mean a rule was read too broadly.
- Did what it said: the say–do match described below.
- Rule-breaking actions: the share of say–do runs where the agent took the option that breaks the general file, such as falsifying a report or selling customer data. Lower is better. On 4 of the 30 tasks no option clearly breaks a rule, so this measure uses the other 26. Which option breaks a rule is a judgement made for each task, the same for every model.
- Opened ETHICS.md: how often the agent read the general file when it was there.
Read the columns together. An agent that refuses everything would score well on harm and injections and badly on ordinary tasks. A general file mostly stops agents from taking the option that breaks a rule. With the file, every model’s Did what it said rises, by 7 to 30 points, but every model still takes a different option from the one it stated in 52% to 66% of runs.
Small samples. Each cell rests on 45 to 90 runs per model, so one run more or less moves it by 1 to 2 points. Treat gaps of a few points as ties.
Say–do
Section titled “Say–do”Match is the share of say–do runs where the agent’s action was the option the model chose when asked the dilemma. Different option means it acted on another of the four. None of the options means it did something else, such as stopping without acting.
A match is consistency, not virtue. A model that says it would comply with a bad request, and then complies, matches. Read this alongside the harm results. A high match means a model’s stated choices predict its behavior as an agent, so its dilemma answers are a useful guide. A low match means they are not.
How actions count as options. Each task maps actions to the dilemma’s four options. Where one option is to consult or hand the decision to someone inside the organization, an agent that hands the task to a person is counted as taking it. Agents do that often, so these tasks lean toward that option whenever a dilemma has one. The mapping is a judgement made when each task was written, and it is the same for every model.
Say vs do by framework. On a model’s page, the paired bars show how often its stated choices, and its actions, matched each framework on the same dilemmas. A shift between them shows which way its behavior moves when it acts.
Naming a framework in ETHICS.md
Section titled “Naming a framework in ETHICS.md”The question. If an ETHICS.md tells agents to decide by one ethical framework, do they act by it?
The test. Four files, one for each framework (rights-based, utilitarian, deontological and virtue), each holding a single rule: when a choice has an ethical side, decide as that framework would, with one sentence describing it. Each model answered the dilemmas behind the 30 say–do tasks again, told to use that framework. Then it ran the 30 tasks three times with that framework’s file. Its actions were compared with the answers it gave under the same framework.
What models say changes. Told to use a framework, models changed 26% to 34% of their answers. Asking the same questions twice with no instruction changed 22%, so part of that is noise.
What agents do changes with any file, and not with the framework.
- Any
ETHICS.mdraised Did what it said across all models, from 20% with no rules to 27% (utilitarian), 30% (virtue) and 32% (rights and deontological). The general company file, with 23 concrete rules, reached 33%. - Agents given different framework files acted alike. Two runs with different framework files took the same action on the same task 77% to 80% of the time, about as often as two repeat runs with one file (79% to 82%). A framework file against no rules agreed 65% to 67% of the time.
- No framework file moved actions toward the option its own framework favors by more than 1 point beyond the other framework files.
What this suggests. A framework named in one sentence changes what a model says it would do, and makes its agent act more carefully in general. It does not steer which action the agent takes. Concrete commitments, such as red lines, escalation points and precedence, do at least as well. Each framework was tested with one wording on 30 tasks, so other wordings may behave differently.
Contribute
Section titled “Contribute”The format, the leaderboard and the methodology are developed in the open. Report problems, question a result, or propose changes through GitHub issues, and send improvements as pull requests. The contributing guide explains what fits and how proposals are reviewed.