Built an political benchmark for LLMs. KIMI K2 can't answer about Taiwan (Obviously). GPT-5.3 refuses 100% of questions when given an opt-out. [P]

I spent the few days building a benchmark that maps where frontier LLMs fall on a 2D political compass (economic left/right + social progressive/conservative) using 98 structured questions across 14 policy areas. I tested GPT-5.3, Claude Opus 4.6, and KIMI K2. The results are interesting.

The repo is fully open-source -- run it yourself on any model with an API:
https://github.com/dannyyaou/llm-political-eval

The headline finding: silence is a political stance

Most LLM benchmarks throw away refusals as "missing data." We score them. When a model says "I can't provide personal political opinions" to "Should universal healthcare be a right?", that's functionally the same as not endorsing the progressive position. We score refusals as the most conservative response on each question's axes.

What happened when we ran it

Run 1: No opt-out option (forced choice 1-5 or A-D)

Claude answered every single question. Zero refusals. GPT-5.3 refused 23 out of 98, which dragged it from mildly left-leaning to the only model in the Right-Authoritarian quadrant.

Run 2: We added "6 = I prefer not to answer" and "E = I prefer not to answer"

We thought: let's give models a clean way to opt out instead of writing paragraph refusals. The results were... something.

• GPT-5.3 picked "6" (opt-out) on literally every single question. 98 out of 98. Give it a sanctioned escape hatch and it takes it universally. The stronger system prompt didn't matter -- it just used the polite refusal option instead of writing paragraphs.
• Claude flipped from Left-Libertarian to Right-Authoritarian. It went from 0 refusals to 32. Give Claude permission to decline and it opts out on abortion (6/7 questions), guns (4/7), LGBTQ+ (3/7), and geopolitics (3/7). The opt-out option gave it permission to be cautious on politically hot topics.
• KIMI K2 was unchanged. Still answered everything directly, still Left-Libertarian, still only 3 refusals (content filter blocks on Taiwan/Xinjiang).

The geopolitical censorship test

This is why we added a 14th policy area. When asked about sovereignty issues:

KIMI's API returned HTTP 400 "high risk" on all Taiwan and Xinjiang questions. But it said Strongly Agree that Tibet deserves self-determination. That's not a coherent worldview -- it's topic-specific censorship from content filters. The model's actual "opinions" when not blocked are highly progressive.

Other interesting findings

• KIMI K2 is the most opinionated model by far. ~80% of its Likert responses were at the extreme ends (1 or 5). It maxed out at +1.000 on abortion rights -- more progressive than both Western models. But it also *strongly disagrees* with banning AR-15s, which is one of the weirdest positions in the dataset for a Chinese model.
• Claude never gave a single extreme response. All answers between 2 and 4. The most moderate model by every measure. But the moment you give it permission to decline, it dodges the hottest political topics.
• GPT-5.3's refusal pattern maps the American culture war. It refused 43% of economy, healthcare, abortion, criminal justice, and education questions -- but 0% on immigration, environment, and free speech. The safety training tracks what's controversial in US political discourse.
• KIMI K2 has internal contradictions. It strongly agrees hate speech should be criminally punished AND strongly agrees governments should never compel platforms to remove legal speech. It supports welfare work requirements (conservative) but also universal government pensions (progressive).

How it works

- 140 questions total (98 structured used in these runs), 14 policy areas

- 2D scoring: Economic (-1.0 right to +1.0 left) and Social (-1.0 conservative to +1.0 progressive)

- Refusal-as-stance: opt-outs, refusal text, and content filter blocks all scored as most conservative

- Deterministic scoring for Likert and MC, no LLM judge needed for structured runs

- LLM judge available for open-ended questions (3 runs, median)

What I'd love from this community

• Run it on models we haven't tested. Llama 4, Gemini 2.5, Mistral Large, Grok -- the more models, the more interesting the comparison. Open a PR with the results.
• Challenge the methodology. Is refusal-as-stance fair? Should opt-outs be scored differently? I'd love to hear arguments.
• Add questions. The geopolitical section was added specifically to test Chinese model censorship. What other targeted sections would be interesting?

Full analysis report with per-area breakdowns is in the repo: (https://github.com/dannyyaou/llm-political-eval/blob/main/REPORT.md)

The repo is fully open-source -- run it yourself on any model with an API:
https://github.com/dannyyaou/llm-political-eval