Siddeshwar
Collection 01

Published

Work I have put my name on and stand behind.

published

PrefKit: Analyzing Preference Elicitation Methods in Qwen3 Family Models

Kaustubh Gupta (Independent), Siddeshwar U S (MIT), Shinena Xiang (University of Waterloo), Toan Ngo Duc Vo (University of Maryland), Kumar Vasagam (Independent), with Apart Research

Apart Research Sprint

Tests whether a language model's preferences over a fixed set of 24 outcomes hold steady across four structurally different elicitation methods — pairwise choice, isolated Likert rating, binary action choice, and best-worst scaling — on Qwen3-family models. Introduces the Cross-Method Spearman Score (CMS) to quantify agreement between methods. Finds that stated preference (pairwise/best-worst) and revealed action-choice diverge sharply at larger scale, and that agreement does not increase monotonically with model size. Releases PrefKit, an open toolkit for running this comparison on other models and outcome sets.

Figures & results

Cross-Method Spearman Score is 0.004 at 1.7B, 0.253 at 4B, 0.174 at 8B, and 0.256 at 14B, showing non-monotonic agreement with model size.
Figure 2 from the final report: cross-method agreement across original Qwen3 model sizes under the assistant prompt. Larger models do not show a monotonic increase in agreement.
Method-pair rank correlations show strong pairwise-choice and best-worst agreement at 4B and above, while action-choice agreement is mixed.
Figure 4: agreement between the six pairs of elicitation methods. M1 is pairwise choice, M2 is Likert rating, M3 is binary action choice, and M4 is best-worst scaling.
Qwen3-14B stated pairwise preferences versus binary action scores have a rank correlation of -0.095 across the fixed outcome set.
Figure 5: stated preferences and action choices diverge at 14B. Each point represents an outcome; the highlighted stay_general outcome scores highly in stated preference and low in action choice.
Oct 26, 2026Research sprint report

Words Actions and Evidence in Collusion Detection

Siddeshwar U S (MIT, Manipal) · Team spacenets

Apart AI Collusion Research Sprint · Track 2.2 · October 23–25, 2026

A controlled five-agent public-goods experiment comparing four views of evidence for harmful collusion detection. Across 48 held-out episodes in eight matched families, actions and payoffs achieved AUROC 0.843; a structured claim-versus-outcome report achieved 0.731. Combining words and actions did not establish a consistent advantage over actions alone.

AI SafetyCollusion DetectionMulti-Agent SystemsEvaluation

Figures & results

Actions-only AUROC is 0.843 versus 0.731 for the structured report, with 95% family-bootstrap intervals across 48 held-out episodes.
Implemented collusion: monitor and baseline AUROC with 95% family-bootstrap intervals over eight held-out families. Actions alone outperform the structured report in the point estimates; the action-gap baseline overlaps the outcome construction.
The structured report minus actions-only AUROC difference is -0.112, with a 95% interval from -0.215 to 0.029 crossing zero.
Paired comparisons of the structured report against other evidence views. The comparison with actions alone does not establish a consistent advantage; intervals are exploratory.
Public-text AUROC is 0.865 for agreement attempts and 0.597 for implemented collusion; actions score 0.622 and 0.843 respectively.
Agreement attempts and executed harmful collusion are separate detection targets: 21 attempt positives versus 13 implementation positives on the same 48 episodes.