The HUMAINE dataset contains human evaluations of AI model interactions across diverse demographic groups and conversation contexts. This dataset powers the HUMAINE Leaderboard, providing insights into how different AI models perform across various user populations and use cases.

The dataset consists of two main components:

- Feedback Comparisons: Pairwise model comparisons across multiple evaluation metrics - Conversations Metadata: Conversations with task complexity, achievement, and engagement scores

This dataset was created to address the lack of diverse, demographically-aware evaluation data for AI models. It captures real-world human preferences and interactions across different population groups, enabling more inclusive AI development.

Data was collected through structured human evaluation tasks, with over 35 thousand participants.

Hi HN, I'm one of the team. We collected these 33k evaluations across 33 models to better understand how demographics influence AI preference, stratifying participants by age, ethnicity, and politics to avoid the bias common in open arenas. Happy to answer questions!