flan-t5-large-finetuned–AIHQ-Rating

A fine-tuned Flan-T5 Large model for rating open-ended questions on "attribution of hostility" and "aggression response", trained on the Ambiguous Intentions Hostility Questionnaire (AIHQ) dataset. This model yields ratings that closely match those of trained human raters.

Model Details

Base model: flan-t5-large
Fine-tuned on: AIHQ open-ended questions rating data
Tasks:
- Rate attribution of hostility on a scale of 1-5
- Rate aggression response on a scale of 1-5

Training Data

The Ambiguous Intentions Hostility Questionnaire (AIHQ) presents short vignettes of ambiguous social scenarios; trained human raters score people's responses for attribution of hostility and aggression response.

Size: ~1,200 annotated responses
Annotators: Trained raters
Splits: 50% train / 50% test

Evaluation

On the 50% test set, this fine-tuned model achieves:

Pearson’s r ≈ 0.87 with human rater scores on attribution of hostility
Pearson’s r ≈ 0.93 with human rater scores on aggression response

On a completely new dataset (with no data included in training data), this fine-tuned model achieves:

Pearson’s r ≈ 0.75 with human rater scores on attribution of hostility
Pearson’s r ≈ 0.83 with human rater scores on aggression response

These metrics demonstrate a high correlation with human rater judgments.

Usage

We developed a browser-based interface that runs on the user's machine and allows users to upload AIHQ responses to obtain model-generated ratings.

The interface, along with installation instructions and example input templates, is available at: https://aihqrating.readthedocs.io/en/latest/index.html.