---
datasets:
- samirmsallem/text_parts_de
language:
- de
base_model:
- deepset/gbert-base
pipeline_tag: text-classification
library_name: transformers
tags:
- science
- german
- text
metrics:
- accuracy
model-index:
- name: checkpoints
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: samirmsallem/text_parts_de
      type: samirmsallem/text_parts_de
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.7030812324929971
---

## Sequence classification model for text parts classification in German scientific texts

**gbert-base-text_parts** is a sequence classification model in the scientific domain in German, finetuned from the model [gbert-base](https://huggingface.co/deepset/gbert-base). 
It was trained using a custom annotated dataset of around 4,800 training and 1,400 test examples containing introduction, main part and conclusion text sequences from scientific papers and theses in german.


The two German BERT models — gbert-base and gbert-large — were evaluated on the same text classification task. This repository contains the model and results for [gbert-base](https://huggingface.co/deepset/gbert-base).

- gbert-base reached a maximum evaluation accuracy of 70.31%, with a minimum evaluation loss of 2.13.
- gbert-large achieved a higher peak accuracy of 78.50% and a lower evaluation loss of 1.67.

Despite its superior peak performance, gbert-large exhibited a consistent degradation in evaluation accuracy and an increase in loss over subsequent epochs. 
This pattern is characteristic of overfitting, where the model fits the training data too closely and fails to generalize well to unseen data.
In contrast, gbert-base maintained more stable performance over time, making it a more robust choice in scenarios where computational resources or dataset size are limited.

|Text Classification Tag| Text Classification Label | Description                             |
| :----:                |    :----:                 |    :----:   |
| 0                     | EINLEITUNG          | The text is an introduction (Einleitung). |
| 1                     | HAUPTTEIL              | The text is a main part (Hauptteil). |
| 2                     | SCHLUSS              | The text is a conclusion (Schluss). |


### Training
Training was conducted on a 10 epoch fine-tuning approach:

| Epoch | Loss    | Accuracy |
|-------|---------|----------|
| 1.0   | **2.1272**  | 0.6835   |
| 2.0   | 2.2270  | 0.7017   |
| 3.0   | 2.3057  | 0.7003   |
| 4.0   | 2.4637  | 0.6996   |
| 5.0   | 2.6276  | 0.6996   |
| 6.0   | 2.6948  | **0.7031** |
| 7.0   | 2.7076  | **0.7031** |
| 8.0   | 2.7590  | **0.7031** |
| 9.0   | 2.7891  | 0.7024   |
| 10.0  | 2.7976  | **0.7031** |





Training was conducted using a standard Text classification objective. The model achieves an accuracy of approximately 70% on the evaluation set.

Here are the overall final metrics on the test dataset after 10 epochs of training:
  - **Accuracy**: 0.7030812324929971
  - **Loss**: 2.7976272106170654