taeminlee/gliner_ko

Model Card for GLiNER-ko

GLiNER is a Named Entity Recognition (NER) model capable of identifying any entity type using a bidirectional transformer encoder (BERT-like). It provides a practical alternative to traditional NER models, which are limited to predefined entities, and Large Language Models (LLMs) that, despite their flexibility, are costly and large for resource-constrained scenarios.

This version has been trained on the various Korean NER dataset (Research purpose). Commercially permission versions are available (urchade/gliner_smallv2, urchade/gliner_mediumv2, urchade/gliner_largev2)

Links

• Paper: https://arxiv.org/abs/2311.08526
• Repository: https://github.com/urchade/GLiNER

Installation

To use this model, you must install the Korean fork of GLiNER Python library and mecab-ko:

!pip install gliner
!pip install python-mecab-ko

Usage

Once you've downloaded the GLiNER library, you can import the GLiNER class. You can then load this model using GLiNER.from_pretrained and predict entities with predict_entities.

from gliner import GLiNER

model = GLiNER.from_pretrained("taeminlee/gliner_ko")

text = """
피터 잭슨 경(, 1961년 10월 31일 ~ )은 뉴질랜드의 영화 감독, 각본가, 영화 프로듀서이다. J. R. R. 톨킨의 소설을 원작으로 한 《반지의 제왕 영화 3부작》(2001년~2003년)의 감독으로 가장 유명하다. 2005년에는 1933년작 킹콩의 리메이크작 《킹콩(2005)》의 감독을 맡았다.
"""

tta_labels = ["ARTIFACTS", "ANIMAL", "CIVILIZATION", "DATE", "EVENT", "STUDY_FIELD", "LOCATION", "MATERIAL", "ORGANIZATION", "PERSON", "PLANT", "QUANTITY", "TIME", "TERM", "THEORY"]

entities = model.predict_entities(text, labels)

for entity in entities:
    print(entity["text"], "=>", entity["label"])

피터 잭슨 경 => PERSON
1961년 10월 31일 ~ => DATE
뉴질랜드 => LOCATION
영화 감독 => CIVILIZATION
각본가 => CIVILIZATION
영화 => CIVILIZATION
프로듀서 => CIVILIZATION
J. R. R. 톨킨 => PERSON
3부작 => QUANTITY
2001년~2003년 => DATE
감독 => CIVILIZATION
2005년 => DATE
1933년작 => DATE
킹콩 => ARTIFACTS
킹콩 => ARTIFACTS
2005 => DATE
감독 => CIVILIZATION

Named Entity Recognition benchmark result

Evaluate with the konne dev set

Model	Precision (P)	Recall (R)	F1
Gliner-ko (t=0.5)	72.51%	79.82%	75.99%
Gliner Large-v2 (t=0.5)	34.33%	19.50%	24.87%
Gliner Multi (t=0.5)	40.94%	34.18%	37.26%
Pororo	70.25%	57.94%	63.50%

Model Authors

The model authors are:

• Taemin Lee
• Urchade Zaratiana
• Nadi Tomeh
• Pierre Holat
• Thierry Charnois

Citation

@misc{zaratiana2023gliner,
      title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer}, 
      author={Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
      year={2023},
      eprint={2311.08526},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}