# data[].attributes.body
Standard evaluation

Evaluation using NLP and other standard metrics.

Auto-evaluation/Self-evaluation (LLM evaluation)

LLMs can be used to check the result of their own or other LLM's outputs.

Human evaluation

Human annotators are used to assess the quality of open-ended model responses.

Hybrid evaluation

Combination of Auto-evaluation and Human evaluation

# data[].attributes.body
See the benchmarks and their implementation and the metrics library in the document.
Analysis of results.

# data[].attributes.body
Do we want to track one goal or several functionalities of the model? For metrics used for a specific task type, see the document.

If we use data that someone else has already used, do we want to compare our metric results with their results? If so, then we must choose the same metrics.

What approach will we use to evaluate our solution?

Are we only interested in the accuracy of the model or other features such as grammar, coherence, relevance, politeness?

Who will be the users? Is it important to also focus on social aspects like bias, fairness, toxicity, misinformation, hallucination?

# data[].attributes.body
The format of data varies based on the model and task.

# data[].attributes.body
GLUE Benchmark

SuperGLUE Benchmark

evals/evals/registry/data at main · openai/evals · GitHub

...

# data[].attributes.body
Evaluation metrics vary by use case.
See Quantitative Metrics, Reference Comparisons and Criteria-Based Evaluation in the document.

# data[].attributes.body
Questions of cost and accuracy.
Is the model freely available? Paid?
Access (API...)
Latency
Which model performs best in your use case on leaderboard?

LLM Explorer: Large Language Model Directory and Analytics (extractum.io)

Holistic Evaluation of Language Models (HELM) (stanford.edu)

Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4

# data[].attributes.body
General-purpose: Designed to understand and process a wide range of language tasks and contexts, without aiming for a particular language domain or use case.

Specific:

Question Answering

Information Retrieval

Document Summarization

Machine Translation

Sentiment analysis

Text Classification

...

# data[].attributes.body
SuperGLUE Benchmark tasks/datasets

# data[].attributes.body
https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/#h-table-of-the-major-existing-evaluation-frameworks

# data[].attributes.body
How to Validate OpenAI GPT Model Performance with Text Summarization | by Mark Chen | Towards Data Science

# data[].attributes.body
helm/src/helm/benchmark/metrics at main · stanford-crfm/helm · GitHub

# data[].attributes.body
lm-evaluation-harness/docs/task_table.md at master · EleutherAI/lm-evaluation-harness · GitHub

# data[].attributes.body
GitHub - huggingface/evaluate: 🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

prepojiť ktore metriky obsahuje?

# data[].attributes.body
GitHub - nyu-mll/jiant: jiant is an nlp toolkit

# data[].attributes.body
Auto-Evaluator (langchain.com)

# data[].attributes.body
logical reasoning
mathematics
computer code

# data[].attributes.body
Text Summarization Llama2: how to Use LLama2 with Langchain | by Tarik Kaoutar (高達烈) | Level Up Coding (gitconnected.com)

# data[].attributes.body
Evaluating Large Language Models (toloka.ai)

# data[].attributes.body
LLM Evaluation Metrics (mosaicml.com)

# data[].attributes.body
METEOR | Machine Translate

# data[].attributes.body
[2108.06130] Semantic Answer Similarity for Evaluating Question Answering Models (arxiv.org)

How to evaluate an LLM on your data? (radekosmulski.com)

evaluate_LLM.ipynb · GitHub

# data[].attributes.body
https://deepgram.com/learn/llm-benchmarks-guide-to-evaluating-language-models

# data[].attributes.body
GitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

# data[].attributes.body
GitHub - stanford-crfm/helm: Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110).

# data[].attributes.body
GitHub - EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.

Evaluating Large Language Models (LLMs) with Eleuther AI | llm-evaluation – Weights & Biases (wandb.ai)

# data[].attributes.headline
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Score

# data[].attributes.subject
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Score

# data[].attributes.headline
BLEU (Bilingual Evaluation Understudy) Score

# data[].attributes.subject
BLEU (Bilingual Evaluation Understudy) Score

# data[].attributes.body
Beyond Metrics: A Hybrid Approach to LLM Performance Evaluation (topbots.com)

Human Evaluation of Large Language Models: How Good is Hugging Face's BLOOM? (surgehq.ai)

# data[].attributes.body
GitHub - facebookresearch/llama-recipes: Examples and recipes for Llama 2 model

# data[].attributes.body
https://tech.ebu.ch/docs/events/webinar_2023_llm_benchmarking/presentations/webinar-llm-tv2.pdf

# data[].attributes.body
GitHub - google/BIG-bench: Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

# data[].attributes.body
GitHub - hendrycks/test: Measuring Massive Multitask Language Understanding | ICLR 2021

# data[].attributes.body
SuperGLUE Benchmark
https://deepgram.com/learn/superglue-llm-benchmark-explained

# data[].attributes.body
https://gluebenchmark.com/

# data[].attributes.body
https://levelup.gitconnected.com/how-to-benchmark-language-models-by-openai-deepmind-google-microsoft-783d4307ec50

# data[].attributes.body
https://www.markiiisys.com/blog/benchmarking-large-language-models-llms-a-quick-tour/