Evaluation LLMs by

EVALUATION

miroslavamatejova — 2023-09-19 15:45:50 UTC

Benchmarks

miroslavamatejova — 2023-09-20 05:39:02 UTC

Metrics

miroslavamatejova — 2023-09-20 05:39:17 UTC

Frameworks

miroslavamatejova — 2023-09-20 05:43:52 UTC

Datasets

miroslavamatejova — 2023-09-20 05:46:19 UTC

miroslavamatejova — 2023-09-20 05:46:55 UTC

https://www.markiiisys.com/blog/benchmarking-large-language-models-llms-a-quick-tour/

miroslavamatejova — 2023-09-20 05:56:52 UTC

https://levelup.gitconnected.com/how-to-benchmark-language-models-by-openai-deepmind-google-microsoft-783d4307ec50

GLUE

miroslavamatejova — 2023-09-20 06:02:56 UTC

https://gluebenchmark.com/

SuperGLUE

miroslavamatejova — 2023-09-20 06:03:13 UTC

SuperGLUE Benchmark
https://deepgram.com/learn/superglue-llm-benchmark-explained

MMLU

miroslavamatejova — 2023-09-20 06:11:12 UTC

GitHub - hendrycks/test: Measuring Massive Multitask Language Understanding | ICLR 2021

Big Bench

miroslavamatejova — 2023-09-20 06:11:52 UTC

GitHub - google/BIG-bench: Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

miroslavamatejova — 2023-09-20 06:25:37 UTC

https://tech.ebu.ch/docs/events/webinar_2023_llm_benchmarking/presentations/webinar-llm-tv2.pdf

How to Benchmark?

miroslavamatejova — 2023-09-20 06:39:30 UTC

Task-Specific Evaluation

miroslavamatejova — 2023-09-20 06:40:01 UTC

Few-Shot Learning Evaluation

miroslavamatejova — 2023-09-20 06:40:15 UTC

Zero-Shot Learning Evaluation

miroslavamatejova — 2023-09-20 06:40:49 UTC

Fine-Tuning Evaluation

miroslavamatejova — 2023-09-20 06:50:51 UTC

GitHub - facebookresearch/llama-recipes: Examples and recipes for Llama 2 model

Human Evaluation

miroslavamatejova — 2023-09-20 06:51:08 UTC

Beyond Metrics: A Hybrid Approach to LLM Performance Evaluation (topbots.com)

Human Evaluation of Large Language Models: How Good is Hugging Face's BLOOM? (surgehq.ai)

Bias and Fairness Evaluation

miroslavamatejova — 2023-09-20 06:51:23 UTC

Safety and Robustness Evaluation

miroslavamatejova — 2023-09-20 06:52:10 UTC

Perplexity

miroslavamatejova — 2023-09-20 06:53:08 UTC

BLEU (Bilingual Evaluation Understudy) Score

miroslavamatejova — 2023-09-20 06:53:29 UTC

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Score

miroslavamatejova — 2023-09-20 06:53:46 UTC

F1 Score, Accuracy, Precision, Recall

miroslavamatejova — 2023-09-20 06:54:00 UTC

Spearman's Rank Correlation Coefficient

miroslavamatejova — 2023-09-20 07:23:55 UTC

.

miroslavamatejova — 2023-09-20 07:24:07 UTC

Language Model Evaluation Harness

miroslavamatejova — 2023-09-20 07:30:15 UTC

GitHub - EleutherAI/lm-evaluation-harness: A framework for few-shot evaluation of autoregressive language models.

Evaluating Large Language Models (LLMs) with Eleuther AI | llm-evaluation – Weights & Biases (wandb.ai)

HELM

miroslavamatejova — 2023-09-20 07:30:32 UTC

GitHub - stanford-crfm/helm: Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110).

OpenAI Evals

miroslavamatejova — 2023-09-20 07:31:36 UTC

GitHub - openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

miroslavamatejova — 2023-09-20 07:43:13 UTC

https://deepgram.com/learn/llm-benchmarks-guide-to-evaluating-language-models

Semantic Answer Similarity (SAS)

miroslavamatejova — 2023-09-20 15:51:02 UTC

[2108.06130] Semantic Answer Similarity for Evaluating Question Answering Models (arxiv.org)

How to evaluate an LLM on your data? (radekosmulski.com)

evaluate_LLM.ipynb · GitHub

METEOR

miroslavamatejova — 2023-09-20 15:52:28 UTC

METEOR | Machine Translate

Tasks

miroslavamatejova — 2023-09-21 05:51:47 UTC

miroslavamatejova — 2023-09-21 05:52:01 UTC

LLM Evaluation Metrics (mosaicml.com)

miroslavamatejova — 2023-09-21 06:26:42 UTC

Evaluating Large Language Models (toloka.ai)

Question answering

miroslavamatejova — 2023-09-21 14:50:46 UTC

Summarization

miroslavamatejova — 2023-09-21 14:51:08 UTC

Text Summarization Llama2: how to Use LLama2 with Langchain | by Tarik Kaoutar (高達烈) | Level Up Coding (gitconnected.com)

Classification

miroslavamatejova — 2023-09-21 14:51:55 UTC

Text generation

miroslavamatejova — 2023-09-21 14:52:14 UTC

Machine translation

miroslavamatejova — 2023-09-22 09:35:51 UTC

Logic, math, code

miroslavamatejova — 2023-09-22 09:41:50 UTC

logical reasoning
mathematics
computer code

Auto-evaluation

miroslavamatejova — 2023-09-25 06:59:08 UTC

Auto-Evaluator (langchain.com)

Jiant

miroslavamatejova — 2023-09-25 15:17:54 UTC

GitHub - nyu-mll/jiant: jiant is an nlp toolkit

miroslavamatejova — 2023-09-25 15:23:55 UTC

GitHub - huggingface/evaluate: 🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

prepojiť ktore metriky obsahuje?

Library

miroslavamatejova — 2023-09-25 15:24:15 UTC

miroslavamatejova — 2023-09-25 15:31:30 UTC

lm-evaluation-harness/docs/task_table.md at master · EleutherAI/lm-evaluation-harness · GitHub

miroslavamatejova — 2023-09-25 15:39:05 UTC

helm/src/helm/benchmark/metrics at main · stanford-crfm/helm · GitHub

miroslavamatejova — 2023-09-25 19:07:21 UTC

How to Validate OpenAI GPT Model Performance with Text Summarization | by Mark Chen | Towards Data Science

BERTscore

miroslavamatejova — 2023-09-25 19:15:04 UTC

Reasoning

miroslavamatejova — 2023-09-25 19:21:49 UTC

General

miroslavamatejova — 2023-09-25 19:22:16 UTC

Specific

miroslavamatejova — 2023-09-25 19:22:23 UTC

miroslavamatejova — 2023-09-25 20:49:22 UTC

https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/#h-table-of-the-major-existing-evaluation-frameworks

Exact match

miroslavamatejova — 2023-09-27 07:58:13 UTC

fluency, coherence, Relevance, Context understanding

miroslavamatejova — 2023-09-27 13:23:50 UTC

miroslavamatejova — 2023-09-28 06:09:03 UTC

SuperGLUE Benchmark tasks/datasets

Definition of business problem

miroslavamatejova — 2023-10-25 08:12:30 UTC

Task type

miroslavamatejova — 2023-10-25 08:12:39 UTC

General-purpose: Designed to understand and process a wide range of language tasks and contexts, without aiming for a particular language domain or use case.
Specific:
- Question Answering
- Information Retrieval
- Document Summarization
- Machine Translation
- Sentiment analysis
- Text Classification
- ...

Model

miroslavamatejova — 2023-10-25 08:13:55 UTC

Questions of cost and accuracy.
Is the model freely available? Paid?
Access (API...)
Latency
Which model performs best in your use case on leaderboard?

Data

miroslavamatejova — 2023-10-25 08:14:01 UTC

Metrics

miroslavamatejova — 2023-10-25 08:14:24 UTC

Evaluation metrics vary by use case.
See Quantitative Metrics, Reference Comparisons and Criteria-Based Evaluation in the document.

Benchmark datasets

miroslavamatejova — 2023-10-25 08:17:33 UTC

Own datasets

miroslavamatejova — 2023-10-25 08:17:38 UTC

The format of data varies based on the model and task.

Evaluation goals

miroslavamatejova — 2023-10-25 08:19:56 UTC

Do we want to track one goal or several functionalities of the model? For metrics used for a specific task type, see the document.
If we use data that someone else has already used, do we want to compare our metric results with their results? If so, then we must choose the same metrics.
What approach will we use to evaluate our solution?
Are we only interested in the accuracy of the model or other features such as grammar, coherence, relevance, politeness?
Who will be the users? Is it important to also focus on social aspects like bias, fairness, toxicity, misinformation, hallucination?

Evaluation

miroslavamatejova — 2023-10-25 08:26:53 UTC

See the benchmarks and their implementation and the metrics library in the document.
Analysis of results.

Approach

miroslavamatejova — 2023-10-26 12:16:59 UTC

Standard evaluation
- Evaluation using NLP and other standard metrics.
Auto-evaluation/Self-evaluation (LLM evaluation)
- LLMs can be used to check the result of their own or other LLM's outputs.
Human evaluation
- Human annotators are used to assess the quality of open-ended model responses.
Hybrid evaluation
- Combination of Auto-evaluation and Human evaluation