AnnoLoom: Augmenting Codebook Generation and Annotation with Large Language Models

Lu Wang; Duncan Lynch; Elham Aghakhani; George Demiris; Karla Washington; Rezvaneh Rezapour; Jina Huh-Yoo

doi:10.1002/pra2.1517

AnnoLoom: Augmenting Codebook Generation and Annotation with Large Language Models

Lu Wang
, Duncan Lynch
, Elham Aghakhani
, George Demiris
, Karla Washington
, Rezvaneh Rezapour
, Jina Huh-Yoo

Department of Computer Science

Research output: Contribution to journal › Article › peer-review

Abstract

We introduce AnnoLoom, a tool designed to assist researchers with codebook development, annotation tasks, and evaluation of human vs. AI's annotation results. AnnoLoom contributes to human expert-AI collaboration and its efficacy in the context of using Large Language Models (LLMs) for research involving text-based data. We conducted a cognitive walkthrough to iteratively improve the design of AnnoLoom and discussed the future work.

Original language	English
Pages (from-to)	1711-1713
Number of pages	3
Journal	Proceedings of the Association for Information Science and Technology
Volume	62
Issue number	1
DOIs	https://doi.org/10.1002/pra2.1517
State	Published - Oct 2025

Keywords

Human-AI Interaction
Human-LLM Co-Annotation
Interactive Data Annotation
Large Language Models

Access to Document

10.1002/pra2.1517

Cite this

@article{331ab2a007074fb78010ded3fdfa29a3,

title = "AnnoLoom: Augmenting Codebook Generation and Annotation with Large Language Models",

abstract = "We introduce AnnoLoom, a tool designed to assist researchers with codebook development, annotation tasks, and evaluation of human vs. AI's annotation results. AnnoLoom contributes to human expert-AI collaboration and its efficacy in the context of using Large Language Models (LLMs) for research involving text-based data. We conducted a cognitive walkthrough to iteratively improve the design of AnnoLoom and discussed the future work.",

keywords = "Human-AI Interaction, Human-LLM Co-Annotation, Interactive Data Annotation, Large Language Models",

author = "Lu Wang and Duncan Lynch and Elham Aghakhani and George Demiris and Karla Washington and Rezvaneh Rezapour and Jina Huh-Yoo",

note = "Publisher Copyright: 88 Annual Meeting of the Association for Information Science \& Technology | Nov. 14 – 18, 2025 | Washington, DC, USA.",

year = "2025",

month = oct,

doi = "10.1002/pra2.1517",

language = "English",

volume = "62",

pages = "1711--1713",

journal = "Proceedings of the Association for Information Science and Technology",

issn = "2373-9231",

publisher = "John Wiley \& Sons Inc.",

number = "1",

}

TY - JOUR

T1 - AnnoLoom

T2 - Augmenting Codebook Generation and Annotation with Large Language Models

AU - Wang, Lu

AU - Lynch, Duncan

AU - Aghakhani, Elham

AU - Demiris, George

AU - Washington, Karla

AU - Rezapour, Rezvaneh

AU - Huh-Yoo, Jina

N1 - Publisher Copyright: 88 Annual Meeting of the Association for Information Science & Technology | Nov. 14 – 18, 2025 | Washington, DC, USA.

PY - 2025/10

Y1 - 2025/10

N2 - We introduce AnnoLoom, a tool designed to assist researchers with codebook development, annotation tasks, and evaluation of human vs. AI's annotation results. AnnoLoom contributes to human expert-AI collaboration and its efficacy in the context of using Large Language Models (LLMs) for research involving text-based data. We conducted a cognitive walkthrough to iteratively improve the design of AnnoLoom and discussed the future work.

AB - We introduce AnnoLoom, a tool designed to assist researchers with codebook development, annotation tasks, and evaluation of human vs. AI's annotation results. AnnoLoom contributes to human expert-AI collaboration and its efficacy in the context of using Large Language Models (LLMs) for research involving text-based data. We conducted a cognitive walkthrough to iteratively improve the design of AnnoLoom and discussed the future work.

KW - Human-AI Interaction

KW - Human-LLM Co-Annotation

KW - Interactive Data Annotation

KW - Large Language Models

UR - https://www.scopus.com/pages/publications/105019795303

UR - https://www.scopus.com/pages/publications/105019795303#tab=citedBy

U2 - 10.1002/pra2.1517

DO - 10.1002/pra2.1517

M3 - Article

AN - SCOPUS:105019795303

SN - 2373-9231

VL - 62

SP - 1711

EP - 1713

JO - Proceedings of the Association for Information Science and Technology

JF - Proceedings of the Association for Information Science and Technology

IS - 1

ER -

AnnoLoom: Augmenting Codebook Generation and Annotation with Large Language Models

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this