Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Daniele Cono D’Elia; Thaleia Dimitra Doudali; Cristiano Giuffrida; Miguel Matos; Mathias Payer; Solal Pirelli; Georgios Portokalidis; Valerio Schiavoni; Salvatore Signorello; Anjo Vahldiek-Oberwagner

doi:10.1145/3736731.3746152

Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Daniele Cono D’Elia
, Thaleia Dimitra Doudali
, Cristiano Giuffrida
, Miguel Matos
, Mathias Payer
, Solal Pirelli
, Georgios Portokalidis
, Valerio Schiavoni
, Salvatore Signorello
, Anjo Vahldiek-Oberwagner

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

Original language	English
Title of host publication	Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025
Pages	108-120
Number of pages	13
ISBN (Electronic)	9798400719585
DOIs	https://doi.org/10.1145/3736731.3746152
State	Published - 21 Oct 2025
Event	3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025 - Vancouver, Canada Duration: 29 Jul 2025 → 31 Jul 2025

Publication series

Name	Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025

Conference

Conference	3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025
Country/Territory	Canada
City	Vancouver
Period	29/07/25 → 31/07/25

Keywords

Artifact Evaluation
Conference-scale artifact evaluation experiences
Conference-scale artifact evaluation practices
Reproducibility

Access to Document

10.1145/3736731.3746152

Cite this

D’Elia, D. C., Doudali, T. D., Giuffrida, C., Matos, M., Payer, M., Pirelli, S., Portokalidis, G., Schiavoni, V., Signorello, S., & Vahldiek-Oberwagner, A. (2025). Lessons Learned from Five Years of Artifact Evaluations at EuroSys. In Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025 (pp. 108-120). (Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025). https://doi.org/10.1145/3736731.3746152

@inproceedings{738955ad78c045a5bfd10229d044d4b9,

title = "Lessons Learned from Five Years of Artifact Evaluations at EuroSys",

abstract = "Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.",

keywords = "Artifact Evaluation, Conference-scale artifact evaluation experiences, Conference-scale artifact evaluation practices, Reproducibility",

author = "D{\textquoteright}Elia, \{Daniele Cono\} and Doudali, \{Thaleia Dimitra\} and Cristiano Giuffrida and Miguel Matos and Mathias Payer and Solal Pirelli and Georgios Portokalidis and Valerio Schiavoni and Salvatore Signorello and Anjo Vahldiek-Oberwagner",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025 ; Conference date: 29-07-2025 Through 31-07-2025",

year = "2025",

month = oct,

day = "21",

doi = "10.1145/3736731.3746152",

language = "English",

series = "Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025",

pages = "108--120",

booktitle = "Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025",

}

D’Elia, DC, Doudali, TD, Giuffrida, C, Matos, M, Payer, M, Pirelli, S, Portokalidis, G, Schiavoni, V, Signorello, S & Vahldiek-Oberwagner, A 2025, Lessons Learned from Five Years of Artifact Evaluations at EuroSys. in Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025. Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025, pp. 108-120, 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025, Vancouver, Canada, 29/07/25. https://doi.org/10.1145/3736731.3746152

Lessons Learned from Five Years of Artifact Evaluations at EuroSys. / D’Elia, Daniele Cono; Doudali, Thaleia Dimitra; Giuffrida, Cristiano et al.
Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025. 2025. p. 108-120 (Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Lessons Learned from Five Years of Artifact Evaluations at EuroSys

AU - D’Elia, Daniele Cono

AU - Doudali, Thaleia Dimitra

AU - Giuffrida, Cristiano

AU - Matos, Miguel

AU - Payer, Mathias

AU - Pirelli, Solal

AU - Portokalidis, Georgios

AU - Schiavoni, Valerio

AU - Signorello, Salvatore

AU - Vahldiek-Oberwagner, Anjo

PY - 2025/10/21

Y1 - 2025/10/21

N2 - Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

AB - Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

KW - Artifact Evaluation

KW - Conference-scale artifact evaluation experiences

KW - Conference-scale artifact evaluation practices

KW - Reproducibility

UR - https://www.scopus.com/pages/publications/105022192359

UR - https://www.scopus.com/pages/publications/105022192359#tab=citedBy

U2 - 10.1145/3736731.3746152

DO - 10.1145/3736731.3746152

M3 - Conference contribution

AN - SCOPUS:105022192359

T3 - Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025

SP - 108

EP - 120

BT - Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025

T2 - 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025

Y2 - 29 July 2025 through 31 July 2025

ER -

D’Elia DC, Doudali TD, Giuffrida C, Matos M, Payer M, Pirelli S et al. Lessons Learned from Five Years of Artifact Evaluations at EuroSys. In Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025. 2025. p. 108-120. (Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025). doi: 10.1145/3736731.3746152

Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Cite this