Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Shaoyi Huang; Dongkuan Xu; Ian En Hsu Yen; Yijue Wang; Sung En Chang; Bingbing Li; Shiyang Chen; Mimi Xie; Sanguthevar Rajasekaran; Hang Liu; Caiwen Ding

doi:10.18653/v1/2022.acl-long.16

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Shaoyi Huang, Dongkuan Xu, Ian En Hsu Yen, Yijue Wang, Sung En Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Sanguthevar Rajasekaran, Hang Liu, Caiwen Ding

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

Original language	English
Title of host publication	ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Editors	Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Pages	190-200
Number of pages	11
ISBN (Electronic)	9781955917216
DOIs	https://doi.org/10.18653/v1/2022.acl-long.16
State	Published - 2022
Event	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland Duration: 22 May 2022 → 27 May 2022

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	1
ISSN (Print)	0736-587X

Conference

Conference	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
Country/Territory	Ireland
City	Dublin
Period	22/05/22 → 27/05/22

Access to Document

10.18653/v1/2022.acl-long.16

Cite this

Huang, S., Xu, D., Yen, I. E. H., Wang, Y., Chang, S. E., Li, B., Chen, S., Xie, M., Rajasekaran, S., Liu, H., & Ding, C. (2022). Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) (pp. 190-200). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1). https://doi.org/10.18653/v1/2022.acl-long.16

Huang, Shaoyi ; Xu, Dongkuan ; Yen, Ian En Hsu et al. / Sparse Progressive Distillation : Resolving Overfitting under Pretrain-and-Finetune Paradigm. ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). editor / Smaranda Muresan ; Preslav Nakov ; Aline Villavicencio. 2022. pp. 190-200 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{2b05d5b3bf0b4e89b1cc19523adc77b2,

title = "Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm",

abstract = "Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.",

author = "Shaoyi Huang and Dongkuan Xu and Yen, {Ian En Hsu} and Yijue Wang and Chang, {Sung En} and Bingbing Li and Shiyang Chen and Mimi Xie and Sanguthevar Rajasekaran and Hang Liu and Caiwen Ding",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 ; Conference date: 22-05-2022 Through 27-05-2022",

year = "2022",

doi = "10.18653/v1/2022.acl-long.16",

language = "English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

pages = "190--200",

editor = "Smaranda Muresan and Preslav Nakov and Aline Villavicencio",

booktitle = "ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)",

}

Huang, S, Xu, D, Yen, IEH, Wang, Y, Chang, SE, Li, B, Chen, S, Xie, M, Rajasekaran, S, Liu, H & Ding, C 2022, Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. in S Muresan, P Nakov & A Villavicencio (eds), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 190-200, 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ireland, 22/05/22. https://doi.org/10.18653/v1/2022.acl-long.16

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. / Huang, Shaoyi; Xu, Dongkuan; Yen, Ian En Hsu et al.
ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). ed. / Smaranda Muresan; Preslav Nakov; Aline Villavicencio. 2022. p. 190-200 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sparse Progressive Distillation

T2 - 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022

AU - Huang, Shaoyi

AU - Xu, Dongkuan

AU - Yen, Ian En Hsu

AU - Wang, Yijue

AU - Chang, Sung En

AU - Li, Bingbing

AU - Chen, Shiyang

AU - Xie, Mimi

AU - Rajasekaran, Sanguthevar

AU - Liu, Hang

AU - Ding, Caiwen

PY - 2022

Y1 - 2022

N2 - Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

AB - Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

UR - http://www.scopus.com/inward/record.url?scp=85138618135&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85138618135&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.acl-long.16

DO - 10.18653/v1/2022.acl-long.16

M3 - Conference contribution

AN - SCOPUS:85138618135

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 190

EP - 200

BT - ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

A2 - Muresan, Smaranda

A2 - Nakov, Preslav

A2 - Villavicencio, Aline

Y2 - 22 May 2022 through 27 May 2022

ER -

Huang S, Xu D, Yen IEH, Wang Y, Chang SE, Li B et al. Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. In Muresan S, Nakov P, Villavicencio A, editors, ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). 2022. p. 190-200. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2022.acl-long.16

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this