TY - GEN
T1 - Practically Correct, Just-in-Time Shell Script Parallelization
AU - Kallas, Konstantinos
AU - Mustafa, Tammam
AU - Bielak, Jan
AU - Karnikis, Dimitris
AU - Dang, Thurston H.Y.
AU - Greenberg, Michael
AU - Vasilakis, Nikos
N1 - Publisher Copyright:
© 2022 by The USENIX Association. All rights reserved.
PY - 2022
Y1 - 2022
N2 - Recent shell-script parallelization systems enjoy mostly automated speedups by parallelizing scripts ahead-of-time. Unfortunately, such static parallelization is hampered by dynamic behavior pervasive in shell scripts-e.g., variable expansion and command substitution-which often requires reasoning about the current state of the shell and filesystem. We present a just-in-time (JIT) shell-script compiler, PASH-JIT, that intermixes evaluation and parallelization during a script's run-time execution. JIT parallelization collects runtime information about the system's state, but must not alter the behavior of the original script and must maintain minimal overhead. PASH-JIT addresses these challenges by (1) using a dynamic interposition framework, guided by a static preprocessing pass, (2) developing runtime support for transparently pausing and resuming shell execution; and (3) operating as a stateful server, communicating with the current shell by passing messages-all without requiring modifications to the system's underlying shell interpreter. When run on a wide variety of benchmarks, including the POSIX shell test suite, PASH-JIT (1) does not break scripts, even in cases that are likely to break shells in widespread use; and (2) offers significant speedups, whenever parallelization is possible. These results show that PASH-JIT can be used as a drop-in replacement for any non-interactive shell use, providing significant speedups without any risk of breakage.
AB - Recent shell-script parallelization systems enjoy mostly automated speedups by parallelizing scripts ahead-of-time. Unfortunately, such static parallelization is hampered by dynamic behavior pervasive in shell scripts-e.g., variable expansion and command substitution-which often requires reasoning about the current state of the shell and filesystem. We present a just-in-time (JIT) shell-script compiler, PASH-JIT, that intermixes evaluation and parallelization during a script's run-time execution. JIT parallelization collects runtime information about the system's state, but must not alter the behavior of the original script and must maintain minimal overhead. PASH-JIT addresses these challenges by (1) using a dynamic interposition framework, guided by a static preprocessing pass, (2) developing runtime support for transparently pausing and resuming shell execution; and (3) operating as a stateful server, communicating with the current shell by passing messages-all without requiring modifications to the system's underlying shell interpreter. When run on a wide variety of benchmarks, including the POSIX shell test suite, PASH-JIT (1) does not break scripts, even in cases that are likely to break shells in widespread use; and (2) offers significant speedups, whenever parallelization is possible. These results show that PASH-JIT can be used as a drop-in replacement for any non-interactive shell use, providing significant speedups without any risk of breakage.
UR - http://www.scopus.com/inward/record.url?scp=85141045193&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85141045193&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85141045193
T3 - Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022
SP - 769
EP - 785
BT - Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022
T2 - 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022
Y2 - 11 July 2022 through 13 July 2022
ER -