Abstract
Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping (LD) approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing LD methods greatly impact the dynamic model’s performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge distillation and LD in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic speech networks. Comprehensive experimentation utilizing well-known speech recognition methods, including conformer and WavLM, on three public benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by 9.32% and 2.25% for high and no dropping cases with 33.3% reduction in training time.
Citation
Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti
Distillation-Based Layer Dropping (DLD): Effective end-to-end Framework for Dynamic Speech Networks
Proceedings of the 51st IEEE International Conference on Acoustics, Speech, and Signal Processing, doi:https://doi.org/10.1109/ICASSP55912.2026.11462850, 2026.
BibTeX
@inproceedings{Hannan2026DLD,
title = {Distillation-Based Layer Dropping (DLD): Effective end-to-end Framework for Dynamic Speech Networks},
author = {Hannan, Abdul and Daniele Falavigna and Nawaz, Shah and Mubashir Noman and Schedl, Markus and Alessio Brutti},
booktitle = {Proceedings of the 51st IEEE International Conference on Acoustics, Speech, and Signal Processing},
publisher = {IEEE},
doi = {https://doi.org/10.1109/ICASSP55912.2026.11462850},
url = {https://ieeexplore.ieee.org/document/11464390/},
year = {2026}
}