Ternary Neural Networks for Resource-Efficient AI Applications

Abstract : —The computation and storage requirements for Deep Neural Networks (DNNs) are usually high. This issue limits their deployability on ubiquitous computing devices such as smart phones, wearables and autonomous drones. In this paper, we propose ternary neural networks (TNNs) in order to make deep learning more resource-efficient. We train these TNNs using a teacher-student approach based on a novel, layer-wise greedy methodology. Thanks to our two-stage training procedure, the teacher network is still able to use state-of-the-art methods such as dropout and batch normalization to increase accuracy and reduce training time. Using only ternary weights and activations, the student ternary network learns to mimic the behavior of its teacher network without using any multiplication. Unlike its {-1,1} binary counterparts, a ternary neural network inherently prunes the smaller weights by setting them to zero during training. This makes them sparser and thus more energy-efficient. We design a purpose-built hardware architecture for TNNs and implement it on FPGA and ASIC. We evaluate TNNs on several benchmark datasets and demonstrate up to 3.1× better energy efficiency with respect to the state of the art while also improving accuracy.
Complete list of metadatas

Cited literature [23 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01481478
Contributor : Hande Alemdar <>
Submitted on : Thursday, March 2, 2017 - 4:22:47 PM
Last modification on : Thursday, October 24, 2019 - 10:35:58 AM
Long-term archiving on : Wednesday, May 31, 2017 - 5:44:26 PM

File

paper_TNN_arxiv.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01481478, version 1
  • ARXIV : 1609.00222

Collections

Citation

Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, Frédéric Pétrot. Ternary Neural Networks for Resource-Efficient AI Applications. International Joint Conference on Neural Networks, May 2017, Anchorage, United States. ⟨hal-01481478⟩

Share

Metrics

Record views

413

Files downloads

1202