Learning the Structure of Deep Architectures Using L1 Regularization

Praveen Kulkarni, Joaquin Zepeda, Frederic Jurie, Patrick Pérez and Louis Chevallier

Abstract

We present a method that formulates the selection of the structure of a deep architecture as a penalized, discriminative learning problem. Up to now, the structure of deep architectures has been fixed by hand, and only the weights are learned using discriminative learning. Our work is a first attempt towards a more formal method of deep structure selection. We consider architectures consisting only of fully-connected layers, and our approach relies on diagonal matrices inserted between subsequent layers. By including an L1 norm of the diagonal entries of said matrices as a regularization penalty, we force the diagonals to be sparse, accordingly selecting the effective number of rows (respectively, columns) of the corresponding layer's (next layer's) weights matrix. We carry out experiments on a standard dataset and show that our method succeeds in selecting the structure of deep architectures of multiple layers. One variant of our architecture results in a feature vector of size as little as $36$, while retaining very high image classification performance.

Session

Poster 1

Files

Extended Abstract (PDF, 128K)
Paper (PDF, 261K)

DOI

10.5244/C.29.23
https://dx.doi.org/10.5244/C.29.23

Citation

Bibtex

@inproceedings{BMVC2015_23,
	title={Learning the Structure of Deep Architectures Using L1 Regularization},
	author={Praveen Kulkarni and Joaquin Zepeda and Frederic Jurie and Patrick Pérez and Louis Chevallier},
	year={2015},
	month={September},
	pages={23.1-23.11},
	articleno={23},
	numpages={11},
	booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
	publisher={BMVA Press},
	editor={Xianghua Xie, Mark W. Jones, and Gary K. L. Tam},
	doi={10.5244/C.29.23},
	isbn={1-901725-53-7},
	url={https://dx.doi.org/10.5244/C.29.23}
}