Learning Activation Functions in Neural Networks
Pakshal Bohra (PhD Student, EPFL, with M. Unser)
Speaker: Pakshal Bohra (PhD Student, EPFL, with M. Unser)
Title: Learning Activation Functions in Neural Networks
Abstract: In this talk, we first present an efficient computational solution to train neural networks (NN) with free-form activation functions. To make the problem well-posed, we augment the cost functional of the NN by adding a suitable regularization term: the sum of the second-order total-variations of the learnable nonlinearities. The representer theorem for NNs states that the optimal activation functions are adaptive linear splines, which allows us to recast the problem as a parametric optimization. The challenging point is that the corresponding basis functions (ReLUs) are poorly conditioned and that the determination of their number and positioning is also part of the problem. We circumvent the difficulty by using an equivalent B-spline basis to encode the activation functions and by expressing the regularization as an l1-penalty. This results in the specification of parametric activation function modules that can be implemented and optimized efficiently on standard development platforms. We provide some experimental results that demonstrate the benefit of our approach. We then illustrate how the proposed module can be adapted to scenarios involving certain constraints on the NN. In particular, we present two such examples—Lipschitz-constrained NNs and the design of a NN-based convex regularizer for inverse problems.
More information: https://scholar.google.com/citations?hl=en&user=lJVrRV4AAAAJ&view_op=list_works&sortby=pubdate
Talk in room M7 101 (Campus Monod, ENS de Lyon)