Stable Deep Learning: How RReLU Initialization Fixes Vanishing Gradients

Basel, Switzerland, Source:

Deep learning has transformed artificial intelligence, yet training deep neural networks remains a complex challenge plagued by issues like vanishing gradients and unstable optimization. A recent study published in Electronics (MDPI) introduces a novel weight initialization scheme specifically designed for Randomized Leaky Rectified Linear Units (RReLU), offering a significant leap forward in model trainability and convergence stability.

While standard activation functions like ReLU are computationally efficient, they often suffer from the "dying ReLU" problem, where neurons become permanently inactive. Alternatives such as LReLU and PReLU attempt to mitigate this, but RReLU stands out by introducing stochasticity. This approach randomly samples the negative slope during training to provide a regularization effect. However, previous initialization methods, such as Xavier and He initialization, were not tailored to account for this random slope distribution, leading to suboptimal performance in deep architectures.

The researchers derived a new variance-preserving initialization formula that explicitly incorporates the statistical properties of RReLU's stochastic slope. By analyzing both forward signal propagation and backward gradient flow, the proposed method ensures that signal variances remain stable across network layers. This theoretical advancement was rigorously tested on feedforward neural networks and deep Convolutional Neural Networks (CNNs) using benchmarks like CIFAR-10, CIFAR-100, and ImageNet.

Experimental results demonstrated that the proposed RReLU-specific initialization significantly outperformed traditional Xavier and He methods, particularly in very deep networks. For example, in 200-layer feedforward nets and 30-layer CNNs, the new scheme maintained stable optimization while Xavier initialization often failed to converge. Furthermore, a comprehensive comparison of rectified activation functions revealed that while PReLU achieved the highest accuracy on large-scale datasets like ImageNet, LReLU and RReLU showed competitive generalization on smaller datasets like CIFAR, especially when paired with their respective optimized initialization strategies.

This research underscores the critical importance of aligning weight initialization with the specific statistical characteristics of activation functions. By bridging the gap between theoretical derivation and practical application, this new scheme provides a robust foundation for training deeper, more complex neural networks without succumbing to optimization pitfalls. As AI models continue to grow in size and complexity, such precise initialization techniques will be essential for unlocking higher performance and reliability in image classification and beyond.