Layer 4 · Foundations

What Is Deep Learning

Why depth increases expressive power (universal approximation vs. depth efficiency) and the vanishing/exploding gradient problem quantified.

15 min read180 XP

1 layerneeds exponentially many units

3 layersmoderate width suffices

6 layerscompact representation

Illustrative: representing certain functions compactly can require exponentially fewer total parameters as depth increases (depth efficiency).

The universal approximation theorem states that a single hidden layer, if made wide enough, can approximate any continuous function on a compact domain to arbitrary precision: