Autoencoders

Problem 2

We train a linear autoencoder to $D$-dimensional data. The autoencoder has a single $K$-dimensional hidden layer, there are no biases, and all activation functions are identity ($\sigma (x)=x$).

• Why is it usually impossible to get zero reconstruction error in this setting if $K<D$?
• Under which conditions is this possible?

Solution

Since all activation functions are identity maps, both the encoder and decoder in our autoencoder correspond to linear transformations: first from ${\mathbb{R}}^D$ to ${\mathbb{R}}^K$, and then from ${\mathbb{R}}^K$ back to ${\mathbb{R}}^D$.

Given that $K<D$, achieving zero reconstruction error would require that all input vectors are mapped uniquely through this transformation without loss of information. However, because the transformation from ${\mathbb{R}}^D$ to ${\mathbb{R}}^K$ is not an invertible (non-singular) mapping when $K<D$, it necessarily has a nontrivial kernel—meaning some nonzero vectors are mapped to zero. As a result, some information is inevitably lost, leading to nonzero reconstruction error in general.

Nevertheless, if the data lies entirely within a $K$-dimensional subspace of ${\mathbb{R}}^D$ (i.e., the row space of the transformation), then the autoencoder can perfectly reconstruct the inputs, achieving zero reconstruction error in practice.