Problem 2
We train a linear autoencoder to -dimensional data. The autoencoder has a single -dimensional hidden layer, there are no biases, and all activation functions are identity ().
- Why is it usually impossible to get zero reconstruction error in this setting if ?
- Under which conditions is this possible?
Solution
Since all activation functions are identity maps, both the encoder and decoder in our autoencoder correspond to linear transformations: first from to , and then from back to .
Given that , achieving zero reconstruction error would require that all input vectors are mapped uniquely through this transformation without loss of information. However, because the transformation from to is not an invertible (non-singular) mapping when , it necessarily has a nontrivial kernel—meaning some nonzero vectors are mapped to zero. As a result, some information is inevitably lost, leading to nonzero reconstruction error in general.
Nevertheless, if the data lies entirely within a -dimensional subspace of (i.e., the row space of the transformation), then the autoencoder can perfectly reconstruct the inputs, achieving zero reconstruction error in practice.