Autoencoder

An autoencoder is a neural network trained to reconstruct its input through a compressed intermediate representation called the latent space, learning efficient data encodings in an unsupervised manner. The architecture consists of an encoder that compresses input to a lower-dimensional latent representation and a decoder that reconstructs the original from this compression.

The network is trained to minimize reconstruction error, the difference between input and output. The bottleneck forces the network to learn the most important features for reconstruction, discarding noise and redundancy. The latent space captures the underlying structure of the data in a compact form.

Autoencoders have several applications: dimensionality reduction where the latent representation acts as a compressed feature vector; denoising where networks trained to reconstruct clean inputs from corrupted ones learn to remove noise; anomaly detection where examples that reconstruct poorly are flagged as anomalies; and pretraining where the encoder initializes other networks.

Sparse autoencoders add regularization that encourages the latent representation to be sparse, learning more interpretable features. Contractive autoencoders penalize sensitivity to input perturbations, learning representations resistant to small changes. Variational autoencoders add probabilistic structure to the latent space, enabling generation of new samples.

Understanding autoencoders provides intuition for representation learning and the information bottleneck principle.