Latent Intrinsics Emerge from Training to Relight

What?

• Purely data-driven method for image relighting.
• Learns to relight scenes by extracting latent representations of scene intrinsics (e.g., albedo) and lighting (extrinsics) from images.
• Link to the paper

Black-box view

• Input: An input image $I_s^{l_1}$ (scene $s$, lighting $l_1$) and a reference image $I_s^{l_2}$ (same scene $s$, target lighting $l_2$)
• Output: A relit image ${\tilde{I}}_s^{l_1\to l_2}$, which approximates $I_s^{l_1}$ but with the lighting from $I_s^{l_2}$.

Why?

Most state-of-the-art methods (e.g., RGB-X¹, IntrinsX²) approach the problem of inverse rendering in an explicit fashion. That is, given some input—usually a photo or a photo with text—they attempt to predict a PBR map consisting of albedo, roughness, metallic properties, and normals. This allows for lighting changes and even editing of material properties, followed by re-rendering the image. However, this approach has two major drawbacks:

• Lack of ground truth data
• Lack of end-to-end learning

First, regarding the lack of ground truth data: while such data does exist (otherwise there would be no deep learning methods for inverse rendering), most of it is not from real-world scenes, as labeling real-world scenes with accurate surface material properties is extremely challenging. Therefore, larger reverse rendering datasets are usually composed of synthetic images rendered from artist-created environments where intrinsic properties are explicitly defined. If the objective is to operate on real-life data, this introduces a domain mismatch problem, as the training data distribution is different from the downstream one, and a model will inevitably have trouble generalising.

Second, the lack of end-to-end learning. When we first predict the intrinsics, we introduce some error in that estimation. Then, when we re-render using a simplified lighting model (since simulating the full complexity of light is computationally infeasible), we introduce another source of error. These errors compound. In contrast, if we could perform relighting in a single pass, we might reduce the cumulative error and achieve better results.

How?

Architecture

The method uses an autoencoder architecture (U-Net) with shared encoder parameters. An encoder network (E) extracts latent intrinsic features and extrinsic features from input images. A decoder network (D) combines these latent representations to generate the relighted image.

Here is an architecture diagram from the paper:

Essentially, the shared encoder extracts scene properties (intrinsics $\{S_{s,i}^{l_1}{\}}_i$) and lighting information (extrinsics $L_s^{l_2}$) separately from both images. Then the decoder combines the intrinsics extracted from the first image and extrinsics from the second image ($D(\{S_s^{l_1}\},L_s^{l_2})$) to create the new relighted image ${\tilde{I}}_s^{l_1\to l_2}$ which should resemble the second image ($I_s^{l_2}$) since the images differ only in lighting conditions so extracted intrinsics should be identical.

Loss Function

The model is trained using a mix of loss functions: relighting loss, intrinsic loss, and extrinsic loss:

$$\mathcal{L}={\mathcal{L}}_{relight}+{\lambda }_{intrinsic}\cdot {\mathcal{L}}_{intrinsic}+{\lambda }_{extrinsic}\cdot {\mathcal{L}}_{extrinsic}$$

Relighting Loss

${\mathcal{L}}_{relight}$ measures the difference between the generated relit image (${\tilde{I}}_s^{l_1\to l_2}$) and the ground truth ($I_s^{l_2}$), and also includes a reconstruction term (${\tilde{I}}_s^{l_2\to l_2}$ vs $I_s^{l_2}$). Given our underlying assumption that all pairs of images in our training data differ only by lighting, we want to enforce that the relit image is close to the reference image from which we have taken the extrinsic component, as well as ensure that we can reconstruct the input image from its latent representation. As such, the relight loss is composed of various difference metrics: L2 pixel loss, SSIM (Structural Similarity Index), and L2 gradient loss, which quantify how far apart the decoded images are from their ground truth. All in all, the relighting loss is given by:

$${\mathcal{L}}_{\text{relight}}:={\mathcal{L}}_{\text{pixel}}\left({\tilde{I}}_s^{l_1\to l_2},I_s^{l_2}\right)+{\mathcal{L}}_{\text{pixel}}\left({\tilde{I}}_s^{l_2\to l_2},I_s^{l_2}\right)$$

Intrinsic Loss

${\mathcal{L}}_{intrinsic}$ encourages intrinsic codes for the same scene under different lighting conditions ($S_{s,i}^{l_1}$ and $S_{s,i}^{l_2}$) to be similar. It does so by including an L2 distance between the intrinsics encoded from the images. Furthermore, it includes a regularization term (${\mathcal{L}}_{reg}$), based on coding rate, to prevent collapse of the latent representation where all features would be highly correlated. Mathematically, it is given by:

$${\mathcal{L}}_{\text{intrinsic}}:=\sum _i{\Vert {\mathbf{S}}_{s,i}^{l_1}-{\mathbf{S}}_{s,i}^{l_2}\Vert }_2+1\text{e-}3\cdot {\mathcal{L}}_{\text{reg}}\left({\mathbf{S}}_{s,i}^{l_1}\right)$$

For more details, please refer to the Latent Intrinsics paper. Furthermore, if you would like to better understand why the specific regularization term they utilise works, refer to this paper.

Extrinsic Loss

Promotes diversity and uniform distribution in the extrinsic (lighting) codes $L_s^l$, also using the same ${\mathcal{L}}_{reg}$ regularizer as in ${\mathcal{L}}_{intrinsic}$. It is simply given by:

$${\mathcal{L}}_{\text{extrinsic}}:={\mathcal{L}}_{\text{reg}}\left({\mathbf{L}}_s^l\right)$$

Footnotes

1. Zeng et al., RGB↔X: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models, 2024, https://zheng95z.github.io/publications/rgbx24 ↩

2. Kocsis et al.,IntrinsiX: High-Quality PBR Generation using Image Priors, 2025 https://arxiv.org/pdf/2504.01008 ↩