Mathematical Depth Questions

Answer: The singular value decomposition A = UΣV¹ decomposes any matrix into orthogonal rotations (U, V) and scaling (Σ). It matters for ML because:

• Low-rank approximation: The Eckart-Young theorem guarantees that truncating the SVD to k singular values gives the best rank-k approximation in Frobenius and spectral norm. This is the foundation of PCA, LoRA (low-rank adaptation), and matrix compression.
• Condition number: The ratio of the largest to smallest singular value (condition number) determines how stable gradient descent is for that layer. Poorly conditioned weight matrices lead to vanishing/exploding gradients.
• Spectral normalization: Constraining the largest singular value of weight matrices stabilizes GAN training (spectral normalization) and controls the Lipschitz constant of the network.
• Pseudoinverse: The Moore-Penrose pseudoinverse via SVD gives the minimum-norm least squares solution, relevant to understanding implicit regularization in overparameterized networks.

Answer: Let s_i = exp(z_i) / sum_j exp(z_j). We need ds_i/dz_j.

Case 1 (i = j): Using the quotient rule, ds_i/dz_i = s_i(1 - s_i). This follows because d/dz_i [exp(z_i)/S] = [exp(z_i) * S - exp(z_i) * exp(z_i)] / S² = s_i - s_i² = s_i(1 - s_i).

Case 2 (i ≠ j): ds_i/dz_j = -s_i * s_j. The numerator exp(z_i) does not depend on z_j, so the derivative comes only from the denominator: d/dz_j [exp(z_i)/S] = -exp(z_i) * exp(z_j) / S² = -s_i * s_j.

Compact form: The Jacobian is J = diag(s) - ss¹, where s is the softmax output vector. This is a rank-1 update of a diagonal matrix, which explains why softmax gradients are computationally efficient.

Connection to ML: When combined with cross-entropy loss L = -log(s_y) for true class y, the gradient simplifies beautifully to dL/dz_i = s_i - 1(i=y), which is why softmax + cross-entropy is the standard classification setup.

Answer: The Hessian H of the loss function encodes the local curvature of the loss landscape. Its eigenvalues determine:

• Convexity: All eigenvalues positive means local convexity (a local minimum). Any negative eigenvalue means a saddle point. High-dimensional loss landscapes have exponentially many saddle points, which is why SGD's noise helps escape them.
• Condition number: The ratio of the largest to smallest positive eigenvalue determines how elongated the loss landscape is. A large condition number means gradient descent oscillates along high-curvature directions while making slow progress along low-curvature directions. This motivates adaptive methods like Adam.
• Learning rate selection: The maximum stable learning rate for gradient descent is 2/lambda_max, where lambda_max is the largest eigenvalue of the Hessian. If your learning rate exceeds this, optimization diverges.
• Sharpness and generalization: Recent work (SAM, sharpness-aware minimization) suggests that minima with smaller Hessian eigenvalues (flatter minima) tend to generalize better. This connects the Hessian spectrum to the generalization behavior of neural networks.

Answer: We want to maximize log p(x) but this requires integrating over the latent variable z: log p(x) = log integral p(x|z)p(z) dz, which is intractable.

Step 1: Introduce a variational distribution q(z|x) and use Jensen's inequality:

log p(x) = log integral p(x|z)p(z) dz = log integral [p(x|z)p(z)/q(z|x)] q(z|x) dz ≥ integral q(z|x) log[p(x|z)p(z)/q(z|x)] dz

Step 2: Expand the bound:

ELBO = E_q[log p(x|z)] - KL(q(z|x) || p(z))

Interpretation: The first term is the reconstruction quality (how well the decoder reconstructs x from sampled z). The second term is the regularization (how close the encoder's posterior is to the prior). The gap between log p(x) and the ELBO is exactly KL(q(z|x) || p(z|x)), which is always non-negative.

Connection to practice: The reparameterization trick z = mu + sigma * epsilon (epsilon ~ N(0,1)) makes the ELBO differentiable with respect to the encoder parameters, enabling end-to-end gradient-based training.

Answer: For a model f_hat trained on data D, predicting target y = f(x) + epsilon where epsilon ~ N(0, sigma²):

E[(y - f_hat(x))²] = Bias²(f_hat(x)) + Var(f_hat(x)) + sigma²

where Bias(f_hat(x)) = E[f_hat(x)] - f(x) and Var(f_hat(x)) = E[(f_hat(x) - E[f_hat(x)])²].

Derivation sketch: Add and subtract E[f_hat(x)] inside the squared error, expand, and use the fact that the noise epsilon is independent of f_hat. The cross term vanishes because E[epsilon] = 0.

Deep learning twist: Classical bias-variance tradeoff predicts that overparameterized models should overfit (high variance). But neural networks exhibit "double descent": beyond the interpolation threshold, test error decreases again as model size increases. This is explained by implicit regularization from SGD and the structure of the function space explored by gradient descent, which favors low-complexity solutions even in overparameterized regimes.

Answer: Adam maintains per-parameter adaptive learning rates using first moment (mean) and second moment (uncentered variance) estimates of the gradients:

m_t = beta_1 * m_{t-1} + (1-beta_1) * g_t (first moment estimate)

v_t = beta_2 * v_{t-1} + (1-beta_2) * g_t² (second moment estimate)

update = m_t / (sqrt(v_t) + epsilon)

Why this helps transformers:

• Parameter scale heterogeneity: Transformers have parameters at very different scales (attention weights, layer norms, embeddings). Adam's per-parameter scaling adapts to each parameter's gradient magnitude.
• Sparse gradients: Attention patterns create sparse gradient flows. Adam's momentum smooths these, while the second moment prevents infrequently updated parameters from getting too-large updates when they finally receive gradients.
• Loss landscape conditioning: Transformer loss landscapes are poorly conditioned (large condition number). SGD requires careful learning rate tuning per layer group. Adam's adaptive rates approximate second-order optimization (diagonal approximation of the inverse Hessian).

Caveat: For certain tasks (image classification with ResNets), SGD with momentum actually generalizes better than Adam. The advantage of Adam is primarily in training speed and stability, not necessarily in final generalization performance.

Answer: For non-convex objectives (like neural network training), SGD cannot guarantee convergence to a global minimum. The standard convergence result is:

Theorem: Under standard assumptions (L-smooth gradients, bounded variance sigma² of stochastic gradients, learning rate eta_t = O(1/sqrt(T))), SGD satisfies:

min_{t=1..T} E[||gradient f(x_t)||²] ≤ O(1/sqrt(T)) + O(sigma²/sqrt(T))

This means SGD converges to a stationary point (where the gradient is small) at rate O(1/sqrt(T)). It does not distinguish between local minima, saddle points, and global minima.

Why this is okay in practice: (1) Saddle points in high dimensions are unstable — SGD's noise naturally escapes them. (2) Recent work suggests that most local minima of neural network loss surfaces have loss values close to the global minimum (the "loss landscape is benign" hypothesis). (3) The stochasticity of mini-batch gradients acts as implicit regularization, guiding SGD toward flatter minima that generalize better.

Learning rate schedules: Cosine annealing, linear warmup, and step decay are all heuristics to balance exploration (high LR) with convergence (low LR). Warmup is particularly important for transformers because the adaptive optimizers' moment estimates are poorly calibrated at initialization.

Answer: For distributions p (true) and q (model):

H(p, q) = -sum p(x) log q(x) = H(p) + KL(p || q)

where H(p) = -sum p(x) log p(x) is the entropy of the true distribution and KL(p || q) = sum p(x) log[p(x)/q(x)] is the KL divergence.

Why this matters: Since H(p) is a constant (it depends only on the true data distribution, not our model), minimizing cross-entropy H(p, q) is equivalent to minimizing KL(p || q). This means training with cross-entropy loss is implicitly fitting our model distribution q to the true distribution p in terms of KL divergence.

Asymmetry of KL: KL(p || q) ≠ KL(q || p). Forward KL (what cross-entropy minimizes) is "mean-seeking" — q tries to cover all modes of p, potentially spreading probability mass too broadly. Reverse KL is "mode-seeking" — q concentrates on the dominant mode of p. This is why GANs (which approximately minimize a symmetric divergence) produce sharper images than VAEs (which minimize forward KL).

Bits-per-character/token: Cross-entropy measured in base-2 logarithms gives the average number of bits needed to encode data under the model distribution. This provides an interpretable evaluation metric for language models.

Answer: The Data Processing Inequality (DPI) states that for a Markov chain X → Y → Z: I(X; Z) ≤ I(X; Y), where I denotes mutual information. Processing data can only destroy information, never create it.

Implications for deep learning:

• Information bottleneck: Each layer of a deep network forms a Markov chain: X → h1 → h2 → ... → Y. By DPI, later layers cannot recover information lost by earlier layers. The information bottleneck theory (Tishby) suggests that optimal representations compress the input X while preserving information about the target Y.
• Representation learning: DPI explains why features at higher layers of a deep network are necessarily more compressed than features at lower layers. Good representations are those that compress away irrelevant information (low I(X; h)) while retaining predictive information (high I(h; Y)).
• Caveat: DPI applies to deterministic networks only approximately. Stochastic networks (dropout, noise injection) and the discrete nature of computation introduce complications. The "information bottleneck in deep learning" hypothesis is still debated.

Answer: Contrastive learning objectives (InfoNCE, SimCLR) can be understood as maximizing a lower bound on the mutual information between different views of the same data point.

The InfoNCE loss: L = -E[log(exp(f(x,x+)) / sum_{k} exp(f(x,x_k)))]

is a lower bound on I(X; X+) - log(N), where N is the number of negative samples. This was shown by Oord et al. (2018).

Intuition: By training the model to distinguish positive pairs (two views of the same image) from negative pairs (views of different images), we learn representations that capture the shared information between views while discarding view-specific noise.

Limitations of the MI interpretation: Tschannen et al. (2020) showed that the success of contrastive learning depends more on the encoder architecture and the specific choice of views than on how tightly the MI bound is optimized. Simply maximizing MI with a more powerful critic does not improve representation quality. This suggests MI maximization is a useful lens but not the complete explanation.