Linear Models & Optimization

Linearity in the parameters, independent errors, constant variance, roughly normal residuals. Heteroscedasticity or correlated errors leave coefficients unbiased but break the standard-error formula, so p-values need robust errors.

It is the change in the target when this feature grows by one unit and the other features stay fixed. Correlated features make coefficients unstable, even flipping signs, and different scales make magnitudes incomparable until standardized.

A linear fit is unbounded, so it predicts probabilities below zero or above one and is skewed by far-off points. Logistic regression squashes the score through a sigmoid into a log-odds model and trains on convex log-loss.

1.87 0.8

Common variants: SGD with momentum (accumulates past gradients), AdaGrad (per-parameter adaptive step), RMSProp (AdaGrad with a decaying average against step collapse), Adam (momentum + RMSProp) and Adan. They differ in how they adapt the step size and smooth the gradient direction.

0 1

On moderate data, with a direct least-squares solve via QR or SVD, never an explicit matrix inverse — that is what sklearn's LinearRegression does. When data outgrow memory, gradient methods take over: full-batch GD uses all samples per step, SGD one sample or a mini-batch and scales best.

2.0 1.0

Each one-unit rise in the feature, others held fixed, adds 0.7 to the log-odds of the positive class, so the odds multiply by exp(0.7) ≈ 2. How far the probability moves depends on where you are on the sigmoid.

0.02 1

0.5 0.95

predict labels a row positive when predict_proba gives it more than 0.5. Lower that threshold on the probabilities and more rows are flagged — recall rises or holds, precision usually drops — with no retraining.

It assumes the features are independent within each class, which real features rarely are. It still classifies well since only the top class matters: double-counted evidence makes probabilities extreme but rarely changes the winner.

Training error keeps falling as the degree grows, while test error falls at first and then rises once the curve starts fitting noise. Pick the degree on validation data, and scale and regularize high-degree terms.

6

The model w0 + w1·x + w2·x² is non-linear in x but linear in its weights, because x² enters it as one more feature column. So ordinary least squares fits it the same way it fits a straight line.

Regularization fights overfitting by adding a weight-norm penalty to the optimization objective. For linear models: L1 (sum of absolute weights), L2 (sum of squared weights), and Elastic Net (a mix of L1 and L2). The penalty discourages overly large weights.

Plot residuals against predictions: a shapeless band around zero is healthy. A curve means the model missed a non-linear pattern; a funnel means the error spread grows with the prediction, and taking the log of the target often helps.

2.0 0.1

fit raises ValueError: it needs a 2-D X shaped (n_samples, n_features)

A generative model learns the joint distribution — class prior plus per-class feature distribution — and derives the posterior. A discriminative one fits the boundary directly. Naive Bayes hits its higher asymptotic error sooner and wins on tiny data.

Minimizing the squared error gives w = (XᵀX)⁻¹Xᵀy. It is avoided because inverting XᵀX is costly on many features, and XᵀX may be singular or ill-conditioned, so the inverse is impossible or numerically unstable. Adding λI fixes the conditioning; gradient methods are preferred.

A symptom of overfitting is very large weights. With huge weights, two near-identical inputs differing by epsilon can yield wildly different predictions — the function is unstable. Penalizing the norm shrinks weights toward zero, giving a smoother, lower-variance function that generalizes better.

Momentum keeps an exponentially-weighted sum of recent gradients and steps along it, damping oscillations. It helps most on elongated (ill-conditioned) level curves: the gradient points across the valley, so plain SGD zig-zags, while momentum averages the swing out and moves along the valley.

Geometrically L1's constraint is a diamond with corners on the axes; the loss level-set usually touches a corner, where a weight is 0. L2's is a circle with no corners. By gradient, L1's |∂R/∂w| is constant so descent reaches 0, while L2's shrinks as w→0 and weights stop short.

The margin is the band between the boundary and the nearest points, and training maximizes its width. Support vectors are the points on or violating it — only they define the solution. Large C narrows the margin, small C widens it.

The L1 part zeroes weights for selection, the L2 part shrinks smoothly and keeps the problem well-posed. On correlated features pure Lasso keeps one arbitrarily and drops the rest; the L2 term spreads weight across the whole group instead.

Under perfect separation the maximum-likelihood optimum does not exist — scaling the weights up pushes log-loss toward zero without reaching it, so they diverge and the solver stops only on its iteration cap. A penalty pins a finite solution.

No. The intercept only shifts the hyperplane to where the data sits; it is not a weight on a feature. Penalizing it pulls the offset toward zero and stops the model from positioning the hyperplane well, so quality gets worse. By convention the bias is excluded from the penalty.

The standard algorithm (Gauss–Jordan / LU) is cubic, O(n³). Strassen-style multiplication gives about O(n^2.81), and asymptotically faster ones exist but carry huge constants and practical constraints, so they are rare. This cubic cost is why closed-form OLS is avoided.