AI & Machine Learning

Aim

To fit a straight line y = w·x + b to a set of scattered 2-D points by minimising the mean-squared error using batch gradient descent, and to observe how the learning rate controls convergence of the loss curve.

Theory

Linear regression assumes the target is an affine function of the input plus noise. Training searches for the parameters (w, b) that minimise the mean-squared error (MSE) over all n samples. Because MSE is convex and differentiable, we can descend its gradient: at every step we move the parameters a small amount (the learning rate) in the direction of steepest decrease.

prediction y_hat = w·x + b

loss (MSE) L = (1/n) Σ (y_hat_i - y_i)^2

gradients dL/dw = (2/n) Σ (y_hat_i - y_i)·x_i , dL/db = (2/n) Σ (y_hat_i - y_i)

update w ← w - η·dL/dw , b ← b - η·dL/db

The coefficient of determination R² = 1 - SS_res/SS_tot reports the fraction of variance explained; R² = 1 is a perfect fit.

Procedure

Press Sample data to load a noisy linear cloud, or click anywhere on the plot to drop your own (x, y) points.
Set the learning rate with the slider (log scale).
Press Train to run gradient descent continuously, or Step to advance a few epochs at a time.
Watch the purple line rotate into place while the amber MSE curve falls each epoch.
Read the converged weight, bias and R²; try a learning rate that is too high to see the loss diverge.

Gradient descent · MSE loss

Click the plot to drop (x, y) points. Gradient descent fits y = w·x + b by minimising mean-squared error; the line and loss curve update every epoch.

Controls

Learn rate0.05

weight w

0.000

bias b

0.000

MSE loss

--

epoch

0

R² goodness of fit

--

References

Bishop, C. M. Pattern Recognition and Machine Learning, Springer 2006 — Ch. 3, Linear Models for Regression.
Hastie, Tibshirani & Friedman. The Elements of Statistical Learning, 2nd ed., Springer 2009 — Ch. 3.
Goodfellow, Bengio & Courville. Deep Learning, MIT Press 2016 — Ch. 4-5 on gradient-based optimisation.

Aim

To train a binary classifier by logistic regression — pushing a linear score through the sigmoid function and minimising binary cross-entropy with gradient descent — and to visualise the learned linear decision boundary.

Theory

Logistic regression models the probability that a point belongs to class 1 as the sigmoid of a linear score. The sigmoid squashes any real number into (0, 1). We fit the weights by minimising binary cross-entropy (log-loss); a convenient property is that its gradient has the same simple form as linear regression — the error (prediction minus label) times the input.

score z = w1·x1 + w2·x2 + b

sigmoid p = σ(z) = 1 / (1 + e^(-z))

loss (BCE) L = -(1/n) Σ [ y·ln(p) + (1-y)·ln(1-p) ]

gradient dL/dw_j = (1/n) Σ (p_i - y_i)·x_ij

The decision boundary is the line z = 0, i.e. where p = 0.5. It is always straight (linear) for this model.

Procedure

Choose the active class (blue or amber) and click the canvas to place training points, or press Sample data.
Set the learning rate and press Train; the sigmoid is fit by gradient descent on cross-entropy.
Watch the white decision line settle between the two clouds while the loss curve drops.
Read the converged weights, loss and training accuracy. Add an outlier and re-train to see the boundary shift.

Sigmoid + cross-entropy · gradient descent

Class 0 Class 1 White line = decision boundary (p = 0.5)

Click to place a point of the active class. Background shading is the predicted probability surface from a real forward pass.

Controls

Active

Learn rate0.50

w1

0.000

w2

0.000

BCE loss

--

accuracy

--

References

Bishop, C. M. Pattern Recognition and Machine Learning, Springer 2006 — Sec. 4.3, Probabilistic Discriminative Models.
Cox, D. R. "The regression analysis of binary sequences", J. Royal Statistical Society B, 1958.
Murphy, K. P. Machine Learning: A Probabilistic Perspective, MIT Press 2012 — Ch. 8.

Aim

To classify points by the majority vote of their k nearest neighbours under Euclidean distance, and to visualise how the decision regions and their smoothness change with k.

Theory

k-Nearest-Neighbours is a non-parametric, "lazy" learner: there is no training phase. To classify a query point we measure its distance to every stored example, take the k closest, and assign the label held by the majority of them. Small k gives jagged, high-variance boundaries that hug noise; large k smooths the regions but can blur genuine class structure (bias-variance trade-off).

distance d(p, q) = sqrt( (p1-q1)^2 + (p2-q2)^2 )

prediction y_hat = mode of labels among the k smallest d(p, q)

Cost is O(n) per query because every training point is examined; this is why k-NN scales poorly to huge datasets without spatial indexes.

Procedure

Select the active class and click to place training points, or press Sample data.
Adjust k with the slider and watch the shaded decision regions reclassify live.
Set k = 1 to see the Voronoi-like jagged boundary, then raise k to smooth it.
Place an isolated point and observe how a larger k makes the model ignore it.

k-Nearest-Neighbours decision regions

Class A Class B Shaded = predicted region (majority vote of k nearest)

Click to place a point of the active class. The grid is reclassified live by real Euclidean distance whenever you add a point or change k.

Controls

Active

Neighbours k5

class A pts

0

class B pts

0

Grid resolution is adaptive. Every cell is classified by majority vote over its k nearest training points.

References

Cover & Hart. "Nearest neighbor pattern classification", IEEE Trans. Information Theory, 1967.
Hastie, Tibshirani & Friedman. The Elements of Statistical Learning, 2nd ed. — Sec. 13.3.
Duda, Hart & Stork. Pattern Classification, 2nd ed., Wiley 2001 — Ch. 4.

Aim

To partition unlabelled 2-D points into k clusters using Lloyd's algorithm with k-means++ seeding, and to confirm that the within-cluster inertia decreases monotonically to convergence.

Theory

k-means is unsupervised: it seeks k centroids that minimise the total squared distance from each point to its nearest centroid (the inertia). Lloyd's algorithm alternates two steps — assign every point to its closest centroid, then update each centroid to the mean of its assigned points. Each step can only lower the inertia, so it converges, though only to a local optimum. k-means++ seeding spreads the initial centroids by distance weighting, which greatly improves the result.

inertia J = Σ_i || x_i - μ_c(i) ||^2

assign c(i) = argmin_k || x_i - μ_k ||^2

update μ_k = mean of all x_i with c(i) = k

Procedure

Press Re-seed blobs to generate fresh Gaussian clusters.
Set the number of clusters k and the point count.
Press Run Lloyd to animate, or Step to watch one assign-update iteration at a time.
Observe the inertia and "moved" distance drop each iteration until "converged" appears.
Re-run a few times with the same k to see different local optima from different seeds.

Lloyd's algorithm

Crosses are centroids; points recolour to their nearest centroid each iteration. Inertia is the total within-cluster squared distance — it only ever decreases.

Controls

Clusters k3

Points220

iteration

0

inertia

--

moved

--

status

idle

References

Lloyd, S. P. "Least squares quantization in PCM", IEEE Trans. Information Theory, 1982.
Arthur & Vassilvitskii. "k-means++: The advantages of careful seeding", SODA, 2007.
MacQueen, J. "Some methods for classification and analysis of multivariate observations", Berkeley Symp., 1967.

Aim

To build a binary decision tree on 2-D labelled points by recursively choosing axis-aligned splits that maximise the impurity decrease (Gini or entropy), and to draw the resulting rectangular decision regions.

Theory

A decision tree partitions feature space with a sequence of axis-aligned thresholds. At each node the algorithm scans every feature and every candidate threshold and picks the split that most reduces impurity. Impurity measures how mixed the labels are in a node; both Gini and entropy are zero for a pure node and maximal for a 50/50 mix. The information gain is the parent impurity minus the size-weighted average impurity of the two children. Splitting stops at a maximum depth or when a node is pure.

Gini G = 1 - Σ_k p_k^2

entropy H = - Σ_k p_k·log2(p_k)

gain IG = I(parent) - (n_L/n)·I(L) - (n_R/n)·I(R)

Procedure

Select the active class and click to place points, or press Sample data for an XOR-like layout.
Choose the impurity criterion (Gini or entropy) and the maximum depth.
Press Build tree; the algorithm recursively finds the best split and shades each leaf rectangle by its majority class.
Increase the depth to watch the boxy boundary carve finer regions and the training accuracy rise.

Recursive Gini / entropy splits

Class A Class B White lines = split thresholds; shading = leaf majority

Click to add points, then Build tree. Every split is chosen by an exhaustive scan for maximum impurity decrease.

Controls

Active

Criterion

Max depth3

leaves

--

depth used

--

accuracy

--

points

0

References

Breiman, Friedman, Olshen & Stone. Classification and Regression Trees (CART), Wadsworth 1984.
Quinlan, J. R. "Induction of decision trees", Machine Learning, 1986 (ID3).
Hastie, Tibshirani & Friedman. The Elements of Statistical Learning — Sec. 9.2.

Aim

To compute the principal components of a 2-D point cloud from its covariance matrix using power iteration, and to draw the principal axes together with the projection of every point onto the first component.

Theory

Principal Component Analysis finds the orthogonal directions of greatest variance in the data. After centering the points, we form the 2x2 covariance matrix; its eigenvectors are the principal axes and its eigenvalues are the variance captured along each. We extract the top eigenvector by power iteration — repeatedly multiplying a random vector by the covariance matrix and renormalising — then deflate to obtain the second. Projecting each point onto the first component reduces it from two dimensions to one while preserving as much spread as possible.

center x' = x - mean(x)

covariance C = (1/n) X'^T X' (2x2)

power iter v ← C v / || C v || (repeat → top eigenvector)

explained ratio = λ1 / (λ1 + λ2)

Procedure

Press Sample data for an elongated tilted cloud, or click to add your own points.
Press Run PCA; the covariance matrix is formed and power iteration converges on the principal axes.
The long cyan arrow is PC1 (most variance), the short purple arrow is PC2; faint lines drop each point onto PC1.
Read the eigenvalues and the variance explained by PC1. Stretch the cloud along one direction to drive that ratio toward 1.

Covariance · power-iteration eigenvectors

PC1 axis PC2 axis projection onto PC1

Click to add points, then Run PCA. The principal axes are real eigenvectors of the covariance matrix found by power iteration.

Controls

λ1 (PC1)

--

λ2 (PC2)

--

explained

--

iters

--

PC1 angle: --. The two axes are orthogonal by construction.

References

Pearson, K. "On lines and planes of closest fit to systems of points in space", Phil. Mag., 1901.
Jolliffe, I. T. Principal Component Analysis, 2nd ed., Springer 2002.
Golub & Van Loan. Matrix Computations, 4th ed., JHU Press 2013 — power iteration.

Aim

To train a multilayer perceptron (2 → 8 → 8 → 1) with two tanh hidden layers and a sigmoid output by full backpropagation, and to watch it learn a non-linear decision boundary on XOR, circles, moons or spiral data.

Theory

A multilayer perceptron stacks linear maps and non-linear activations so it can represent decision boundaries no single line could. The forward pass computes activations layer by layer; backpropagation then applies the chain rule backwards to obtain the gradient of the loss with respect to every weight. For the sigmoid output with binary cross-entropy the output error simplifies neatly to (prediction - target). Xavier initialisation keeps the initial signal variance stable across layers.

forward a^(l) = f( W^(l) a^(l-1) + b^(l) ), f = tanh (hidden), σ (output)

output delta δ^(L) = a^(L) - y

hidden delta δ^(l) = (W^(l+1)^T δ^(l+1)) ⊙ (1 - a^(l)^2)

gradient dL/dW^(l) = δ^(l) (a^(l-1))^T

Procedure

Pick a dataset (XOR, two circles, two moons or spiral).
Set the learning rate and press Train to run backprop with full-batch gradient descent.
Watch the coloured prediction surface bend to separate the classes while the BCE loss falls and accuracy climbs.
Press Reset to re-initialise the weights; try the spiral, which needs many epochs and the right learning rate.

2 → 8 → 8 → 1 MLP · backpropagation

A real multilayer perceptron with two tanh hidden layers and a sigmoid output, trained by backprop + full-batch gradient descent. The background shading is the network's live prediction surface; the loss is binary cross-entropy.

Controls

Dataset

Learn rate0.20

epoch

0

BCE loss

--

accuracy

--

status

idle

Weights use Xavier initialisation. Decision boundary is recomputed on a coarse grid each frame through a real forward pass.

References

Rumelhart, Hinton & Williams. "Learning representations by back-propagating errors", Nature, 1986.
Glorot & Bengio. "Understanding the difficulty of training deep feedforward neural networks", AISTATS, 2010 (Xavier init).
Goodfellow, Bengio & Courville. Deep Learning, MIT Press 2016 — Ch. 6.

Aim

To compare three gradient-descent optimizers — plain SGD, SGD with momentum, and Adam — by watching their paths descend a 2-D loss surface from the same start, and to compare their loss-versus-iteration curves.

Theory

All three optimizers follow the negative gradient, but they differ in how they smooth and scale the step. Plain SGD takes a fixed multiple of the gradient and oscillates in narrow valleys. Momentum accumulates an exponentially-decayed velocity, damping oscillation and accelerating along consistent directions. Adam keeps running averages of both the gradient (first moment) and its square (second moment), bias-corrects them, and divides the step by the root of the second moment — giving an adaptive per-parameter learning rate.

SGD θ ← θ - η g

momentum v ← β v + g ; θ ← θ - η v

Adam m ← β1 m + (1-β1) g ; s ← β2 s + (1-β2) g^2

θ ← θ - η·m_hat / (sqrt(s_hat) + ε)

Procedure

Choose a loss surface (a stretched bowl, a saddle, or Rosenbrock's banana valley).
Click anywhere on the contour map to set a common starting point for all three optimizers.
Set the base learning rate and press Run; the three paths descend simultaneously.
Compare the trajectories on the contours and the loss curves below — note how Adam and momentum usually reach the minimum first.

SGD vs Momentum vs Adam on a 2-D surface

SGD Momentum Adam minimum

Click the contour map to set the start point. Each optimizer descends the real analytic gradient of the chosen surface.

Controls

Surface

Learn rate0.04

iteration

0

SGD loss

--

Mom loss

--

Adam loss

--

References

Kingma & Ba. "Adam: A method for stochastic optimization", ICLR, 2015.
Polyak, B. T. "Some methods of speeding up the convergence of iteration methods", USSR Comput. Math., 1964 (momentum).
Ruder, S. "An overview of gradient descent optimization algorithms", arXiv:1609.04747, 2016.

Aim

To apply 3x3 convolution kernels to a small hand-drawn grayscale image, to follow the resulting feature map through a ReLU non-linearity and 2x2 max-pooling, and to see how a convolutional layer builds a translation-tolerant feature hierarchy.

Theory

A convolutional layer slides a small learnable kernel over the image, computing at every position the weighted sum of the local patch. This shares weights across the image and detects the same local pattern (an edge, a blob) wherever it occurs. A ReLU then discards negative responses, keeping only evidence for the feature, and max-pooling downsamples each 2x2 block to its maximum — shrinking the map while keeping the strongest activations, which gives a little translation invariance. Stacking these stages forms a hierarchy: edges, then parts, then objects.

convolution S(i,j) = Σ(u=-1..1) Σ(v=-1..1) K(u,v) · I(i+u, j+v)

ReLU R(i,j) = max(0, S(i,j))

max-pool 2x2 P(i,j) = max( R(2i,2j), R(2i+1,2j), R(2i,2j+1), R(2i+1,2j+1) )

Sobel-x K = [ -1 0 +1 ; -2 0 +2 ; -1 0 +1 ] (vertical-edge detector)

The output map of a valid (no-padding) convolution shrinks by one pixel on every side; each pooling stage halves the resolution.

Procedure

Draw a digit or shape on the left grid by dragging the mouse, or press Sample digit to load a built-in numeral.
Pick a kernel (Sobel edge, Gaussian blur, sharpen, or a custom 3x3 you type in).
Watch the feature map update live as the kernel convolves the input.
Toggle ReLU and max-pool to see the rectified and downsampled stages of the hierarchy.
Read the output map size; note how valid convolution and pooling both shrink it.

Convolution feature hierarchy

Input (draw here) Conv map ReLU Pooled

Drag on the leftmost grid to paint. Every other panel is recomputed by a real 3x3 convolution, ReLU and 2x2 max-pool — no precomputed images.

Controls

Kernel

Stage

Pooling

input

16x16

conv map

14x14

pooled

7x7

max act

--

References

LeCun, Bottou, Bengio & Haffner. "Gradient-based learning applied to document recognition", Proc. IEEE, 1998 (LeNet).
Krizhevsky, Sutskever & Hinton. "ImageNet classification with deep convolutional neural networks", NeurIPS, 2012 (AlexNet).
Goodfellow, Bengio & Courville. Deep Learning, MIT Press 2016 — Ch. 9, Convolutional Networks.

Aim

To fit a polynomial of user-chosen degree to noisy data using ridge (L2-regularized) least squares solved in closed form by the normal equations, to observe under- and over-fitting through train versus validation error, and to estimate the true generalization error by k-fold cross-validation.

Theory

Polynomial regression builds a design matrix whose columns are powers of x, then solves a linear least-squares problem for the coefficients. Ridge regression adds an L2 penalty lambda on the coefficient sizes; this shrinks them, trading a little extra bias for a large drop in variance and taming the wild oscillations of high-degree fits. The solution has a closed form — the regularized normal equations — which we solve here by hand-rolled Gaussian elimination. A low-degree model underfits (high bias); a high-degree, unregularized model overfits (high variance). k-fold cross-validation rotates each fold as the validation set and averages the held-out error to estimate generalization without a separate test set.

design X = [ 1, x, x^2, ... , x^d ] (one row per sample)

ridge cost J = || X w - y ||^2 + λ || w ||^2

normal eqn w = ( X^T X + λ I )^(-1) X^T y

k-fold CV CV = (1/k) Σ(f=1..k) MSE( fold f held out )

The penalty is conventionally not applied to the intercept (the first I entry is set to 0).

Procedure

Press Sample data to draw noisy points from a hidden smooth curve, or click the plot to add your own.
Set the polynomial degree and the ridge penalty lambda with the sliders.
The fit, the train MSE and a held-out validation MSE update live; raise the degree with lambda = 0 to watch the curve oscillate and the validation error climb (overfitting).
Increase lambda to shrink the coefficients and smooth the fit back down.
Press Run k-fold CV to report the cross-validated error for the current settings.

Polynomial ridge fit · normal equations

Click the plot to add points. The fit solves the regularized normal equations by real Gaussian elimination; the lower chart sweeps train vs validation MSE across degrees.

Controls

Degree3

Lambda0

Folds k5

train MSE

--

valid MSE

--

CV MSE

--

||w||

--

Regime: --

References

Hoerl & Kennard. "Ridge regression: biased estimation for nonorthogonal problems", Technometrics, 1970.
Hastie, Tibshirani & Friedman. The Elements of Statistical Learning, 2nd ed., Springer 2009 — Sec. 3.4 (ridge) & Ch. 7 (model assessment, CV).
Bishop, C. M. Pattern Recognition and Machine Learning, Springer 2006 — Sec. 1.3 & 3.2, the bias-variance decomposition.

Aim

To train a softmax (multinomial logistic regression) classifier on the real 150-row Iris dataset with a train/test split, to report test accuracy, a confusion matrix and per-class precision and recall, and to visualize the learned decision regions over any two chosen features.

Theory

Softmax regression generalizes logistic regression to several classes: each class has its own weight vector, the scores are exponentiated and normalized into a probability distribution, and training minimizes the multi-class cross-entropy by gradient descent. We standardize the four features (zero mean, unit variance) so the gradient steps are well scaled. The data is split into train and test partitions; we fit on train only and report metrics on the unseen test set. The confusion matrix C has C[t][p] equal to the number of test samples whose true class is t and predicted class is p; from it precision and recall follow per class.

softmax p_k = e^(z_k) / Σ_j e^(z_j) , z_k = w_k · x + b_k

loss (CE) L = -(1/n) Σ_i ln p_(y_i)(x_i)

gradient dL/dz_k = p_k - 1[k = y]

precision P_k = C[k][k] / Σ_t C[t][k]

recall R_k = C[k][k] / Σ_p C[k][p]

Procedure

Choose the two features to plot for the decision-region view (e.g. petal length vs petal width).
Set the train/test split fraction and the learning rate.
Press Train; softmax regression fits all four features by gradient descent on the training split.
Read the test accuracy, the 3x3 confusion matrix and the per-class precision and recall.
Inspect the shaded decision regions over the two selected features; petal measurements separate the species almost perfectly, sepal measurements much less so.

Iris · softmax classifier · decision regions

setosa versicolor virginica filled = test · ring = train

All 150 Iris rows are embedded in the script. The shaded regions come from a real forward pass of the trained softmax model over the two selected features (the other two held at their mean).

Controls

X axis

Y axis

Test split0.30

Learn rate0.32

test acc

--

epoch

0

Confusion matrix (test)

Per-class precision / recall appear after training.

References

Fisher, R. A. "The use of multiple measurements in taxonomic problems", Annals of Eugenics, 1936 (the Iris dataset).
Bishop, C. M. Pattern Recognition and Machine Learning, Springer 2006 — Sec. 4.3.4, multiclass logistic regression (softmax).
Powers, D. M. W. "Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation", J. Machine Learning Tech., 2011.

Linear regression via gradient descent

Aim

Theory

Procedure

Gradient descent · MSE loss

Controls

R² goodness of fit

References

Aim

Theory

Procedure

Sigmoid + cross-entropy · gradient descent

Controls

References

Aim

Theory

Procedure

k-Nearest-Neighbours decision regions

Controls

References

Aim

Theory

Procedure

Lloyd's algorithm

Controls

References

Aim

Theory

Procedure

Recursive Gini / entropy splits

Controls

References

Aim

Theory

Procedure

Covariance · power-iteration eigenvectors

Controls

References

Aim

Theory

Procedure

2 → 8 → 8 → 1 MLP · backpropagation

Controls

References

Aim

Theory

Procedure

SGD vs Momentum vs Adam on a 2-D surface

Controls

References

Aim

Theory

Procedure

Convolution feature hierarchy

Controls

References

Aim

Theory

Procedure

Polynomial ridge fit · normal equations

Controls

References

Aim

Theory

Procedure

Iris · softmax classifier · decision regions

Controls

Confusion matrix (test)

References