Computer Vision

Last Night Exam Preparation | All Topics at a Glance

1

Thresholding

Global, Adaptive, Otsu's Method | Bimodal Histogram | Variance

2

Morphology

Erosion, Dilation, Opening, Closing, Top-Hat | Structuring Elements

3

Sobel & Laplacian

Image Smoothing, Median Filter, 1st & 2nd Order Derivatives

4

Noise Models

Gaussian, Salt & Pepper, Poisson, Speckle | Mathematical Models

5

Noise Removal Filters

Mean, Median, Gaussian, Adaptive Filter | Math

6

Segmentation

Scenario-based | Mathematical Methods | Type Selection

7

Harris Corner Detection

M Matrix, Eigenvalues, Corner Response Function

8

Geometric Transformations

Translation, Rotation, Scaling, Shearing, Homogeneous Coordinates

9

Neural Network

Layers: Input, Hidden, Output | Activation Functions

10

Backpropagation

Chain Rule, Loss, Gradient Descent | Network Flow

11

Optimizers

SGD, Momentum, Adam | When to Use

12

Vision Transformer (ViT)

Patch Embedding, Self-Attention, Key Terms

13

CNN

Convolution, Pooling, FC Layers | Architecture Diagram

14

Vision Transformer Architecture

Layer-by-Layer Walkthrough | How It Works

← Back to Topics

Thresholding

Global Thresholding
Adaptive Thresholding
Otsu's Method
Edge Detection

Edge detection comparison (Wikimedia Commons CC)

Grayscale 45 120 180 210 90 30 200 195 140 60 230 215 T = 128 Binary Output 0 0 1 1 0 0 1 1 0 0 1 1 f(x,y) > 128 ?
Thresholding: Grayscale pixels → Binary output based on threshold T

Definition

Thresholding converts a grayscale image into a binary image (0 or 1) by comparing each pixel intensity to a threshold value \(T\).

Definition
$$ g(x,y) = \begin{cases} 1 & \text{if } f(x,y) > T \\ 0 & \text{if } f(x,y) \leq T \end{cases} $$

Where \(f(x,y)\) is the input pixel intensity and \(g(x,y)\) is the output binary pixel.

How it works:

  1. Convert image to grayscale (if color)
  2. Compute histogram of pixel intensities
  3. Select a threshold value \(T\)
  4. Apply the formula above to every pixel
Key: Global thresholding uses a single value of \(T\) for the entire image. Works only when lighting is uniform.

When to use:

  • Image has clearly separable foreground and background
  • Histogram shows two distinct peaks (bimodal)
  • Lighting is uniform across the image
Exam: If asked "Draw the binary output" — just apply the threshold to each pixel and draw 0/1 grid.

Adaptive (Local) Thresholding

Computes a different threshold \(T\) for each pixel based on its local neighbourhood.

$$ T(x,y) = \frac{1}{|N|}\sum_{(s,t) \in N_{xy}} f(s,t) - C $$

Where:

  • \(N_{xy}\) = neighbourhood of pixel \((x,y)\) (e.g., 15×15 window)
  • \(|N|\) = number of pixels in neighbourhood
  • \(C\) = constant subtracted (e.g., 5–15)

Variants:

MethodT Calculation
MeanAverage of neighbourhood
Weighted (Gaussian)Gaussian-weighted average (center pixels weigh more)
MedianMedian of neighbourhood values

Window Size Effect:

  • Small window (e.g., 3×3): Captures fine details, more noise sensitive
  • Large window (e.g., 31×31): Smoother result, may miss small objects
Advantage: Handles non-uniform illumination, shadows, and varying contrast across the image.
Disadvantage: Computationally expensive. Window size \(k\) is critical — too small = noisy, too large = lose detail.

Otsu's Method (Automatic Global)

Fully automatic global thresholding. Finds optimal \(T\) that maximizes between-class variance and minimizes within-class variance.

Exam Important

Prerequisites:

  • Histogram must be bimodal (two peaks with a valley between)
  • Two classes: foreground (dark) and background (light)

Algorithm:

  1. Compute histogram of image, normalize to get probabilities \(p_i\) for intensity \(i\)
  2. For each candidate threshold \(k = 0, 1, \ldots, L-1\):
$$ \omega_0(k) = \sum_{i=0}^{k} p_i \quad \text{(class 0 probability)} $$ $$ \omega_1(k) = \sum_{i=k+1}^{L-1} p_i = 1 - \omega_0(k) \quad \text{(class 1 probability)} $$ $$ \mu_0(k) = \frac{\sum_{i=0}^{k} i \cdot p_i}{\omega_0(k)} \quad \text{(class 0 mean)} $$ $$ \mu_1(k) = \frac{\sum_{i=k+1}^{L-1} i \cdot p_i}{\omega_1(k)} \quad \text{(class 1 mean)} $$ $$ \mu_T = \sum_{i=0}^{L-1} i \cdot p_i \quad \text{(global mean)} $$

Between-Class Variance:

$$ \sigma_B^2(k) = \omega_0(k)\left[\mu_0(k) - \mu_T\right]^2 + \omega_1(k)\left[\mu_1(k) - \mu_T\right]^2 $$

Simplified form:

$$ \sigma_B^2(k) = \omega_0(k) \cdot \omega_1(k) \cdot \left[\mu_0(k) - \mu_1(k)\right]^2 $$

Within-Class Variance:

$$ \sigma_W^2(k) = \omega_0(k)\sigma_0^2(k) + \omega_1(k)\sigma_1^2(k) $$

Total variance: \(\sigma_T^2 = \sigma_B^2 + \sigma_W^2\) (constant)

Optimal threshold \(T^*\): value of \(k\) that maximizes \(\sigma_B^2(k)\) (equivalently minimizes \(\sigma_W^2\)).

Exam: "What is Otsu's method?" = Automatic threshold selection for bimodal histograms by maximizing between-class variance. "When does it fail?" = Non-bimodal histograms, uneven illumination.
← Prev Next: Morphology →
← Back to Topics

Morphology

Erosion & Dilation
Opening, Closing & Gradient
Top-Hat Transform

Structuring Element (SE)

A small template/mask (e.g., 3×3 square, disk, cross) that slides over the binary image performing logical operations.

Common SEs: BOX(3,3), DISK(5), RING(5), CROSS(3), LINE(r,angle)

Morphological Dilation

Morphological Dilation: Expands foreground objects (Wikimedia Commons CC)

Original 3×3 SE Dilation (⊕) Erosion (⊖) ⊕ Dilation = expands white ⊖ Erosion = shrinks white
Morphological Dilation expands foreground; Erosion shrinks it

Dilation: \(I \oplus SE\)

Expands foreground objects, fills small gaps/holes.

$$ (I \oplus SE)(x,y) = \max_{(s,t) \in SE} \left\{ f(x+s, y+t) \right\} $$

For binary: pixel becomes 1 if any SE-position overlaps foreground (logical OR).

Example:

3×3 all-1 SE on a 3×3 block of 1s → output becomes 5×5 block of 1s (expanded by 1 pixel on each side).

Effect: Enlarges bright regions, connects nearby objects, fills small holes.

Erosion: \(I \ominus SE\)

Shrinks foreground objects, removes small/thin structures.

$$ (I \ominus SE)(x,y) = \min_{(s,t) \in SE} \left\{ f(x-s, y-t) \right\} $$

For binary: pixel becomes 1 only if all SE-positions fall on foreground (logical AND).

Example:

3×3 all-1 SE on a 3×3 block of 1s → output becomes a single central pixel (shrunk by 1 pixel on each side).

Effect: Removes small bright noise, breaks thin connections, strips boundary pixels.
Exam: Dilation = expands, Erosion = shrinks. Apply given SE to a small binary matrix manually.

Opening: \(I \circ SE = (I \ominus SE) \oplus SE\)

Erosion first, then Dilation.

$$ \text{Opening} = \text{Erosion} \rightarrow \text{Dilation} $$
  • Removes small bright noise (small objects)
  • Breaks thin connections between objects
  • Preserves large objects that fully contain the SE

Example: Open a 5×5 block with 3×3 SE

  1. Erode: 5×5 → 3×3 (border removed)
  2. Dilate: 3×3 → 5×5 (restored)

Result: large block preserved, any 1×1 or 2×2 isolated noise removed.

Closing: \(I \bullet SE = (I \oplus SE) \ominus SE\)

Dilation first, then Erosion.

$$ \text{Closing} = \text{Dilation} \rightarrow \text{Erosion} $$
  • Fills small dark holes inside objects
  • Fuses narrow gaps between objects
  • Smooths outer contours

Example: Close a ring (5×5 with 1-pixel hole) with 3×3 SE

  1. Dilate: hole filled (becomes solid)
  2. Erosion: restores original size, hole stays filled

Morphological Gradient

$$ G(I) = (I \oplus SE) - (I \ominus SE) $$

Result = difference between dilated and eroded = thickened boundary/edge of objects.

This is a simple edge detector for binary/grayscale images.

Exam: Opening = E then D (removes noise). Closing = D then E (fills holes). Gradient = dilate minus erode (edges).

Top-Hat Transform

White Top-Hat

$$ \text{WTH}(I) = I - (I \circ SE) $$

Original minus Opening

  • Extracts small bright details that were removed by opening
  • Useful for correcting uneven illumination

Black (Bottom) Top-Hat

$$ \text{BTH}(I) = (I \bullet SE) - I $$

Closing minus Original

  • Extracts small dark details (dark spots/valleys)
  • Finds small dark regions surrounded by bright areas
Memory trick: White Top-Hat = what was lost by opening (bright stuff). Black Top-Hat = what was gained by closing (dark stuff).
Exam: Top-hat used for uneven illumination correction. WTH removes illumination variation; BTH finds dark defects.
← Prev: Thresholding Next: Sobel & Laplacian →
← Back to Topics

Sobel & Laplacian Operators

Image Smoothing
Sobel Operator
Laplacian Operator

Image Smoothing (Noise Reduction)

Removes high-frequency noise by averaging/filtering. Smoothing = blurring the image.

Original Image Sobel Edge Detection

Left: Original | Right: Sobel edge detection (Wikimedia Commons CC)

Sobel Gx Kernel — Vertical Edge Detection Gx Kernel -1 0 1 -2 0 2 -1 0 1 Detects vertical edge changes Image Patch 50 60 20 180 200 40 210 220 70 Gx Result High = edge Low = no edge G = sqrt(Gx² + Gy²) strong weak weak strong strong med
Sobel Gx kernel: negative weights on left, positive on right — detects vertical intensity changes (horizontal edges)

1. Mean (Box) Filter

$$ g(x,y) = \frac{1}{mn}\sum_{(s,t) \in S_{xy}} f(s,t) $$

3×3 mean kernel: \(\frac{1}{9}\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}\)

Averages all pixels in neighbourhood. Blurs edges equally. Good for uniform noise.

2. Weighted (Gaussian) Filter

$$ G(x,y) = \frac{1}{2\pi\sigma^2} e^{-\frac{x^2+y^2}{2\sigma^2}} $$

3×3 Gaussian kernel (approx, \(\sigma \approx 0.85\)):

$$ \frac{1}{16}\begin{bmatrix}1&2&1\\2&4&2\\1&2&1\end{bmatrix} $$

Center pixels get more weight. Preserves edges better than mean filter.

3. Median Filter

$$ g(x,y) = \text{median}\{f(s,t) : (s,t) \in S_{xy}\} $$

Sorts all pixel values in neighbourhood and picks the middle value.

  • Best for salt & pepper noise (removes outliers without blurring edges)
  • Non-linear filter
  • 3×3 neighbourhood: take median of 9 values
Exam: Median filter is the best choice for salt & pepper noise. It preserves edges while removing impulse noise.

Sobel Operator (1st Order Derivative)

Detects edges by computing gradient magnitude and direction using two 3×3 kernels.

Exam Important

Horizontal Kernel \(G_x\)

$$ G_x = \begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix} $$

Detects vertical edges

Vertical Kernel \(G_y\)

$$ G_y = \begin{bmatrix}-1&-2&-1\\0&0&0\\1&2&1\end{bmatrix} $$

Detects horizontal edges

Note: Center row/column has weight 2 (Gaussian smoothing in perpendicular direction).

Gradient Magnitude:

$$ G = \sqrt{G_x^2 + G_y^2} \approx |G_x| + |G_y| $$

Gradient Direction:

$$ \theta = \arctan\left(\frac{G_y}{G_x}\right) $$ $$ \theta_{\text{edge}} = \theta + 90° $$

Edge direction is perpendicular to gradient direction.

Edge Strength Classification:

MagnitudeClassification
\(G > T\)Strong edge
\(T/2 < G \leq T\)Weak edge
\(G \leq T/2\)Non-edge
Exam: Given a 3×3 image patch, apply \(G_x\) and \(G_y\) kernels to find edge magnitude and direction. Remember center weight = 2.

Laplacian Operator (2nd Order Derivative)

Edge = zero-crossing of the Laplacian. Positive → peak, Negative → valley, sign change = edge.

4-Connected Kernel:

$$ \nabla^2 f = \begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix} $$

8-Connected Kernel:

$$ \nabla^2 f = \begin{bmatrix}-1&-1&-1\\-1&8&-1\\-1&-1&-1\end{bmatrix} $$

Zero-Crossing Detection:

After applying Laplacian, scan output. A zero-crossing occurs when:

  • Value changes from positive to negative (or vice versa)
  • At least one value ≥ threshold in magnitude

Problem: Noise Sensitivity

2nd derivative amplifies noise. Practical solution:

$$ \text{Pipeline: } f(x,y) \xrightarrow{\text{Gaussian}} \text{Smooth} \xrightarrow{\text{Laplacian}} \nabla^2(G * f) \xrightarrow{\text{Zero-Cross}} \text{Edges} $$

Laplacian of Gaussian (LoG): First smooth with Gaussian, then apply Laplacian.

$$ \text{LoG kernel} = \nabla^2 G(x,y) = -\frac{1}{\pi\sigma^4}\left(1 - \frac{x^2+y^2}{2\sigma^2}\right)e^{-\frac{x^2+y^2}{2\sigma^2}} $$
Exam: Laplacian zero-crossings = edges. Very noise sensitive → always smooth first (Gaussian) then Laplacian = LoG.
← Prev: Morphology Next: Noise Models →
← Back to Topics

Noise Models

Gaussian & Salt Pepper
Poisson & Speckle

Gaussian Noise

Additive noise. Each pixel value has random noise added from a Gaussian distribution. Caused by electronic circuit noise, sensor heat.

Salt and Pepper Noise Gaussian Noise

Left: Salt & Pepper noise | Right: Gaussian noise (Wikimedia Commons CC)

Common Noise Models Gaussian N(μ, σ²) Additive Salt & Pepper p(z)=p_p/2 Impulse Poisson P(k)=λ^k e^-λ/k! Photon count Speckle g = f + f·η Multiplicative
Four noise types: Gaussian (additive), Salt & Pepper (impulse), Poisson (photon counting), Speckle (multiplicative)
$$ g(x,y) = f(x,y) + \eta(x,y) $$ $$ \eta(x,y) \sim \mathcal{N}(\mu, \sigma^2) $$ $$ p(z) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(z-\mu)^2}{2\sigma^2}} $$

\(\mu\) = mean of noise, \(\sigma\) = standard deviation (higher = more noise). Typically \(\mu = 0\) for zero-mean noise.

Appearance: Overall grainy/fuzzy look. Every pixel slightly shifted.

Salt & Pepper (Impulse) Noise

Random pixels become either white (255, salt) or black (0, pepper). Caused by transmission errors, dead/hot pixels.

$$ g(x,y) = \begin{cases} 0 & \text{with probability } p_p/2 \\ f(x,y) & \text{with probability } 1-p_p \\ 255 & \text{with probability } p_p/2 \end{cases} $$

\(p_p\) = probability of noise (e.g., 0.1 means 10% pixels corrupted).

Appearance: Scattered white and black dots. Most pixels remain unchanged.
Exam: Best removed by Median filter (not mean filter, which smears them).

Poisson (Shot) Noise

Noise inherent to photon counting. Follows Poisson distribution. Common in low-light photography.

$$ P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!} $$

\(\lambda\) = average number of photons collected. Variance = mean = \(\lambda\).

When \(\lambda\) is large, Poisson approximates Gaussian with \(\sigma^2 = \lambda\).

Appearance: Grainy in dark regions (low light = fewer photons = more relative noise).

Speckle Noise

Multiplicative noise. Common in radar, ultrasound, SAR imaging.

$$ g(x,y) = f(x,y) + f(x,y) \cdot \eta(x,y) = f(x,y)(1 + \eta(x,y)) $$ $$ \eta(x,y) \sim \mathcal{N}(0, \sigma^2) $$

Noise is proportional to pixel intensity — brighter regions have more absolute noise.

Appearance: Grainy texture that varies with local brightness. Darker areas less affected.
Exam: Gaussian = additive. Speckle = multiplicative. Salt & Pepper = impulse. Poisson = photon-counting (variance = mean).
← Prev: Sobel & Laplacian Next: Noise Removal Filters →
← Back to Topics

Noise Removal Filters

Mean & Median
Gaussian Filter
Adaptive Filter

Mean (Arithmetic) Filter

Noise Removal Pipeline Noisy Image Salt & pepper noise 3×3 Mean Filter Averages 9 pixels Σ / 9 = mean Clean Image Filter Comparison Mean → blurs edges Median → best for S&P Gaussian → weighted avg Adaptive → edge-aware ĝ = (1/mn) Σ g(s,t) Mean: linear filter
Noise removal: Noisy image → Filter (e.g. 3×3 mean) → Clean image. Median best for salt & pepper.
$$ \hat{f}(x,y) = \frac{1}{mn}\sum_{(s,t)\in S_{xy}} g(s,t) $$

Replaces each pixel with the average of its \(m \times n\) neighbourhood. Linear. Blurs edges. Best for Gaussian noise.

Median Filter

$$ \hat{f}(x,y) = \text{median}_{(s,t)\in S_{xy}}\{g(s,t)\} $$

Non-linear. Sorts neighbourhood values, picks middle. Best for salt & pepper noise. Preserves edges better than mean.

Gaussian Smoothing Filter

$$ G_\sigma(x,y) = \frac{1}{2\pi\sigma^2} e^{-\frac{x^2+y^2}{2\sigma^2}} $$ $$ \hat{f} = G_\sigma * g = \sum_{(s,t)} G_\sigma(s,t) \cdot g(x-s, y-t) $$

\(\sigma\) controls smoothing: larger \(\sigma\) = more blur = more noise removed but edges also blurred.

Separable: 2D Gaussian = product of two 1D Gaussians (efficiency trick).

Adaptive (Local) Filter

Adjusts behaviour based on local statistics. Preserves edges while removing noise.

Adaptive Mean Filter:

$$ \hat{f}(x,y) = g(x,y) - \frac{\sigma^2_\eta}{\sigma^2_L}\left(g(x,y) - \mu_L\right) $$
  • \(g(x,y)\) = noisy pixel value
  • \(\mu_L\) = local mean of \(m \times n\) neighbourhood
  • \(\sigma^2_L\) = local variance of neighbourhood
  • \(\sigma^2_\eta\) = noise variance (estimated from image or given)

How it works:

  • In flat regions: \(\sigma^2_L \approx \sigma^2_\eta\), filter averages heavily → removes noise
  • In edge regions: \(\sigma^2_L \gg \sigma^2_\eta\), filter keeps original value → preserves edge

Adaptive Median Filter:

Works in levels. Increases window size until conditions met:

Level A: \(A_1 = z_{med} - z_{min}\), \(A_2 = z_{med} - z_{max}\). If \(A_1 > 0\) and \(A_2 < 0\), go to Level B. Else increase window size.

Level B: \(B_1 = z_{xy} - z_{min}\), \(B_2 = z_{xy} - z_{max}\). If \(B_1 > 0\) and \(B_2 < 0\), output \(z_{xy}\). Else output \(z_{med}\).

Where \(z_{min}, z_{max}, z_{med}\) are min, max, median of current window; \(z_{xy}\) is center pixel.

Advantage: Can remove salt & pepper noise that is denser than standard median can handle, while preserving more detail.
Exam: Adaptive filter: noise removal is proportional to local noise level. High local variance = edge = keep. Low local variance = flat = smooth.
← Prev: Noise Models Next: Segmentation →
← Back to Topics

Segmentation Scenario

Mathematical Methods
Scenario Selection

Image Segmentation

Partitioning an image into meaningful regions (objects vs background, or multiple objects).

Image Segmentation — Splitting Into Regions Original Image Noisy / mixed Threshold / Cluster Label pixels Segmented Object Regions separated Methods 1. Thresholding 2. Region Growing 3. Edge-based 4. K-means g(x,y) = class_k if f(x,y) ≤ T Pick based on scenario
Segmentation partitions an image into meaningful regions — separating objects from background or each other

1. Thresholding-based

$$ g(x,y) = \begin{cases} \text{class}_1 & \text{if } f(x,y) \leq T \\ \text{class}_2 & \text{if } f(x,y) > T \end{cases} $$

Simple, fast. Works when classes have distinct intensity ranges.

2. Region-based (Region Growing)

Start from seed pixels. Grow region by adding neighbouring pixels that satisfy a similarity criterion (intensity difference < threshold).

$$ \text{Add pixel } (x,y) \text{ if } |f(x,y) - \mu_R| < T $$

Where \(\mu_R\) = mean intensity of current region \(R\).

3. Edge-based

Detect edges first (Sobel, Canny). Then close gaps and fill enclosed regions. Use edge boundaries to define segments.

4. Clustering (K-means)

$$ \text{Assign pixel to cluster } k = \arg\min_j \| f(x,y) - \mu_j \|^2 $$

Iterate: update means \(\mu_j\), reassign pixels until convergence.

Which Segmentation to Apply?

ScenarioBest MethodWhy
Document with black text on whiteGlobal Thresholding / OtsuBimodal histogram, uniform lighting
Image with shadowsAdaptive ThresholdingNon-uniform illumination
Medical image (tumor on uniform tissue)Region GrowingSimilar intensity, connected region
Multiple colored objectsK-means ClusteringMultiple intensity/color classes
Object boundaries neededEdge-based (Canny)Sharp intensity changes at boundaries
Texture-based regionsTexture analysis + clusteringIntensity alone insufficient
Exam: Given a scenario, identify: (1) lighting condition, (2) number of classes, (3) noise level, (4) need for boundaries vs regions. Then pick the method.
← Prev: Noise Removal Filters Next: Harris Corner →
← Back to Topics

Harris Corner Detection

Mathematics
Theory & Algorithm

Harris Corner Detection

Corner = point where intensity changes significantly in all directions. Detects stable, repeatable feature points.

Exam Important
Harris Corner — Flat vs Edge vs Corner Flat Region No gradient in any direction λ₁ ≈ 0, λ₂ ≈ 0 Edge ← one strong gradient → Change along one direction only λ₁ ≫ 0, λ₂ ≈ 0 Corner two strong gradients crossing at this point λ₁ ≫ 0, λ₂ ≫ 0 R = det(M) − k·[trace(M)]² → high R = corner
Harris detector: flat = no eigenvalue, edge = one large eigenvalue, corner = both eigenvalues large → high corner response

Step 1: Shift Window & Measure Intensity Change

$$ E(u,v) = \sum_{(x,y) \in W} \left[ I(x+u, y+v) - I(x,y) \right]^2 $$

Where \(W\) is the window, \((u,v)\) is the shift.

Step 2: Taylor Expansion Approximation

$$ I(x+u,y+v) \approx I(x,y) + u\frac{\partial I}{\partial x} + v\frac{\partial I}{\partial y} $$ $$ E(u,v) \approx \begin{bmatrix}u & v\end{bmatrix} M \begin{bmatrix}u \\ v\end{bmatrix} $$ $$ M = \sum_{(x,y)\in W} \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix} $$

Where \(I_x = \frac{\partial I}{\partial x}\), \(I_y = \frac{\partial I}{\partial y}\).

Step 3: Eigenvalue Analysis

Eigenvalues of \(M\): \(\lambda_1, \lambda_2\)

CaseEigenvaluesShape
Flat region\(\lambda_1 \approx 0, \lambda_2 \approx 0\)No change in any direction
Edge\(\lambda_1 \gg \lambda_2 \approx 0\)Change along one direction only
Corner\(\lambda_1 \gg 0, \lambda_2 \gg 0\)Change in all directions

Step 4: Corner Response Function (CRF)

$$ R = \det(M) - k \cdot [\text{trace}(M)]^2 $$ $$ \det(M) = \lambda_1 \lambda_2 $$ $$ \text{trace}(M) = \lambda_1 + \lambda_2 $$
  • \(R > T\) (threshold) → corner
  • \(|R| \leq T\) → edge or flat
  • \(R < -T\) → local maximum (potential corner in some variants)

\(k\) = empirically chosen constant (typically 0.04 – 0.06).

Exam: Given eigenvalues, classify as flat/edge/corner. Given image patch with gradients, compute \(M\) matrix and \(R\).

Theory & Algorithm

Complete Algorithm:

  1. Compute \(I_x\) and \(I_y\) using Sobel operators
  2. Compute products: \(I_x^2\), \(I_y^2\), \(I_x I_y\) for each pixel
  3. Apply Gaussian smoothing to each product (sum over window)
  4. Compute \(R\) for each pixel using CRF formula
  5. Apply threshold: keep pixels where \(R > T\)
  6. Non-maximum suppression: In local neighbourhood, keep only the pixel with maximum \(R\) (remove multiple responses at same corner)

Properties of Harris Detector:

  • Rotation invariant: Corner detected regardless of image rotation (eigenvalues unchanged)
  • NOT scale invariant: A corner may not be detected at different scale
  • Fast to compute (uses only derivatives and Gaussian)
  • Produces many responses per corner → need non-max suppression
Improvement: Scale-invariant detection can be achieved by combining with scale-space (e.g., SIFT).
← Prev: Segmentation Next: Geometric Transformations →
← Back to Topics

Geometric Transformations

Scaling & Rotation
Translation & Shearing
Homogeneous & Composites

Scaling

Geometric Transformations — Rotation, Scaling, Translation Rotation (θ) R = [cos -sin; sin cos] det(R) = 1 Preserves area Scaling (sx, sy) 1x 1.5x S = [sx 0; 0 sy] sx,sy > 1 = stretch Uniform or non-uniform Translation (tx, ty) tx, ty x' = x + tx, y' = y + ty Needs homogeneous coords T = [1 0 tx; 0 1 ty; 0 0 1]
Rotation preserves area, Scaling changes size, Translation shifts position — all expressed as 3×3 matrices in homogeneous coordinates
$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

\(s_x > 1\) = stretch horizontally, \(s_x < 1\) = shrink. \(s_x = s_y\) = uniform scaling.

Rotation

Rotate point \((x,y)\) by angle \(\theta\) about the origin:

$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

Derivation:

  • Rotate by angle \(\theta\): \(x' = x\cos\theta - y\sin\theta\), \(y' = x\sin\theta + y\cos\theta\)
  • In matrix form: \(\mathbf{x'} = R_\theta \mathbf{x}\)

Properties: \(\det(R) = \cos^2\theta + \sin^2\theta = 1\) (preserves area), \(R^T R = I\) (orthogonal), \(R^{-1} = R_{-\theta}\).

Exam: Rotation matrix: cos in diagonal, -sin/sin in off-diagonal. \(\cos\theta\) on top-left, \(\sin\theta\) on bottom-left.

Translation

$$ x' = x + t_x, \quad y' = y + t_y $$

Shifts every point by \((t_x, t_y)\). Cannot be represented as a simple 2×2 matrix multiplication (requires homogeneous coordinates).

Shearing

X-Shear (horizontal):

$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} 1 & sh_x \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

\(x' = x + sh_x \cdot y\), \(y' = y\)

Y-Shear (vertical):

$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ sh_y & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

\(x' = x\), \(y' = sh_y \cdot x + y\)

Shearing slants the image. A square becomes a parallelogram.

Homogeneous Coordinates

Represent 2D point \((x,y)\) as 3D: \((x, y, 1)\). This allows translation to be expressed as matrix multiplication.

All transforms in 3×3 matrix form:

Translation:

$$ T = \begin{bmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{bmatrix} $$

Scaling:

$$ S = \begin{bmatrix} s_x & 0 & 0 \\ 0 & s_y & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Rotation:

$$ R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Shearing (X):

$$ Sh_x = \begin{bmatrix} 1 & sh_x & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Composite Transformations

Multiply matrices in order of application (right to left):

$$ \mathbf{x'} = T \cdot R \cdot S \cdot \mathbf{x} $$

Applied as: Scale first, then Rotate, then Translate.

Exam: Order matters! \(T \cdot R \neq R \cdot T\) in general. Always apply transformations right-to-left: rightmost first.
← Prev: Harris Corner Next: Neural Network →
← Back to Topics

Neural Network (Theory)

Definition of Layers
Activation Functions

Neural Network Definition

A neural network is a computational model inspired by biological neurons. It learns to map inputs to outputs by adjusting weights through training.

Input Layer Hidden Layer(s) Output Layer
Fully Connected Neural Network (3-3-3-2 architecture)
Input x₁ x₂ x₃ 3 nodes Hidden h₁ h₂ h₃ h₄ 4 nodes Output o₁ o₂ 2 nodes w₁₁ w₂₁ v₁ v₂ Forward
Hand-drawn 3-layer neural network: Input (3) → Hidden (4) → Output (2), with labeled weights

Types of Layers:

LayerRoleDetails
Input LayerReceives raw dataOne neuron per feature/pixel. No computation, just passes values forward.
Hidden Layer(s)Feature extractionApplies weights, bias, and activation function. Can have multiple layers (deep network). Each neuron computes: \(z = \sum w_i x_i + b\), then \(a = f(z)\).
Output LayerFinal predictionProduces the output. Activation depends on task: sigmoid (binary), softmax (multi-class), linear (regression).
Deep Learning = neural network with multiple hidden layers.

Activation Functions

Introduce non-linearity. Without them, stacked linear layers = single linear layer.

FunctionFormulaRangeUse
Sigmoid\(\sigma(z) = \frac{1}{1+e^{-z}}\)(0, 1)Binary classification output
Tanh\(\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\)(-1, 1)Hidden layers (zero-centered)
ReLU\(f(z) = \max(0, z)\)[0, ∞)Most common hidden layer
Leaky ReLU\(f(z) = \max(\alpha z, z)\), \(\alpha=0.01\)(-∞, ∞)Prevents dead neurons
Softmax\(S(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}\)[0, 1], sum=1Multi-class output
Exam: Why non-linearity? Because composition of linear functions is still linear. Without activation, network cannot learn complex patterns.
← Prev: Geometric Transformations Next: Backpropagation →
← Back to Topics

Backpropagation

Key Terms
Network Flow

Key Terms

Input x₁, x₂ Hidden h₁, h₂ Output ŷ Loss L(ŷ, y) True y forward pass → ← backward (chain rule ∂L/∂w) ∂L/∂v = (∂L/∂ŷ)(∂ŷ/∂h) ∂L/∂w = (∂L/∂h)(∂h/∂w)
Backpropagation: Forward pass (green) computes loss, backward pass (red dashed) propagates gradients via chain rule
TermDefinition
Loss Function \(L(\hat{y}, y)\)Measures how far predictions \(\hat{y}\) are from true labels \(y\). E.g., MSE: \(L = \frac{1}{N}\sum(\hat{y}-y)^2\), Cross-entropy: \(L = -\sum y \log \hat{y}\)
Weights \(w\)Learnable parameters connecting neurons. \(z = wx + b\).
Bias \(b\)Additional parameter that shifts the activation function.
Forward PassCompute output: input → weighted sum → activation → ... → output → loss
Backward PassCompute gradients of loss w.r.t. each weight using the chain rule.
Chain Rule\(\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}\)
Learning Rate \(\eta\)Step size for weight updates. Too large = diverge, too small = slow convergence.
Gradient Descent\(w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w}\)

Backpropagation with Network

Forward Pass (→) and Backward Pass (←) Input Hidden Output Loss x₁ x₂ h₁ h₂ ŷ L w₁₁ w₂₁ w₁₂ w₂₂ v₁ v₂ Backprop: Compute ∂L/∂w using chain rule through each layer Forward: x → h → ŷ → L
Feedforward network with labeled weights for backpropagation

Backpropagation Steps (for network above):

Forward Pass:

$$ h_j = f\left(\sum_i w_{ji} x_i + b_j\right), \quad \hat{y} = g\left(\sum_j v_j h_j + b_o\right) $$ $$ L = \frac{1}{2}(\hat{y} - y)^2 $$

Backward Pass (Chain Rule):

$$ \frac{\partial L}{\partial v_j} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial v_j} = (\hat{y} - y) \cdot g'(\cdot) \cdot h_j $$ $$ \frac{\partial L}{\partial w_{ji}} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_j} \cdot \frac{\partial h_j}{\partial w_{ji}} = (\hat{y}-y) \cdot g'(\cdot) \cdot v_j \cdot f'(\cdot) \cdot x_i $$

Weight Update:

$$ w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w} $$
Exam: Backprop = applying chain rule from output back to input. Compute error at output, propagate backwards, update each weight.
← Prev: Neural Network Next: Optimizers →
← Back to Topics

Optimizers

SGD & Momentum
Adam & When to Use

SGD (Stochastic Gradient Descent)

Loss Landscape: Optimizer Paths to Minimum Loss weight → min SGD oscillates Momentum smooth path Adam fastest convergence SGD Momentum Adam
Optimizer comparison on a loss landscape: SGD oscillates, Momentum smooths the path, Adam converges fastest
$$ w_{t+1} = w_t - \eta \cdot \nabla L(w_t) $$
  • Updates weights using gradient of loss w.r.t. a single sample or mini-batch
  • Fast but noisy updates → may oscillate around minimum
  • Learning rate \(\eta\) is fixed (or decayed manually)

SGD with Momentum

$$ v_t = \beta \cdot v_{t-1} + \nabla L(w_t) $$ $$ w_{t+1} = w_t - \eta \cdot v_t $$
  • \(\beta\) = momentum coefficient (typically 0.9)
  • Accumulates past gradients as "velocity"
  • Accelerates convergence, dampens oscillations
  • Helps escape shallow local minima

Adam (Adaptive Moment Estimation)

Combines Momentum + RMSProp. Maintains both first moment (mean) and second moment (variance) of gradients.

$$ m_t = \beta_1 m_{t-1} + (1-\beta_1)\nabla L(w_t) \quad \text{(1st moment - mean)} $$ $$ v_t = \beta_2 v_{t-1} + (1-\beta_2)(\nabla L(w_t))^2 \quad \text{(2nd moment - variance)} $$ $$ \hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t} \quad \text{(bias correction)} $$ $$ w_{t+1} = w_t - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} $$

Default: \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\).

When to Use Which?

OptimizerWhen to Use
SGDSimple problems, convex loss, when you want fine control. Good generalization.
SGD + MomentumMost deep learning tasks. Good default. Faster than plain SGD.
AdamFast prototyping, sparse gradients (NLP), non-stationary objectives. Default choice for most problems.
RMSPropRNNs, non-stationary targets. Adapts learning rate per parameter.
Exam: Adam = momentum + adaptive learning rate. Best general-purpose optimizer. SGD may generalize better but needs more tuning.
← Prev: Backpropagation Next: ViT →
← Back to Topics

Vision Transformer (ViT)

Theoretical Explanation
Key Terms

Vision Transformer (ViT)

ViT: Image → Patches → Transformer → Classification Image 224×224 split 16×16 patches N=196 Linear Projection P×P×C → D +pos Transformer Encoder L layers [CLS] Token extract Classifi- cation MLP head H×W×C flattened D-dim self-attention softmax
ViT pipeline: Split image into 16×16 patches, linearly project, add positional encoding, pass through Transformer encoder, use [CLS] token for classification

Core Idea:

Split an image into fixed-size patches, linearly embed them, add positional encodings, and feed to a standard Transformer encoder.

ViT Architecture Pipeline Image 224x224 Patch Embed 16x16 patches 196 patches + Position Encoding Learnable Transformer Encoder L layers MSA + MLP [CLS] Token Extract MLP Head Class Inside Transformer Encoder Layer (repeated L times): Multi-Head Self-Attn + Norm MLP + Norm Residual connections around each sub-layer
Vision Transformer (ViT) Architecture

Step-by-step:

  1. Patch Creation: Image \(H \times W \times C\) divided into \(N = \frac{H \times W}{P^2}\) patches of size \(P \times P\) (e.g., 16×16)
  2. Linear Projection: Each patch flattened to vector, projected to embedding dimension \(D\) via learned matrix \(W_E\)
  3. [CLS] Token: Prepend a learnable classification token
  4. Positional Encoding: Add learnable position embeddings to each patch token
  5. Transformer Encoder: L layers of Multi-Head Self-Attention + MLP with residual connections
  6. Classification: Output of [CLS] token fed to MLP head for final prediction

Key Terms

TermDefinition
Patch EmbeddingConvert image patches to vectors. Each \(P \times P \times 3\) patch → \(D\)-dim vector via linear projection.
Self-AttentionEach patch attends to all other patches. Computes relationships: \(\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\)
Multi-Head AttentionMultiple attention heads in parallel, each learning different relationships. Concatenated and projected.
[CLS] TokenSpecial learnable token prepended to sequence. Its final representation used for classification.
Positional EncodingLearnable embeddings added to patch tokens to encode spatial position (ViT uses learnable, not sinusoidal).
Layer NormApplied before (pre-norm) each sub-layer. Stabilizes training.
MLPTwo linear layers with GELU activation. Hidden dim = 4× embedding dim.
Exam: ViT splits image into patches, embeds them, adds positional info, runs through Transformer encoder, uses [CLS] token for classification.
← Prev: Optimizers Next: CNN →
← Back to Topics

Convolutional Neural Network (CNN)

Architecture & Figure
Component Explanation

CNN Architecture

CNN: Input → Conv → ReLU → Pool → FC → Output In 32×32 Input 3×3 kernel Conv ReLU max(0,x) 28×28 MaxPool FC Out softmax 10 classes 32×32×3 28×28×6 28×28×6 14×14×6 120 10 spatial dims shrink ↑ depth increases →
CNN architecture: Spatial dimensions shrink through conv/pool layers while feature depth increases, ending with FC for classification
Typical CNN Architecture (e.g., LeNet/AlexNet style) Input 32x32x3 Conv1 28x28x6 Pool1 14x14x6 Conv2 10x10x16 Pool2 5x5x16 Conv3 1x1x120 FC1 84 FC2 10 Output Softmax Conv + ReLU Max Pooling Fully Connected Classification
CNN: Convolution + Pooling layers (feature extraction) followed by FC layers (classification)

Convolutional layers extract features (edges, textures, shapes). Pooling layers reduce spatial dimensions. FC layers perform classification.

Component Explanation

1. Convolutional Layer

Slides kernels/filters over input. Each kernel detects specific features.

$$ \text{Output}(i,j) = \sum_m \sum_n I(i+m, j+n) \cdot K(m,n) + b $$
  • Stride: Step size of kernel movement (1 = every pixel)
  • Padding: Add zeros around border to control output size
  • Output size: \(\frac{W - K + 2P}{S} + 1\)
  • Multiple filters: Each filter produces one feature map. Stacked = depth.

2. Activation (ReLU)

$$ \text{ReLU}(x) = \max(0, x) $$

Applied element-wise after convolution. Introduces non-linearity.

3. Pooling Layer

Max Pooling: Takes maximum value in each window. Reduces spatial dimensions, provides translation invariance.

$$ \text{MaxPool}(i,j) = \max_{(m,n) \in W_{ij}} f(m,n) $$

Average Pooling: Takes mean of each window.

4. Fully Connected (FC) Layer

Flattens feature maps into 1D vector. Each neuron connected to all previous neurons. Final layers for classification.

$$ z = W \cdot \text{flatten}(\text{features}) + b, \quad \text{output} = \text{softmax}(z) $$

Feature Learning Hierarchy:

Layer DepthLearnsExample
Early layersLow-level featuresEdges, corners, colors
Middle layersMid-level featuresTextures, patterns, shapes
Deep layersHigh-level featuresObjects, faces, scenes
Exam: CNN = weight sharing (same kernel across image) + local connectivity + pooling = translation invariant, parameter efficient.
← Prev: ViT Next: ViT Layer Architecture →
← Back to Topics

Vision Transformer - Layer Architecture

Layer Architecture
How It Works

Vision Transformer Layer Architecture (Page 7)

Single Transformer Encoder Layer with Residual Connections Input x Layer Norm Multi-Head Self-Attn Add + Norm MLP GELU Add + Norm x' Residual Connection x + MSA(LN(x)) Residual Connection x + MLP(LN(x)) pre-norm 4D hidden
Single ViT encoder layer: Input → LayerNorm → MSA → Add+Norm → MLP → Add+Norm, with residual connections (yellow dashed) bypassing each sub-layer

Each Transformer encoder layer in ViT consists of two sub-layers with residual connections and layer normalization.

Single ViT Encoder Layer (Detailed) Input: Sequence of patch embeddings [x₁, x₂, ..., x₟, xₗ⁸] Multi-Head Self-Attention (MSA) Add & Norm (Residual + LayerNorm) MLP (GELU activation) Add & Norm (Residual + LayerNorm) Output: Same shape sequence of embeddings Residual Residual Inside MSA (per head): Q = XW⁷, K = XW⁸, V = XW⁻ Attn = softmax(QK⁳ / √dₗ)V Multi-head: concat heads then linear projection
Detailed single ViT encoder layer with residual connections

Mathematical Details:

Multi-Head Self-Attention:

$$ \text{head}_i = \text{Attention}(XW_i^Q, XW_i^K, XW_i^V) $$ $$ \text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$ $$ \text{MSA}(X) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O $$

Where \(d_k = D/h\) (embedding dim / number of heads).

MLP:

$$ \text{MLP}(x) = W_2 \cdot \text{GELU}(W_1 x + b_1) + b_2 $$

Hidden dimension = \(4D\) (4x embedding dim).

Residual + LayerNorm:

$$ x_{out} = \text{LayerNorm}(x + \text{SubLayer}(x)) $$

How It Works (End to End)

  1. Input Image (e.g., 224×224×3)
  2. Patch Splitting: Divide into 16×16 patches → 196 patches, each 16×16×3 = 768 dims
  3. Linear Projection: Each patch → \(D\)-dim embedding (e.g., \(D=768\))
  4. Add [CLS] token: Prepend learnable [CLS] token → sequence length becomes 197
  5. Add Position Embeddings: Add learnable positional embeddings to all tokens
  6. Pass through L Transformer Encoder Layers:
    • Each layer: MSA → Add+Norm → MLP → Add+Norm
    • Self-attention captures global relationships between all patches
    • MLP processes each patch independently with non-linearity
    • Residual connections preserve information
  7. Extract [CLS] output: Take the [CLS] token's final representation
  8. Classification Head: [CLS] output → MLP/Linear → class probabilities

Key Differences from CNN:

AspectCNNViT
Receptive fieldLocal (grows with depth)Global (from first layer)
Inductive biasTranslation invariance, localityNone (learns from data)
Data requirementLess data neededNeeds large dataset or pretraining
Spatial structurePreserved via poolingFlattened (positional encoding restores)
Exam: ViT = patches + positional encoding + Transformer encoder. No convolution. Self-attention enables global context from the start. Needs more data than CNN.
← Prev: CNN Next →