Thresholding

Global Thresholding

Adaptive Thresholding

Otsu's Method

Thresholding converts grayscale to binary: compare each pixel to threshold T and output 0 or 1

Thresholding: Grayscale pixels → Binary output based on threshold T

Definition of Thresholding (Ch-8)

Thresholding converts a grayscale image into a binary image (0 or 1) by comparing each pixel intensity to a threshold value $T$.

Definition

$$
g(x,y) = \begin{cases} 1 & \text{if } f(x,y) > T \\ 0 & \text{if } f(x,y) \leq T \end{cases}
$$

Where $f(x,y)$ is the input pixel intensity and $g(x,y)$ is the output binary pixel.

How Thresholding Works (Ch-8):

Convert image to grayscale (if color)
Compute histogram of pixel intensities
Select a threshold value $T$
Apply the formula above to every pixel

Key: Global thresholding uses a single value of $T$ for the entire image. Works only when lighting is uniform.

When to use:

Image has clearly separable foreground and background
Histogram shows two distinct peaks (bimodal)
Lighting is uniform across the image

Exam: If asked "Draw the binary output" — just apply the threshold to each pixel and draw 0/1 grid.

Adaptive thresholding computes a different T for each pixel based on its local neighbourhood, handling shadows and non-uniform lighting

Adaptive (Local) Thresholding

Computes a different threshold $T$ for each pixel based on its local neighbourhood.

$$
T(x,y) = \frac{1}{|N|}\sum_{(s,t) \in N_{xy}} f(s,t) - C
$$

Where:

$N_{xy}$ = neighbourhood of pixel $(x,y)$ (e.g., 15×15 window)
$|N|$ = number of pixels in neighbourhood
$C$ = constant subtracted (e.g., 5–15)

Variants:

Method	T Calculation
Mean	Average of neighbourhood
Weighted (Gaussian)	Gaussian-weighted average (center pixels weigh more)
Median	Median of neighbourhood values

Window Size Effect:

Small window (e.g., 3×3): Captures fine details, more noise sensitive
Large window (e.g., 31×31): Smoother result, may miss small objects

Advantage: Handles non-uniform illumination, shadows, and varying contrast across the image.

Disadvantage: Computationally expensive. Window size $k$ is critical — too small = noisy, too large = lose detail.

Otsu's method finds the threshold that maximizes between-class variance in a bimodal histogram

Otsu's Method (Automatic Global)

Fully automatic global thresholding. Finds optimal $T$ that maximizes between-class variance and minimizes within-class variance.

Exam Important

Prerequisites:

Histogram must be bimodal (two peaks with a valley between)
Two classes: foreground (dark) and background (light)

Algorithm:

Compute histogram of image, normalize to get probabilities $p_i$ for intensity $i$
For each candidate threshold $k = 0, 1, \ldots, L-1$:

$$
\omega_0(k) = \sum_{i=0}^{k} p_i \quad \text{(class 0 probability)}
$$
$$
\omega_1(k) = \sum_{i=k+1}^{L-1} p_i = 1 - \omega_0(k) \quad \text{(class 1 probability)}
$$
$$
\mu_0(k) = \frac{\sum_{i=0}^{k} i \cdot p_i}{\omega_0(k)} \quad \text{(class 0 mean)}
$$
$$
\mu_1(k) = \frac{\sum_{i=k+1}^{L-1} i \cdot p_i}{\omega_1(k)} \quad \text{(class 1 mean)}
$$
$$
\mu_T = \sum_{i=0}^{L-1} i \cdot p_i \quad \text{(global mean)}
$$

Between-Class Variance:

$$ \sigma_B^2(k) = \omega_0(k)\left[\mu_0(k) - \mu_T\right]^2 + \omega_1(k)\left[\mu_1(k) - \mu_T\right]^2 $$

Simplified form:

$$ \sigma_B^2(k) = \omega_0(k) \cdot \omega_1(k) \cdot \left[\mu_0(k) - \mu_1(k)\right]^2 $$

Within-Class Variance:

$$
\sigma_W^2(k) = \omega_0(k)\sigma_0^2(k) + \omega_1(k)\sigma_1^2(k)
$$

Total variance: $\sigma_T^2 = \sigma_B^2 + \sigma_W^2$ (constant)

Optimal threshold $T^*$: value of $k$ that maximizes $\sigma_B^2(k)$ (equivalently minimizes $\sigma_W^2$).

Exam: "What is Otsu's method?" = Automatic threshold selection for bimodal histograms by maximizing between-class variance. "When does it fail?" = Non-bimodal histograms, uneven illumination.

← Prev Next: Morphology →

← Back to Topics

Morphology

Erosion & Dilation

Opening, Closing & Gradient

Top-Hat Transform

Structuring Element (SE)

A small template/mask (e.g., 3×3 square, disk, cross) that slides over the binary image performing logical operations.

Common SEs: BOX(3,3), DISK(5), RING(5), CROSS(3), LINE(r,angle)

Dilation with a 3×3 all-ones kernel expands each foreground pixel to a 3×3 block

Morphological Dilation expands foreground; Erosion shrinks it

Dilation: $I \oplus SE$

Expands foreground objects, fills small gaps/holes.

$$ (I \oplus SE)(x,y) = \max_{(s,t) \in SE} \left\{ f(x+s, y+t) \right\} $$

For binary: pixel becomes 1 if any SE-position overlaps foreground (logical OR).

Example:

3×3 all-1 SE on a 3×3 block of 1s → output becomes 5×5 block of 1s (expanded by 1 pixel on each side).

Effect: Enlarges bright regions, connects nearby objects, fills small holes.

Erosion: $I \ominus SE$

Shrinks foreground objects, removes small/thin structures.

$$ (I \ominus SE)(x,y) = \min_{(s,t) \in SE} \left\{ f(x-s, y-t) \right\} $$

For binary: pixel becomes 1 only if all SE-positions fall on foreground (logical AND).

Example:

3×3 all-1 SE on a 3×3 block of 1s → output becomes a single central pixel (shrunk by 1 pixel on each side).

Effect: Removes small bright noise, breaks thin connections, strips boundary pixels.

Exam: Dilation = expands, Erosion = shrinks. Apply given SE to a small binary matrix manually.

Opening removes small bright noise (E then D); Closing fills small dark holes (D then E)

Opening: $I \circ SE = (I \ominus SE) \oplus SE$

Erosion first, then Dilation.

$$
\text{Opening} = \text{Erosion} \rightarrow \text{Dilation}
$$

Removes small bright noise (small objects)
Breaks thin connections between objects
Preserves large objects that fully contain the SE

Example: Open a 5×5 block with 3×3 SE

Erode: 5×5 → 3×3 (border removed)
Dilate: 3×3 → 5×5 (restored)

Result: large block preserved, any 1×1 or 2×2 isolated noise removed.

Closing: $I \bullet SE = (I \oplus SE) \ominus SE$

Dilation first, then Erosion.

$$
\text{Closing} = \text{Dilation} \rightarrow \text{Erosion}
$$

Fills small dark holes inside objects
Fuses narrow gaps between objects
Smooths outer contours

Example: Close a ring (5×5 with 1-pixel hole) with 3×3 SE

Dilate: hole filled (becomes solid)
Erosion: restores original size, hole stays filled

Morphological Gradient

$$
G(I) = (I \oplus SE) - (I \ominus SE)
$$

Result = difference between dilated and eroded = thickened boundary/edge of objects.

This is a simple edge detector for binary/grayscale images.

Exam: Opening = E then D (removes noise). Closing = D then E (fills holes). Gradient = dilate minus erode (edges).

What Dilation Does (Ch-9)

Dilation EXPANDS white/foreground: every white pixel grows into its neighbours. Effects: objects get fatter, small gaps close, thin breaks join, tiny dark holes fill.

5 Morphology Maths (Ch-9)

1. Dilation: $(I \oplus SE)(x,y) = \max_{(s,t)\in SE} f(x{+}s, y{+}t)$ — OR logic, expands

2. Erosion: $(I \ominus SE)(x,y) = \min_{(s,t)\in SE} f(x{-}s, y{-}t)$ — AND logic, shrinks

3. Opening: $I \circ SE = (I \ominus SE) \oplus SE$ — E then D, removes small bright dots

4. Closing: $I \bullet SE = (I \oplus SE) \ominus SE$ — D then E, fills small dark holes

5. Gradient: $G = (I \oplus SE) - (I \ominus SE)$ — dilated minus eroded = boundary outline

White Top-Hat = Original minus Opening — extracts small bright details while correcting uneven illumination

Top-Hat Transform

White Top-Hat

$$ \text{WTH}(I) = I - (I \circ SE) $$

Original minus Opening

Extracts small bright details that were removed by opening
Useful for correcting uneven illumination

Black (Bottom) Top-Hat

$$ \text{BTH}(I) = (I \bullet SE) - I $$

Closing minus Original

Extracts small dark details (dark spots/valleys)
Finds small dark regions surrounded by bright areas

Memory trick: White Top-Hat = what was lost by opening (bright stuff). Black Top-Hat = what was gained by closing (dark stuff).

Exam: Top-hat used for uneven illumination correction. WTH removes illumination variation; BTH finds dark defects.

← Prev: Thresholding Next: Sobel & Laplacian →

← Back to Topics

Sobel & Laplacian Operators

Image Smoothing

Sobel Operator

Laplacian Operator

Image Smoothing (Noise Reduction)

Removes high-frequency noise by averaging/filtering. Smoothing = blurring the image.

Sobel Gx (vertical edges) + Gy (horizontal edges) → gradient magnitude G and direction θ

Sobel Gx kernel: negative weights on left, positive on right — detects vertical intensity changes (horizontal edges)

1. Mean (Box) Filter

$$ g(x,y) = \frac{1}{mn}\sum_{(s,t) \in S_{xy}} f(s,t) $$

3×3 mean kernel: $\frac{1}{9}\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}$

Averages all pixels in neighbourhood. Blurs edges equally. Good for uniform noise.

2. Weighted (Gaussian) Filter

$$ G(x,y) = \frac{1}{2\pi\sigma^2} e^{-\frac{x^2+y^2}{2\sigma^2}} $$

3×3 Gaussian kernel (approx, $\sigma \approx 0.85$):

$$ \frac{1}{16}\begin{bmatrix}1&2&1\\2&4&2\\1&2&1\end{bmatrix} $$

Center pixels get more weight. Preserves edges better than mean filter.

3. Median Filter

$$
g(x,y) = \text{median}\{f(s,t) : (s,t) \in S_{xy}\}
$$

Sorts all pixel values in neighbourhood and picks the middle value.

Best for salt & pepper noise (removes outliers without blurring edges)
Non-linear filter
3×3 neighbourhood: take median of 9 values

Exam: Median filter is the best choice for salt & pepper noise. It preserves edges while removing impulse noise.

4. Bilateral Filter

$$ g(x,y) = \frac{\sum_{(s,t)} f(s,t) \cdot w_s \cdot w_r}{\sum w_s \cdot w_r}, \quad w_s = e^{-\frac{d^2}{2\sigma_s^2}}, \; w_r = e^{-\frac{\Delta I^2}{2\sigma_r^2}} $$

Smooths flat areas but preserves edges: weights fall when intensity differs a lot.

First 4 Smoothing Filters (Ch-10)

#	Filter	Math idea	Best for
1	Mean (Box)	Average of neighbourhood	Uniform/Gaussian noise, simple
2	Gaussian	Weighted average, centre matters most	Gaussian noise, pre-Sobel smoothing
3	Median	Middle value after sorting	Salt & pepper (impulse) noise
4	Bilateral	Spatial + intensity weighting	Denoise while keeping edges sharp

Exam (Ch-10): "First 4" = Mean, Gaussian, Median, Bilateral. Median = only non-linear one. Always smooth BEFORE Sobel/Laplacian.

Sobel kernels: Gx detects vertical edges (horizontal gradient), Gy detects horizontal edges (vertical gradient). Center weight=2 adds Gaussian smoothing.

Sobel Operator (1st Order Derivative)

Detects edges by computing gradient magnitude and direction using two 3×3 kernels.

Exam Important

Horizontal Kernel $G_x$

$$ G_x = \begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix} $$

Detects vertical edges

Vertical Kernel $G_y$

$$ G_y = \begin{bmatrix}-1&-2&-1\\0&0&0\\1&2&1\end{bmatrix} $$

Detects horizontal edges

Note: Center row/column has weight 2 (Gaussian smoothing in perpendicular direction).

Gradient Magnitude:

$$
G = \sqrt{G_x^2 + G_y^2} \approx |G_x| + |G_y|
$$

Gradient Direction:

$$
\theta = \arctan\left(\frac{G_y}{G_x}\right)
$$
$$
\theta_{\text{edge}} = \theta + 90°
$$

Edge direction is perpendicular to gradient direction.

Edge Strength Classification:

Magnitude	Classification
$G > T$	Strong edge
$T/2 < G \leq T$	Weak edge
$G \leq T/2$	Non-edge

Exam: Given a 3×3 image patch, apply $G_x$ and $G_y$ kernels to find edge magnitude and direction. Remember center weight = 2.

Laplacian kernels: 4-connected uses 4 orthogonal neighbours, 8-connected uses all 8. Zero-crossings in output = edges.

Laplacian Operator (2nd Order Derivative)

Edge = zero-crossing of the Laplacian. Positive → peak, Negative → valley, sign change = edge.

4-Connected Kernel:

$$ \nabla^2 f = \begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix} $$

8-Connected Kernel:

$$ \nabla^2 f = \begin{bmatrix}-1&-1&-1\\-1&8&-1\\-1&-1&-1\end{bmatrix} $$

Zero-Crossing Detection:

After applying Laplacian, scan output. A zero-crossing occurs when:

Value changes from positive to negative (or vice versa)
At least one value ≥ threshold in magnitude

Problem: Noise Sensitivity

2nd derivative amplifies noise. Practical solution:

$$ \text{Pipeline: } f(x,y) \xrightarrow{\text{Gaussian}} \text{Smooth} \xrightarrow{\text{Laplacian}} \nabla^2(G * f) \xrightarrow{\text{Zero-Cross}} \text{Edges} $$

Laplacian of Gaussian (LoG): First smooth with Gaussian, then apply Laplacian.

$$ \text{LoG kernel} = \nabla^2 G(x,y) = -\frac{1}{\pi\sigma^4}\left(1 - \frac{x^2+y^2}{2\sigma^2}\right)e^{-\frac{x^2+y^2}{2\sigma^2}} $$

Exam: Laplacian zero-crossings = edges. Very noise sensitive → always smooth first (Gaussian) then Laplacian = LoG.

← Prev: Morphology Next: Noise Models →

← Back to Topics

Noise Models

Gaussian & Salt Pepper

Poisson & Speckle

Gaussian Noise

Additive noise. Each pixel value has random noise added from a Gaussian distribution. Caused by electronic circuit noise, sensor heat.

Salt & Pepper = impulse noise (random black/white dots); Gaussian = additive grainy noise (every pixel shifted)

Four noise types: Gaussian (additive), Salt & Pepper (impulse), Poisson (photon counting), Speckle (multiplicative)

$$
g(x,y) = f(x,y) + \eta(x,y)
$$
$$
\eta(x,y) \sim \mathcal{N}(\mu, \sigma^2)
$$
$$
p(z) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(z-\mu)^2}{2\sigma^2}}
$$

$\mu$ = mean of noise, $\sigma$ = standard deviation (higher = more noise). Typically $\mu = 0$ for zero-mean noise.

Appearance: Overall grainy/fuzzy look. Every pixel slightly shifted.

Salt & Pepper (Impulse) Noise

Random pixels become either white (255, salt) or black (0, pepper). Caused by transmission errors, dead/hot pixels.

$$
g(x,y) = \begin{cases} 0 & \text{with probability } p_p/2 \\ f(x,y) & \text{with probability } 1-p_p \\ 255 & \text{with probability } p_p/2 \end{cases}
$$

$p_p$ = probability of noise (e.g., 0.1 means 10% pixels corrupted).

Appearance: Scattered white and black dots. Most pixels remain unchanged.

Exam: Best removed by Median filter (not mean filter, which smears them).

Poisson noise follows photon counting statistics (Var=Mean); Speckle is multiplicative (brighter = noisier)

Poisson (Shot) Noise

Noise inherent to photon counting. Follows Poisson distribution. Common in low-light photography.

$$
P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}
$$

$\lambda$ = average number of photons collected. Variance = mean = $\lambda$.

When $\lambda$ is large, Poisson approximates Gaussian with $\sigma^2 = \lambda$.

Appearance: Grainy in dark regions (low light = fewer photons = more relative noise).

Speckle Noise

Multiplicative noise. Common in radar, ultrasound, SAR imaging.

$$
g(x,y) = f(x,y) + f(x,y) \cdot \eta(x,y) = f(x,y)(1 + \eta(x,y))
$$
$$
\eta(x,y) \sim \mathcal{N}(0, \sigma^2)
$$

Noise is proportional to pixel intensity — brighter regions have more absolute noise.

Appearance: Grainy texture that varies with local brightness. Darker areas less affected.

Exam: Gaussian = additive. Speckle = multiplicative. Salt & Pepper = impulse. Poisson = photon-counting (variance = mean).

← Prev: Sobel & Laplacian Next: Noise Removal Filters →

← Back to Topics

Noise Removal Filters

Mean & Median

Gaussian Filter

Adaptive Filter

Mean (Arithmetic) Filter

Noise removal: Noisy image → Filter (e.g. 3×3 mean) → Clean image. Median best for salt & pepper.

$$ \hat{f}(x,y) = \frac{1}{mn}\sum_{(s,t)\in S_{xy}} g(s,t) $$

Replaces each pixel with the average of its $m \times n$ neighbourhood. Linear. Blurs edges. Best for Gaussian noise.

Median Filter

$$
\hat{f}(x,y) = \text{median}_{(s,t)\in S_{xy}}\{g(s,t)\}
$$

Non-linear. Sorts neighbourhood values, picks middle. Best for salt & pepper noise. Preserves edges better than mean.

Gaussian kernel: center pixel has highest weight (0.44), weights decrease with distance from center

Gaussian Smoothing Filter

$$
G_\sigma(x,y) = \frac{1}{2\pi\sigma^2} e^{-\frac{x^2+y^2}{2\sigma^2}}
$$
$$
\hat{f} = G_\sigma * g = \sum_{(s,t)} G_\sigma(s,t) \cdot g(x-s, y-t)
$$

$\sigma$ controls smoothing: larger $\sigma$ = more blur = more noise removed but edges also blurred.

Separable: 2D Gaussian = product of two 1D Gaussians (efficiency trick).

Adaptive filter: in flat regions local variance ≈ noise variance → average; at edges local variance ≫ noise variance → preserve

Adaptive (Local) Filter

Adjusts behaviour based on local statistics. Preserves edges while removing noise.

Adaptive Mean Filter:

$$
\hat{f}(x,y) = g(x,y) - \frac{\sigma^2_\eta}{\sigma^2_L}\left(g(x,y) - \mu_L\right)
$$

$g(x,y)$ = noisy pixel value
$\mu_L$ = local mean of $m \times n$ neighbourhood
$\sigma^2_L$ = local variance of neighbourhood
$\sigma^2_\eta$ = noise variance (estimated from image or given)

How it works:

In flat regions: $\sigma^2_L \approx \sigma^2_\eta$, filter averages heavily → removes noise
In edge regions: $\sigma^2_L \gg \sigma^2_\eta$, filter keeps original value → preserves edge

Adaptive Median Filter:

Works in levels. Increases window size until conditions met:

Level A: $A_1 = z_{med} - z_{min}$, $A_2 = z_{med} - z_{max}$. If $A_1 > 0$ and $A_2 < 0$, go to Level B. Else increase window size.

Level B: $B_1 = z_{xy} - z_{min}$, $B_2 = z_{xy} - z_{max}$. If $B_1 > 0$ and $B_2 < 0$, output $z_{xy}$. Else output $z_{med}$.

Where $z_{min}, z_{max}, z_{med}$ are min, max, median of current window; $z_{xy}$ is center pixel.

Advantage: Can remove salt & pepper noise that is denser than standard median can handle, while preserving more detail.

Exam: Adaptive filter: noise removal is proportional to local noise level. High local variance = edge = keep. Low local variance = flat = smooth.

← Prev: Noise Models Next: Segmentation →

← Back to Topics

Segmentation Scenario

Mathematical Methods

Scenario Selection

Image Segmentation

Partitioning an image into meaningful regions (objects vs background, or multiple objects).

Segmentation partitions an image into meaningful regions — separating objects from background or each other

1. Thresholding-based

$$ g(x,y) = \begin{cases} \text{class}_1 & \text{if } f(x,y) \leq T \\ \text{class}_2 & \text{if } f(x,y) > T \end{cases} $$

Simple, fast. Works when classes have distinct intensity ranges.

2. Region-based (Region Growing)

Start from seed pixels. Grow region by adding neighbouring pixels that satisfy a similarity criterion (intensity difference < threshold).

$$ \text{Add pixel } (x,y) \text{ if } |f(x,y) - \mu_R| < T $$

Where $\mu_R$ = mean intensity of current region $R$.

3. Edge-based

Detect edges first (Sobel, Canny). Then close gaps and fill enclosed regions. Use edge boundaries to define segments.

4. Clustering (K-means)

$$ \text{Assign pixel to cluster } k = \arg\min_j \| f(x,y) - \mu_j \|^2 $$

Iterate: update means $\mu_j$, reassign pixels until convergence.

Decision flowchart: choose segmentation method based on lighting, edges, and region properties

Which Segmentation to Apply?

Scenario	Best Method	Why
Document with black text on white	Global Thresholding / Otsu	Bimodal histogram, uniform lighting
Image with shadows	Adaptive Thresholding	Non-uniform illumination
Medical image (tumor on uniform tissue)	Region Growing	Similar intensity, connected region
Multiple colored objects	K-means Clustering	Multiple intensity/color classes
Object boundaries needed	Edge-based (Canny)	Sharp intensity changes at boundaries
Texture-based regions	Texture analysis + clustering	Intensity alone insufficient

Colour Segmentation (Ch-12)

Segment in colour space (RGB / HSV) instead of grayscale when objects differ by colour, not intensity. HSV is preferred: Hue separates colour from lighting.

$$ \text{mask}(x,y) = \begin{cases} 1 & \text{if } H_{low} \le H(x,y) \le H_{high} \text{ and } S(x,y) > S_{min} \\ 0 & \text{otherwise} \end{cases} $$

Edge-Based Segmentation — 3 Types + Math (Ch-12)

Type	Operator	Math
1. Roberts (2×2, fastest)	$G_x = \begin{bmatrix}1&0\\0&-1\end{bmatrix}$, $G_y = \begin{bmatrix}0&1\\-1&0\end{bmatrix}$	$G = \sqrt{G_x^2+G_y^2}$
2. Prewitt (3×3, uniform weights)	$G_x = \begin{bmatrix}-1&0&1\\-1&0&1\\-1&0&1\end{bmatrix}$	Same magnitude formula; less noise-sensitive than Roberts
3. Sobel (3×3, centre ×2)	$G_x = \begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix}$	$G = \sqrt{G_x^2+G_y^2}$, $\theta = \tan^{-1}(G_y/G_x)$

Exam (Ch-12): Roberts = 2×2 (noise-sensitive). Prewitt = Sobel with all 1s (no centre emphasis). Sobel = best of the three for noisy images.

Exam: Given a scenario, identify: (1) lighting condition, (2) number of classes, (3) noise level, (4) need for boundaries vs regions. Then pick the method.

← Prev: Noise Removal Filters Next: Harris Corner →

← Back to Topics

Harris Corner Detection

Mathematics

Theory & Algorithm

Harris Corner Detection

Corner = point where intensity changes significantly in all directions. Detects stable, repeatable feature points.

Exam Important

Harris detector: flat = no eigenvalue, edge = one large eigenvalue, corner = both eigenvalues large → high corner response

Step 1: Shift Window & Measure Intensity Change

$$ E(u,v) = \sum_{(x,y) \in W} \left[ I(x+u, y+v) - I(x,y) \right]^2 $$

Where $W$ is the window, $(u,v)$ is the shift.

Step 2: Taylor Expansion Approximation

$$
I(x+u,y+v) \approx I(x,y) + u\frac{\partial I}{\partial x} + v\frac{\partial I}{\partial y}
$$
$$
E(u,v) \approx \begin{bmatrix}u & v\end{bmatrix} M \begin{bmatrix}u \\ v\end{bmatrix}
$$
$$
M = \sum_{(x,y)\in W} \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix}
$$

Where $I_x = \frac{\partial I}{\partial x}$, $I_y = \frac{\partial I}{\partial y}$.

Step 3: Eigenvalue Analysis

Eigenvalues of $M$: $\lambda_1, \lambda_2$

Case	Eigenvalues	Shape
Flat region	$\lambda_1 \approx 0, \lambda_2 \approx 0$	No change in any direction
Edge	$\lambda_1 \gg \lambda_2 \approx 0$	Change along one direction only
Corner	$\lambda_1 \gg 0, \lambda_2 \gg 0$	Change in all directions

Step 4: Corner Response Function (CRF)

$$
R = \det(M) - k \cdot [\text{trace}(M)]^2
$$
$$
\det(M) = \lambda_1 \lambda_2
$$
$$
\text{trace}(M) = \lambda_1 + \lambda_2
$$

$R > T$ (threshold) → corner
$|R| \leq T$ → edge or flat
$R < -T$ → local maximum (potential corner in some variants)

$k$ = empirically chosen constant (typically 0.04 – 0.06).

Exam: Given eigenvalues, classify as flat/edge/corner. Given image patch with gradients, compute $M$ matrix and $R$.

Harris detector: classify pixels by eigenvalues of M — flat (both small), edge (one large), corner (both large)

Theory & Algorithm

Complete Algorithm:

Compute $I_x$ and $I_y$ using Sobel operators
Compute products: $I_x^2$, $I_y^2$, $I_x I_y$ for each pixel
Apply Gaussian smoothing to each product (sum over window)
Compute $R$ for each pixel using CRF formula
Apply threshold: keep pixels where $R > T$
Non-maximum suppression: In local neighbourhood, keep only the pixel with maximum $R$ (remove multiple responses at same corner)

Properties of Harris Detector:

Rotation invariant: Corner detected regardless of image rotation (eigenvalues unchanged)
NOT scale invariant: A corner may not be detected at different scale
Fast to compute (uses only derivatives and Gaussian)
Produces many responses per corner → need non-max suppression

Bisection of Response Space (Ch-13)

The corner response $R$ bisects every pixel into 3 regions by threshold $T$:

$$
\text{Pixel} = \begin{cases} \text{Corner} & R > T \quad (R \text{ large positive}) \\ \text{Edge} & R < 0 \quad (R \text{ large negative}) \\ \text{Flat} & |R| \text{ small} \end{cases}
$$

Memory trick (Bisection): Positive-split = corner, Negative-split = edge, Near-zero = flat. Then non-max suppression keeps only local maxima.

Improvement: Scale-invariant detection can be achieved by combining with scale-space (e.g., SIFT).

← Prev: Segmentation Next: Geometric Transformations →

← Back to Topics

Geometric Transformations

Scaling & Rotation

Translation & Shearing

Homogeneous & Composites

Scaling

Rotation preserves area, Scaling changes size, Translation shifts position — all expressed as 3×3 matrices in homogeneous coordinates

$$
\begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}
$$

$s_x > 1$ = stretch horizontally, $s_x < 1$ = shrink. $s_x = s_y$ = uniform scaling.

Rotation

Rotate point $(x,y)$ by angle $\theta$ about the origin:

$$
\begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}
$$

Derivation:

Rotate by angle $\theta$: $x' = x\cos\theta - y\sin\theta$, $y' = x\sin\theta + y\cos\theta$
In matrix form: $\mathbf{x'} = R_\theta \mathbf{x}$

Properties: $\det(R) = \cos^2\theta + \sin^2\theta = 1$ (preserves area), $R^T R = I$ (orthogonal), $R^{-1} = R_{-\theta}$.

Exam: Rotation matrix: cos in diagonal, -sin/sin in off-diagonal. $\cos\theta$ on top-left, $\sin\theta$ on bottom-left.

Translation shifts points (not a 2×2 matrix); Shearing slants the image turning squares into parallelograms

Translation

$$
x' = x + t_x, \quad y' = y + t_y
$$

Shifts every point by $(t_x, t_y)$. Cannot be represented as a simple 2×2 matrix multiplication (requires homogeneous coordinates).

Shearing

X-Shear (horizontal):

$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} 1 & sh_x \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

$x' = x + sh_x \cdot y$, $y' = y$

Y-Shear (vertical):

$$ \begin{bmatrix} x' \\ y' \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ sh_y & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} $$

$x' = x$, $y' = sh_y \cdot x + y$

Shearing slants the image. A square becomes a parallelogram.

Homogeneous coordinates: promote 2D to 3D so all transforms (including translation) become 3×3 matrix multiplications

Homogeneous Coordinates

Represent 2D point $(x,y)$ as 3D: $(x, y, 1)$. This allows translation to be expressed as matrix multiplication.

All transforms in 3×3 matrix form:

Translation:

$$ T = \begin{bmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{bmatrix} $$

Scaling:

$$ S = \begin{bmatrix} s_x & 0 & 0 \\ 0 & s_y & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Rotation:

$$ R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Shearing (X):

$$ Sh_x = \begin{bmatrix} 1 & sh_x & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

5th type — Reflection / Mirroring (Ch-14):

$$ F_x = \begin{bmatrix} -1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}, \quad F_y = \begin{bmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Flips the image across an axis. $\det(F) = -1$ (orientation reversing) — unlike rotation with $\det = +1$.

5 Types at a Glance (Ch-14, 2D)

#	Type	What it does	2D matrix
1	Translation	Shift by $(t_x,t_y)$	Needs homogeneous 3×3 $T$
2	Rotation	Rotate by $\theta$ about origin	$\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}$
3	Scaling	Resize by $(s_x,s_y)$	$\begin{bmatrix}s_x&0\\0&s_y\end{bmatrix}$
4	Shearing	Slant; square → parallelogram	$\begin{bmatrix}1&sh_x\\sh_y&1\end{bmatrix}$
5	Reflection	Mirror across axis	$\begin{bmatrix}-1&0\\0&1\end{bmatrix}$ or $\begin{bmatrix}1&0\\0&-1\end{bmatrix}$

Exam (Ch-14): Only Translation CANNOT be written as 2×2 — that is why homogeneous 3×3 exists. Reflection = only type with determinant −1.

Composite Transformations

Multiply matrices in order of application (right to left):

$$ \mathbf{x'} = T \cdot R \cdot S \cdot \mathbf{x} $$

Applied as: Scale first, then Rotate, then Translate.

Exam: Order matters! $T \cdot R \neq R \cdot T$ in general. Always apply transformations right-to-left: rightmost first.

← Prev: Harris Corner Next: Neural Network →

← Back to Topics

Neural Network (Theory)

Definition of Layers

Activation Functions

Neural Network Definition

A neural network is a computational model inspired by biological neurons. It learns to map inputs to outputs by adjusting weights through training.

Fully Connected Neural Network (3-3-3-2 architecture)

Hand-drawn 3-layer neural network: Input (3) → Hidden (4) → Output (2), with labeled weights

Types of Layers:

Layer	Role	Details
Input Layer	Receives raw data	One neuron per feature/pixel. No computation, just passes values forward.
Hidden Layer(s)	Feature extraction	Applies weights, bias, and activation function. Can have multiple layers (deep network). Each neuron computes: $z = \sum w_i x_i + b$, then $a = f(z)$.
Output Layer	Final prediction	Produces the output. Activation depends on task: sigmoid (binary), softmax (multi-class), linear (regression).

Deep Learning = neural network with multiple hidden layers.

Activation Functions

Sigmoid: (0,1) centered at ½ | Tanh: (-1,+1) zero-centered | ReLU: max(0,z) most common

Introduce non-linearity. Without them, stacked linear layers = single linear layer.

Function	Formula	Range	Use
Sigmoid	$\sigma(z) = \frac{1}{1+e^{-z}}$	(0, 1)	Binary classification output
Tanh	$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$	(-1, 1)	Hidden layers (zero-centered)
ReLU	$f(z) = \max(0, z)$	[0, ∞)	Most common hidden layer
Leaky ReLU	$f(z) = \max(\alpha z, z)$, $\alpha=0.01$	(-∞, ∞)	Prevents dead neurons
Softmax	$S(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$	[0, 1], sum=1	Multi-class output

Exam: Why non-linearity? Because composition of linear functions is still linear. Without activation, network cannot learn complex patterns.

← Prev: Geometric Transformations Next: Backpropagation →

← Back to Topics

Backpropagation

Key Terms

Network Flow

Key Terms

Backpropagation: Forward pass (green) computes loss, backward pass (red dashed) propagates gradients via chain rule

Term	Definition
Loss Function $L(\hat{y}, y)$	Measures how far predictions $\hat{y}$ are from true labels $y$. E.g., MSE: $L = \frac{1}{N}\sum(\hat{y}-y)^2$, Cross-entropy: $L = -\sum y \log \hat{y}$
Weights $w$	Learnable parameters connecting neurons. $z = wx + b$.
Bias $b$	Additional parameter that shifts the activation function.
Forward Pass	Compute output: input → weighted sum → activation → ... → output → loss
Backward Pass	Compute gradients of loss w.r.t. each weight using the chain rule.
Chain Rule	$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}$
Learning Rate $\eta$	Step size for weight updates. Too large = diverge, too small = slow convergence.
Gradient Descent	$w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w}$

Backpropagation with Network

Forward pass (green) computes loss; backward pass (red dashed) propagates gradients via chain rule to update weights

Feedforward network with labeled weights for backpropagation

Backpropagation Steps (for network above):

Forward Pass:

$$ h_j = f\left(\sum_i w_{ji} x_i + b_j\right), \quad \hat{y} = g\left(\sum_j v_j h_j + b_o\right) $$ $$ L = \frac{1}{2}(\hat{y} - y)^2 $$

Backward Pass (Chain Rule):

$$
\frac{\partial L}{\partial v_j} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial v_j} = (\hat{y} - y) \cdot g'(\cdot) \cdot h_j
$$
$$
\frac{\partial L}{\partial w_{ji}} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h_j} \cdot \frac{\partial h_j}{\partial w_{ji}} = (\hat{y}-y) \cdot g'(\cdot) \cdot v_j \cdot f'(\cdot) \cdot x_i
$$

Weight Update:

$$ w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w} $$

Gradient Problems (Ch-18): Vanishing & Exploding Gradients

$$
\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \prod_{l} f'_l(\cdot) \cdot w_l \quad \text{(product over layers)}
$$

Problem	Cause	Symptom	Fix
Vanishing gradient	Many small derivatives multiplied (sigmoid/tanh saturate, $f' \le 0.25$)	Early layers stop learning	ReLU, BatchNorm, residual links, careful init
Exploding gradient	Large weights $\times$ large derivatives compound	Loss = NaN, weights blow up	Gradient clipping, smaller LR, BatchNorm

Exam (Ch-18): Vanishing = gradients → 0 in deep nets with sigmoid. Exploding = gradients → ∞. ReLU + BatchNorm + clipping are the standard answers.

Exam: Backprop = applying chain rule from output back to input. Compute error at output, propagate backwards, update each weight.

← Prev: Neural Network Next: Optimizers →

← Back to Topics

Optimizers

SGD & Momentum

Adam & When to Use

SGD (Stochastic Gradient Descent)

Optimizer comparison on a loss landscape: SGD oscillates, Momentum smooths the path, Adam converges fastest

$$
w_{t+1} = w_t - \eta \cdot \nabla L(w_t)
$$

Updates weights using gradient of loss w.r.t. a single sample or mini-batch
Fast but noisy updates → may oscillate around minimum
Learning rate $\eta$ is fixed (or decayed manually)

SGD with Momentum

$$
v_t = \beta \cdot v_{t-1} + \nabla L(w_t)
$$
$$
w_{t+1} = w_t - \eta \cdot v_t
$$

$\beta$ = momentum coefficient (typically 0.9)
Accumulates past gradients as "velocity"
Accelerates convergence, dampens oscillations
Helps escape shallow local minima

Adam (Adaptive Moment Estimation)

Adam combines momentum smoothing (left) with per-parameter adaptive learning rate scaling (right)

Combines Momentum + RMSProp. Maintains both first moment (mean) and second moment (variance) of gradients.

$$
m_t = \beta_1 m_{t-1} + (1-\beta_1)\nabla L(w_t) \quad \text{(1st moment - mean)}
$$
$$
v_t = \beta_2 v_{t-1} + (1-\beta_2)(\nabla L(w_t))^2 \quad \text{(2nd moment - variance)}
$$
$$
\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t} \quad \text{(bias correction)}
$$
$$
w_{t+1} = w_t - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}
$$

Default: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$.

When to Use Which?

Optimizer	When to Use
SGD	Simple problems, convex loss, when you want fine control. Good generalization.
SGD + Momentum	Most deep learning tasks. Good default. Faster than plain SGD.
Adam	Fast prototyping, sparse gradients (NLP), non-stationary objectives. Default choice for most problems.
RMSProp	RNNs, non-stationary targets. Adapts learning rate per parameter.

Learning Rate (Ch-19)

$\eta$ = step size in $w \leftarrow w - \eta \nabla L$. Too large = overshoot/diverge; too small = slow crawl or stuck. Schedules: step decay, exponential decay, cosine annealing.

Overfitting vs Underfitting (Ch-19)

	Overfitting	Underfitting
Meaning	Memorises training noise; train loss low, val loss high	Too simple; both train and val loss high
Looks like	Huge gap between train/val curves	Both curves flat and high
Fix	More data, augmentation, dropout, weight decay, early stopping	Bigger model, more epochs, higher LR, better features

Exam (Ch-19): SGD = fine control, may generalise better. Adam = fast default. Fix overfitting with dropout/early-stopping; fix underfitting with bigger model + train longer.

Exam: Adam = momentum + adaptive learning rate. Best general-purpose optimizer. SGD may generalize better but needs more tuning.

← Prev: Backpropagation Next: ViT →

← Back to Topics

Vision Transformer (ViT)

Theoretical Explanation

Key Terms

Vision Transformer (ViT)

ViT pipeline: Split image into 16×16 patches, linearly project, add positional encoding, pass through Transformer encoder, use [CLS] token for classification

Core Idea:

Split an image into fixed-size patches, linearly embed them, add positional encodings, and feed to a standard Transformer encoder.

Vision Transformer (ViT) Architecture

Step-by-step:

Patch Creation: Image $H \times W \times C$ divided into $N = \frac{H \times W}{P^2}$ patches of size $P \times P$ (e.g., 16×16)
Linear Projection: Each patch flattened to vector, projected to embedding dimension $D$ via learned matrix $W_E$
[CLS] Token: Prepend a learnable classification token
Positional Encoding: Add learnable position embeddings to each patch token
Transformer Encoder: L layers of Multi-Head Self-Attention + MLP with residual connections
Classification: Output of [CLS] token fed to MLP head for final prediction

Key Terms

Self-attention: Input → Q,K,V projections → scaled dot-product QKᵀ → softmax weights → weighted sum with V

Term	Definition
Patch Embedding	Convert image patches to vectors. Each $P \times P \times 3$ patch → $D$-dim vector via linear projection.
Self-Attention	Each patch attends to all other patches. Computes relationships: $\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$
Multi-Head Attention	Multiple attention heads in parallel, each learning different relationships. Concatenated and projected.
[CLS] Token	Special learnable token prepended to sequence. Its final representation used for classification.
Positional Encoding	Learnable embeddings added to patch tokens to encode spatial position (ViT uses learnable, not sinusoidal).
Layer Norm	Applied before (pre-norm) each sub-layer. Stabilizes training.
MLP	Two linear layers with GELU activation. Hidden dim = 4× embedding dim.

Exam: ViT splits image into patches, embeds them, adds positional info, runs through Transformer encoder, uses [CLS] token for classification.

← Prev: Optimizers Next: CNN →

← Back to Topics

Convolutional Neural Network (CNN)

Architecture & Figure

Component Explanation

CNN Architecture

CNN architecture: Spatial dimensions shrink through conv/pool layers while feature depth increases, ending with FC for classification

CNN: Convolution + Pooling layers (feature extraction) followed by FC layers (classification)

Convolutional layers extract features (edges, textures, shapes). Pooling layers reduce spatial dimensions. FC layers perform classification.

Component Explanation

Convolution: A 3×3 kernel slides over the input (red outline = current position), computing dot products to produce the output feature map

1. Convolutional Layer

Slides kernels/filters over input. Each kernel detects specific features.

$$ \text{Output}(i,j) = \sum_m \sum_n I(i+m, j+n) \cdot K(m,n) + b $$

Stride: Step size of kernel movement (1 = every pixel)
Padding: Add zeros around border to control output size
Output size: $\frac{W - K + 2P}{S} + 1$
Multiple filters: Each filter produces one feature map. Stacked = depth.

2. Activation (ReLU)

$$ \text{ReLU}(x) = \max(0, x) $$

Applied element-wise after convolution. Introduces non-linearity.

3. Pooling Layer

Max Pooling: Takes maximum value in each window. Reduces spatial dimensions, provides translation invariance.

$$ \text{MaxPool}(i,j) = \max_{(m,n) \in W_{ij}} f(m,n) $$

Average Pooling: Takes mean of each window.

4. Fully Connected (FC) Layer

Flattens feature maps into 1D vector. Each neuron connected to all previous neurons. Final layers for classification.

$$ z = W \cdot \text{flatten}(\text{features}) + b, \quad \text{output} = \text{softmax}(z) $$

Feature Learning Hierarchy:

Layer Depth	Learns	Example
Early layers	Low-level features	Edges, corners, colors
Middle layers	Mid-level features	Textures, patterns, shapes
Deep layers	High-level features	Objects, faces, scenes

Dilated (Atrous) Convolution (Ch-16/17)

Spreads kernel taps by dilation rate $r$: a 3×3 kernel with $r=2$ sees a 5×5 area with only 9 weights. Grows receptive field without extra parameters or downsampling.

$$ \text{Effective size} = K + (K-1)(r-1), \quad \text{e.g. } K=3, r=2 \Rightarrow 5\times5 $$

Exam (Ch-16/17): Dilation $r=1$ = normal conv. Higher $r$ = wider view, same weights. Used in segmentation (DeepLab) to keep resolution.

Exam: CNN = weight sharing (same kernel across image) + local connectivity + pooling = translation invariant, parameter efficient.

← Prev: ViT Next: ViT Layer Architecture →

← Back to Topics

Vision Transformer - Layer Architecture

Layer Architecture

How It Works

Vision Transformer Layer Architecture (Page 7)

Single ViT encoder layer: Input → LayerNorm → MSA → Add+Norm → MLP → Add+Norm, with residual connections (yellow dashed) bypassing each sub-layer

Each Transformer encoder layer in ViT consists of two sub-layers with residual connections and layer normalization.

Detailed single ViT encoder layer with residual connections

Mathematical Details:

Multi-Head Self-Attention:

$$
\text{head}_i = \text{Attention}(XW_i^Q, XW_i^K, XW_i^V)
$$
$$
\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
$$
$$
\text{MSA}(X) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O
$$

Where $d_k = D/h$ (embedding dim / number of heads).

MLP:

$$ \text{MLP}(x) = W_2 \cdot \text{GELU}(W_1 x + b_1) + b_2 $$

Hidden dimension = $4D$ (4x embedding dim).

Residual + LayerNorm:

$$ x_{out} = \text{LayerNorm}(x + \text{SubLayer}(x)) $$

How It Works (End to End)

End-to-end ViT pipeline: Image split into patches, linearly projected with positional embeddings, processed by L Transformer encoder layers, then [CLS] token is classified

Input Image (e.g., 224×224×3)
Patch Splitting: Divide into 16×16 patches → 196 patches, each 16×16×3 = 768 dims
Linear Projection: Each patch → $D$-dim embedding (e.g., $D=768$)
Add [CLS] token: Prepend learnable [CLS] token → sequence length becomes 197
Add Position Embeddings: Add learnable positional embeddings to all tokens
Pass through L Transformer Encoder Layers:
- Each layer: MSA → Add+Norm → MLP → Add+Norm
- Self-attention captures global relationships between all patches
- MLP processes each patch independently with non-linearity
- Residual connections preserve information
Extract [CLS] output: Take the [CLS] token's final representation
Classification Head: [CLS] output → MLP/Linear → class probabilities

Key Differences from CNN:

Aspect	CNN	ViT
Receptive field	Local (grows with depth)	Global (from first layer)
Inductive bias	Translation invariance, locality	None (learns from data)
Data requirement	Less data needed	Needs large dataset or pretraining
Spatial structure	Preserved via pooling	Flattened (positional encoding restores)

Exam: ViT = patches + positional encoding + Transformer encoder. No convolution. Self-attention enables global context from the start. Needs more data than CNN.

← Prev: CNN Next →

Magnitude	Classification
\(G > T\)	Strong edge
\(T/2 < G \leq T\)	Weak edge
\(G \leq T/2\)	Non-edge

Type	Operator	Math
1. Roberts (2×2, fastest)	\(G_x = \begin{bmatrix}1&0\\0&-1\end{bmatrix}\), \(G_y = \begin{bmatrix}0&1\\-1&0\end{bmatrix}\)	\(G = \sqrt{G_x^2+G_y^2}\)
2. Prewitt (3×3, uniform weights)	\(G_x = \begin{bmatrix}-1&0&1\\-1&0&1\\-1&0&1\end{bmatrix}\)	Same magnitude formula; less noise-sensitive than Roberts
3. Sobel (3×3, centre ×2)	\(G_x = \begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix}\)	\(G = \sqrt{G_x^2+G_y^2}\), \(\theta = \tan^{-1}(G_y/G_x)\)

Case	Eigenvalues	Shape
Flat region	\(\lambda_1 \approx 0, \lambda_2 \approx 0\)	No change in any direction
Edge	\(\lambda_1 \gg \lambda_2 \approx 0\)	Change along one direction only
Corner	\(\lambda_1 \gg 0, \lambda_2 \gg 0\)	Change in all directions

#	Type	What it does	2D matrix
1	Translation	Shift by \((t_x,t_y)\)	Needs homogeneous 3×3 \(T\)
2	Rotation	Rotate by \(\theta\) about origin	\(\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}\)
3	Scaling	Resize by \((s_x,s_y)\)	\(\begin{bmatrix}s_x&0\\0&s_y\end{bmatrix}\)
4	Shearing	Slant; square → parallelogram	\(\begin{bmatrix}1&sh_x\\sh_y&1\end{bmatrix}\)
5	Reflection	Mirror across axis	\(\begin{bmatrix}-1&0\\0&1\end{bmatrix}\) or \(\begin{bmatrix}1&0\\0&-1\end{bmatrix}\)

Function	Formula	Range	Use
Sigmoid	\(\sigma(z) = \frac{1}{1+e^{-z}}\)	(0, 1)	Binary classification output
Tanh	\(\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\)	(-1, 1)	Hidden layers (zero-centered)
ReLU	\(f(z) = \max(0, z)\)	[0, ∞)	Most common hidden layer
Leaky ReLU	\(f(z) = \max(\alpha z, z)\), \(\alpha=0.01\)	(-∞, ∞)	Prevents dead neurons
Softmax	\(S(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}\)	[0, 1], sum=1	Multi-class output

Term	Definition
Loss Function \(L(\hat{y}, y)\)	Measures how far predictions \(\hat{y}\) are from true labels \(y\). E.g., MSE: \(L = \frac{1}{N}\sum(\hat{y}-y)^2\), Cross-entropy: \(L = -\sum y \log \hat{y}\)
Weights \(w\)	Learnable parameters connecting neurons. \(z = wx + b\).
Bias \(b\)	Additional parameter that shifts the activation function.
Forward Pass	Compute output: input → weighted sum → activation → ... → output → loss
Backward Pass	Compute gradients of loss w.r.t. each weight using the chain rule.
Chain Rule	\(\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}\)
Learning Rate \(\eta\)	Step size for weight updates. Too large = diverge, too small = slow convergence.
Gradient Descent	\(w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w}\)

Term	Definition
Patch Embedding	Convert image patches to vectors. Each \(P \times P \times 3\) patch → \(D\)-dim vector via linear projection.
Self-Attention	Each patch attends to all other patches. Computes relationships: \(\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\)
Multi-Head Attention	Multiple attention heads in parallel, each learning different relationships. Concatenated and projected.
[CLS] Token	Special learnable token prepended to sequence. Its final representation used for classification.
Positional Encoding	Learnable embeddings added to patch tokens to encode spatial position (ViT uses learnable, not sinusoidal).
Layer Norm	Applied before (pre-norm) each sub-layer. Stabilizes training.
MLP	Two linear layers with GELU activation. Hidden dim = 4× embedding dim.

Computer Vision

Thresholding

Morphology

Sobel & Laplacian

Noise Models

Noise Removal Filters

Segmentation

Harris Corner Detection

Geometric Transformations

Neural Network

Backpropagation

Optimizers

Vision Transformer (ViT)

CNN

Vision Transformer Architecture

Thresholding

Definition of Thresholding (Ch-8)

How Thresholding Works (Ch-8):

When to use:

Adaptive (Local) Thresholding

Variants:

Window Size Effect:

Otsu's Method (Automatic Global)

Prerequisites:

Algorithm:

Between-Class Variance:

Within-Class Variance:

Morphology

Structuring Element (SE)

Dilation: \(I \oplus SE\)

Example:

Erosion: \(I \ominus SE\)

Example:

Opening: \(I \circ SE = (I \ominus SE) \oplus SE\)

Example: Open a 5×5 block with 3×3 SE

Closing: \(I \bullet SE = (I \oplus SE) \ominus SE\)

Example: Close a ring (5×5 with 1-pixel hole) with 3×3 SE

Morphological Gradient

What Dilation Does (Ch-9)

5 Morphology Maths (Ch-9)

Top-Hat Transform

White Top-Hat

Black (Bottom) Top-Hat

Sobel & Laplacian Operators

Image Smoothing (Noise Reduction)

1. Mean (Box) Filter

2. Weighted (Gaussian) Filter

3. Median Filter

4. Bilateral Filter

First 4 Smoothing Filters (Ch-10)

Sobel Operator (1st Order Derivative)

Horizontal Kernel \(G_x\)

Vertical Kernel \(G_y\)

Gradient Magnitude:

Gradient Direction:

Edge Strength Classification:

Laplacian Operator (2nd Order Derivative)

4-Connected Kernel:

8-Connected Kernel:

Zero-Crossing Detection:

Problem: Noise Sensitivity

Noise Models

Gaussian Noise

Salt & Pepper (Impulse) Noise

Poisson (Shot) Noise

Speckle Noise

Noise Removal Filters

Mean (Arithmetic) Filter

Median Filter

Gaussian Smoothing Filter

Adaptive (Local) Filter

Adaptive Mean Filter:

Adaptive Median Filter:

Segmentation Scenario

Image Segmentation

1. Thresholding-based

2. Region-based (Region Growing)

3. Edge-based

4. Clustering (K-means)

Which Segmentation to Apply?