Convolutional neural networks

CNNs are the default tool whenever the input is an image, or anything else laid out on a grid. The lecture opens with examples: predicting a person's age from a face photo, classifying skin cancer from medical images, object detection (drawing a box around every car, truck and person) and object segmentation (labeling which pixels belong to each object). Video works too: it's a stack of images over time, so you can run a CNN on each frame or use 3-D convolutions over the whole block. The professor's rule (01:28): "if you have image data, then the convolutional neural network is the network you should use."

What makes image classification hard

The same object can look very different: front, side or back view, bright or dark lighting, different contrast, and different position, rotation and scale in the frame. The position problem alone breaks an MLP. Flatten an image of a "5" and the pixels that make up the 5 land at certain entries of the vector. Shift the 5 a few pixels and those pixels land at completely different entries. To the MLP these are unrelated inputs, so it has to learn separate weights for every position the 5 might appear in, even though it's the same shape.

Before CNNs, the fix was to engineer the input by hand:

• Hand-crafted features such as SIFT (scale-invariant feature transform) or HOG (histogram of oriented gradients): find a region of interest, summarize it as one feature vector, and feed that vector to a classifier such as an SVM (week 5).
• Landmarks for well-structured objects like faces: detect key points (eye corners, nose tip) and match them against a template.
• Preprocessing: crop and center the object (e.g. cut a patch around a small cat) before classifying it.

The two ideas behind a CNN

A CNN replaces the hand engineering with two structural assumptions (slide 11):

• Sparse connectivity. Each output neuron is connected only to a small patch of the input (say 3×3 pixels), not to every pixel as in an MLP's fully connected layer. Nearby pixels are strongly related; far-apart pixels much less so (Bishop's motivation).
• Parameter sharing. The same weights are used for every patch. A detector that finds a vertical edge in one place is just as useful everywhere else.

The professor names the payoff of each (12:30): sparse connections make the computation faster, and sharing makes the model smaller (fewer parameters to learn). Later we'll see that in a classic CNN, most of the parameters actually sit in the fully connected layers at the end.

A kernel (also called a filter or feature detector) is a small grid of weights, typically 3×3 or 5×5; 3×3 is the most common. To apply it, place it over a patch of the image, multiply each weight by the pixel under it, add everything up, and write the sum into one cell of the output. Then slide the kernel one step and repeat. This is the sliding window. The grid of outputs is called a feature map (or activation map), and the patch of input pixels that one output cell depends on is that cell's receptive field. The kernel size sets how big the receptive field is.

1. Output size: a 3×3 window fits in 5 − 3 + 1 = 3 positions across and 3 down, so the map is 3×3.
2. Top-left window (columns 1–3, rows 1–3) is all 1s: each row gives , so . Flat region, no edge.
3. Top-middle window (columns 2–4, rows 1–3): every row is (1, 1, 7), giving three times, so . The window straddles the edge in all three rows.
4. Bottom-right window (columns 3–5, rows 3–5): rows (1, 7, 7), (7, 7, 7), (7, 7, 7) give , so . Only one row still sees the edge.
5. Full map: . The big magnitudes trace the edge, including its jog. (Checked with scipy.signal.correlate2d.)

The sign tells you the direction of the edge: this kernel gives negative values for dark-to-bright (left to right) and positive values for bright-to-dark. After a ReLU, only the positive responses survive, so a network that wants dark-to-bright edges simply learns the mirrored kernel. The horizontal version, rows (1, 1, 1), (0, 0, 0), (−1, −1, −1), responds to top-versus-bottom changes (slide 40). Run on a handwritten digit, the vertical filter lights up the left and right sides of each stroke, and the horizontal one the tops and bottoms.

Where do kernels come from? In classical image processing you design them by hand (edge filters, Sobel filters). In a CNN they're learned by gradient descent. "A CNN can learn whatever it finds best based on optimizing the objective" (slide 40). Remarkably, the first layer of a trained CNN ends up looking like a bank of edge and color-blob detectors (Section 9).

Two settings control how the kernel moves:

• Padding : add a border of extra pixels around the image before sliding. Usually the border is zeros (zero padding); mirroring the edge pixels also works. The professor notes the choice makes only a marginal difference to accuracy. Padding lets the kernel center on edge pixels, so the map doesn't shrink.
• Stride : how many pixels the window jumps between positions. Stride 1 visits every position; stride 2 skips every other one and roughly halves the width and height.

Stride versus pooling. The professor points out (26:20) that a stride-2 convolution already halves the map, so you may not need a separate pooling layer to shrink it. Many modern networks do exactly that.

One kernel finds one kind of pattern, so a layer uses many kernels. Each kernel produces its own feature map, and the maps are stacked; each one is called a channel. "6@28×28" on the LeNet diagram means 6 kernels, so 6 channels, each 28×28.

Images have channels too: a grayscale image has one, a color image has three (red, green, blue). When the input has channels, each kernel is a block. It covers the patch in every input channel at once and adds everything into one number. So a layer's weights form a 4-D array:

A pooling layer shrinks each feature map by summarizing small blocks. The older name, used on the LeNet diagram, is subsampling. Slide the pooling window as in a convolution (most often 2×2 with stride 2) and replace each block with one number:

• Max pooling: keep the largest value in the block. The most common choice.
• Mean (average) pooling: keep the average.

Pooling works on each channel separately, so the number of channels doesn't change, and the output size follows the same formula as convolution.

Slides 27–28 show the same thing with 3×3 blocks and stride 3 on a 6×6 map: max gives (8, 5 / 6, 3) and mean gives (3.78, 2.33 / 3, 1.22). I re-checked those numbers; they're right.

What pooling is for

• Less computation and memory: each 2×2 pool cuts the number of values by 4.
• Tolerance to small shifts: if a strong response moves by a pixel but stays inside the same pooling block, the max doesn't change.
• A pyramid of scales: repeated pooling views the image at coarser and coarser resolution, which the professor says (37:40) gives some ability to handle scale.

Downsides (slide 27): information is lost. That may be fine for classification ("is there a cat?"), but hurts tasks where exact positions matter, such as face recognition (relative positions of eyes and mouth) or segmentation. Rotation is another weak spot (37:57): a kernel isn't rotation-invariant, so rotating a patch changes its response, and plain CNNs don't handle rotation unless specially designed or trained on rotated examples. The slide's practical advice: "some image preprocessing still recommended".

No learnable parameters (slide 28, lecture 39:44): max and mean are fixed rules. People have tried learnable pooling; it's slower and hasn't paid off, so today almost nobody uses it.

The lecture's running example is LeNet-5 (LeCun, Bottou, Bengio and Haffner, 1998), built to read handwritten digits. A 32×32 grayscale image goes through alternating convolution and subsampling, then fully connected layers, ending in 10 outputs, one per digit.

Two things to notice. First, C5's 5×5 kernel exactly covers S4's 5×5 maps, so each of its 120 outputs sees all values: it's a convolution in name but a fully connected layer in effect (the professor's "16 × 25 = 400" on slide 13, and his description of C5 as "a mapping from S4 after vectorization"). Second, C5 + F6 + output hold 59,134 of the 61,706 parameters, about 96%. That's the lecture's point (48:54): in a classic CNN, the dense layers at the end hold most of the parameters and are the computational bottleneck. (The parameter counts assume every C3 kernel sees all 6 S2 maps, as a modern implementation would. The 1998 original wired C3 to subsets of maps and used a slightly different output layer, which brings its total to about 60,000.)

Vocabulary from the diagram (slide 14)

• "Feature detectors" = kernels = filters, reused across positions (weight sharing).
• "6@28×28": the number before @ is the number of feature maps (channels); after @ is the feature-map size.
• "Subsampling" = what we now call pooling.
• "Gaussian connections": LeNet's output layer measured squared distances to class templates, which is essentially a fully connected layer with mean-squared-error loss. Today you'd use a fully connected layer + softmax + cross-entropy loss (week 13).

Hidden layers, features and the classifier

The input and output layers are fixed by the problem; everything in between is hidden. Each "bunch" of feature maps produced by a convolution, and each fully connected layer, counts as one layer. Pooling layers don't count: they have no weights. When a paper talks about "the feature vector" of an image, it usually means the activations of the second-to-last layer (here the 84-dimensional F6). The final layer, which turns those features into class scores, is "the classifier". Slide 21 draws the line more loosely: everything fully connected is the "regular classifier" and everything convolutional is the "automatic feature extractor".

Compare the first layer with a fully connected layer producing the same 28×28×8 = 6,272 outputs from the 784 pixels: parameters, about 61,500 times the convolution's 80. Even in this small net, the one fully connected layer holds 86% of all the parameters.

The lecture defines the receptive field as the patch of pixels one feature-map cell sees, and says the kernel size sets it. That's true for the first layer. Deeper layers see more: a layer-2 neuron looks at a 3×3 patch of layer-1 cells, and each of those looks at its own 3×3 patch of pixels, so together they cover 5×5 pixels.

1. Three 3×3 convolutions, stride 1: . Each layer adds 2.
2. Conv 3×3 → max-pool 2×2 (stride 2) → conv 3×3: , then with jump 2, then . After pooling, each extra layer adds twice as much: pooling speeds up the growth.
3. LeNet-5: C1 (5) → S2 (6) → C3 (14) → S4 (16) → C5 (32). A C5 neuron sees the whole 32×32 image, which is what you want right before classifying.

Training a CNN is the same gradient descent with backpropagation as for last week's MLP. The only new wrinkle is the "additional weight sharing constraint" (slide 23): each kernel weight is used at every position, so it influences the loss through many paths. The multivariable chain rule (slide 24) says what to do: add up the contributions from every path.

1. Forward: , . Loss .
2. Output errors: , .
3. touched (for ) and (for ):

4. touched and :

5. Gradient step with : . A finite-difference check of the loss gives the same gradient .

Because every position sends its gradient into the same weight, each copy stays identical after the update. That's how the constraint on slide 25 (the same weight at two places in the image) is maintained automatically.

AlexNet (Krizhevsky, Sutskever and Hinton, 2012) is the breakthrough that made CNNs mainstream. It won the ImageNet challenge with a top-5 error of about 17%, against 26% for the runner-up (Géron), whose methods were based on hand-crafted features. Its structure is LeNet's, scaled up: five convolutional layers with max pooling, then three fully connected layers. What was new:

• ReLU activations instead of sigmoid/tanh, which train much faster in deep nets (week 13).
• GPU training, with the network split across two GPUs (that's why slide 29's diagram has two parallel streams).
• Regularization with dropout and data augmentation (random shifts, flips, lighting changes), from Géron.

ImageNet (the challenge version): about 1.2 million training images in 1,000 classes collected from the internet. Top-5 accuracy counts a prediction as correct if the true label is among the model's 5 highest-scoring classes; top-1 requires the single best guess to be right.

Fixed input size (43:18): images are resized to one size, usually 224×224 (still standard today, because bigger inputs cost much more computation). That's 224·224·3 = 150,528 input numbers. The size has to be fixed because the flattened vector going into the first fully connected layer must have a fixed length. (Géron's fully convolutional networks replace the dense layers with convolutions and accept any size.)

Where the parameters are. Counting AlexNet layer by layer (as one network, without the two-GPU split): the five convolutional layers have about 3.7 million parameters, while the three fully connected layers have about 58.6 million, roughly 94% of the total. The first one alone (6×6×256 = 9,216 inputs → 4,096 outputs) has 37.8 million. That's the "bottleneck" the professor points to at 44:07: dense layers are slow and huge; convolutions are cheap because their connections are sparse.

After AlexNet

Slide 33 (Canziani et al., 2016) compares later architectures (VGG, GoogLeNet/Inception, ResNet, and others) by accuracy, computation and size. The lecture's summary: accuracy improved a lot over the years, top-1 accuracy on ImageNet is now close to 90%, top-5 is near 100%, and the professor considers ImageNet classification essentially "solved". Géron covers these architectures in the sections after AlexNet (GoogLeNet, VGGNet, ResNet), which are reading, not lecture material.

What a CNN sees

The learned kernels can be visualized. AlexNet's first-layer kernels (slide 30) look like oriented edge detectors and color blobs, much like the hand-designed edge filters of image processing. Zeiler and Fergus (2014, slides 41–44) went deeper: take a strongly activated neuron, push its activation back down to pixel space (backpropagating and "unpooling"), and see which image patterns fire it. A hierarchy appears:

• Layer 1: edges and simple color patterns.
• Layer 2: corners, textures, simple shapes.
• Layer 3: object parts (a mesh pattern, a wheel, a face-like arrangement).
• Layers 4–5: whole objects or object classes (dog faces, people, bicycles), grouped like clusters across many images.

The professor's takeaway (53:43): a CNN classifier is really detecting specific concepts in the image and classifying from those detections. That's why the same networks transfer so naturally to object detection.