Overview

By isolating and manipulating different frequency components of an image, a wide range of visual effects can be achieved, from seamless blending to edge detection. This project explores different techniques and tools like the Fourier transform, derivative of gaussian, laplacian stacks among others to filter out features from images, enhance them or create cool visual effects.

Finite Difference Operator

Edge detection is essential in computer vision, as edges highlight rapid intensity changes, often marking object boundaries. A common method involves calculating the image gradient, which shows the direction and strength of these changes. Finite difference operators are an effective way to approximate derivatives and extract the gradient for edge detection.

Convolving an image with finite difference operators provides a good approximation of its gradient. Convolution with the $D_x$ operator returns the gradient in the x direction, highlighting horizontal edges, while $D_y$ captures vertical edges. This process shifts the kernel across the image and calculates the weighted sum of pixel intensities, estimating the local derivative. Formally, this computes the partial derivatives, revealing the image's intensity changes in each direction.

\frac{\partial I}{\partial x}= I * D_x\ \ \ \ \frac{\partial I}{\partial y}= I * D_y

where

D_x = \begin{bmatrix}-1&1\\\end{bmatrix}

and

D_y = \begin{bmatrix}-1\\1\\\end{bmatrix}

, and

*

denotes the convolution operator.

By combining the x and y gradients, we can compute the magnitude and direction of the overall gradient, providing crucial information about the edges and intensity transitions within the image. To find the edges of an image one could first compute the gradient magnitude $\Delta I = \sqrt{\left(\frac{\partial I}{\partial x}\right)^2 + \left(\frac{\partial I}{\partial y}\right)^2}$ and then converting the gradient magnitude image to a binary image by setting an arbitrary threshold.

Derivative of Gaussian (DoG) Filter

However, the edge image appears to have a lot of noise. To find the edges more accurately one could attempt to apply a Gaussian filter to reduce noise as high frequency patterns, effectively blurring the image, and then follow the procedure we did.

I_B = I * G

\Delta I_B = \sqrt{\left(\frac{\partial I_B}{\partial x}\right)^2 + \left(\frac{\partial I_B}{\partial y}\right)^2}

However, we can slightly modify the operations we just did to reduce the number of convolutions we perform on the original image. One could convolve the gaussian kernel with

D_x

and

D_y

before applyign the Gaussian filter to the original image. We define this as the derivative of gaussian such that

DoG_x = G * D_x,\ \ \ \ DoG_y = G * D_y

\Delta I_B = \sqrt{\left(I * DoG_x\right)^2 + \left(I * DoG_y\right)^2}

By applying the gaussian filter, we effectively blur the original image and filter out high frequencies which appear as noise when filtering the edges. After blurring the image the edges are clearer and sharper, while high frequency areas like the grass the gradient is big but not consistent is no longer detected as edges. Furthermore, the process of convolving the original image with the gaussian filter and then the derivatives should be equivalent to convoling the original image direclty with the DoG, which we can verify by subtracting the two resulting images and check if the resulting matrix only contains 0s, which is the case. Note that to visualize the DoG filtered image, I initially used the same convolution method, but for an exact match, I applied the full convolution approach.

Image Sharpening

A fairly straightforward and simple method to sharpen an image is by filtering the high freuqencies of an image and adding them to the original. To produce a high pass filter one can use the a low pass filter with a gaussian blur $G$ like we did before, and substract the low frequencies from the original image. Then, our sharpened image can be calculated

\begin{align*} I_s & = I + \alpha \cdot (I - (I * G))\\ &= (1 + \alpha) \cdot I - \alpha \cdot (I * G)\\ &= I * ((1 + \alpha)\cdot \delta - \alpha \cdot G) \end{align*}

where $\delta$ is the unit impulse. This equation suggest that we can apply our sharpening filter with a single convolution.

Hover your mouse over the following examples to see the orginal version of the image.

The latter example takes fairly clear image taken with a digital camera and blurs it before applying our sharpening operation. We notice that the end result gets close to the original image but it is not as good. We expect this due to the application of the gaussian filter to our orginal image and once again later to filter the high frequencies of the image, where we are producing new values for the pixels by taking a weighted mean.

Hybrid Images

Hybrid images blend the sharp details (high frequencies) of one image with the softer, broader features (low frequencies) of another. At close range, the detailed image is more noticeable, whereas from a distance, the smoother features take over. This results in a visual effect that changes depending on how far the viewer is from the image. To create a hybrid image, one can separate the high and low-frequency components of the images. The low-frequency components are obtained by applying a Gaussian filter, which smooths the image. The high-frequency components are extracted by subtracting the blurred image from the original, leaving the finer details. For example, by combining the low-frequency content of image 1 with the high-frequency details of image 2, a hybrid image is produced, where different features become visible at varying viewing distances.

To visualize the difference range of frequencies between images we utilize the Fourier transform, a mathematical tool that decomposes an image into its frequency components. It converts the spatial domain (the pixel values of the image) into the frequency domain, representing how fast pixel values change across the image. Low frequencies correspond to smooth, gradual variations (broad structures), while high frequencies capture sharp edges and fine details.

Note that the effect in the last example is not very clear. The images are difficult to separate due to their very similar high-frequency components.

Gaussian and Laplacian Stacks

A Gaussian stack is a series of progressively blurred versions of an image, created by repeatedly applying a Gaussian filter. Each level of the stack retains the original image size but removes more high-frequency details, preserving only the broader structures. In contrast, a Laplacian stack is built by subtracting consecutive levels of the Gaussian stack. This reveals the high-frequency details lost between levels, emphasizing fine structures and edges.

To compute a Guassian stack one can repeatedly apply a Gaussian filter to the previous image in the stack, starting from the original image. Then

S_{G,\ i} = S_{G,\ i -1 } * G\ \ \ \ \text{where}\ \ S_{G,\ 0} = I

S_{G} =\left[I,\ I*G,\ I * G * G,\ \cdots\right]

Then, to compute a Laplacian stack, we subtract consecutive levels of the Gaussian stack. At the final level, the Laplacian stack hold the final gaussian level. So

S_{L,\ i} = S_{G,\ i} - S_{G,\ i+1}\ \ \ \ \text{and}\ \ S_{L,\ n} = S_{G,\ n}

S_{L} =\left[I - I * G,\ I*G - I * G * G,\ I * G * G - I * G * G * G,\ \cdots\right]

Multiresolution Blending

Summing all the levels of the Laplacian stack, along with the final Gaussian level, reconstructs the original image as we reintroduce the high-frequency details lost in the blurring process. Each Laplacian level captures the difference between successive Gaussian layers, so adding these differences back restores the image's full frequency range, resulting in an accurate reconstruction.

A binary mask helps blend two images by determining which parts of each image are visible in the final result. For each level of the Laplacian stacks, the mask decides how much of each image's details contribute to the blended image. In areas where the mask is 1, we take details from the first image, and where it’s 0, we use the second image. Gradually combining the details from each image across multiple frequency levels ensures a smooth, natural blend, creating seamless transitions between the two images. For each layer in the stack we compute the blending as

S_{L,\ I_B,\ i} = S_{M,\ i}\cdot S_{L,\ I_1,\ i} + (1 - S_{M,\ i})\cdot S_{L,\ I_2,\ i}

Then we can reconstruct the blended images by taking the sum of our laplacian blended stack.

I_{B} = \sum_{i}S_{L,\ I_B,\ i}

by Jorge Diaz Chao