CS180-Proj1: Colorizing the Prokudin-Gorskii Photo Collection

David Wei
david_wei@berkeley.edu

1. Introduction

Sergey Prokudin-Gorskii photographed the Russian Empire using black-and-white negatives with red, green and blue filters respectively, hoping future technologies could stack the three layers to produce colored image (See Prokudin-Gorskii's work here ). The goal of this project is to take the digitized negatives of Prokudin-Gorskii's work and produce the single RGB color image. Note that the three layers are not accurately aligned. Therefore, in order to produce a clear picture, an automated aligning process is required.

2. Layer Alignment by Edge Detection (Bells and Whistles)

To accurately align the blue and green layers to the red layer, I created a loss function to represent how accurately the layers are aligned. The optimal alignment is calculated by finding the alignment with the least loss. I initially employed L2 loss as loss function, only to find the results not satisfying. I discovered that a bright pixel in one layer does not necessarily align to another bright pixel in a different layer, as a color may result in a high value when passed through one filter, while leaving a low value when passed though another filter.

Even though pixel values in different layers may not align, edges of shapes and figures should align. Therefore, to optimize my model, I adapted the famous Sobel Operator to detect the edges in each layer before alignment, and the loss function is defined as the L2 loss between the layers' edges. Figure 1 shows the image emir.tif's original blue layer and its edges detected by the Sobel Operator.

Figure 1a
Figure 1a: emir.tif's blue layer.
Figure 1b
Figure 1b: Edge of emir.tif's blue layer.

3. Exhaustive Search

To find the alignment resulting in the lowest loss, I exhaustively searched all the possible candidates within a range, where the center of the blue or green layer may be aligned to one of the pixels in a square on the red layer. The square is defined by its center and its length of side. I initially set the square at the centre of the red layer, with its sides 21 pixels long. After traversing all 441 candidates, the alignment with the lowest loss is chosen. I adapted this process on the three smaller jpg images, cathedral.jpg, monastery.jpg and tobolsk.jpg respectively. In Figure 2, 3, 4, I compare the images stacked directly without the alignment process to the aligned images.

Figure 2a
Figure 2a: Colored cathedral.jpg without auto alignment.
No blue shift; No green shift.
Figure 2b
Figure 2b: Colored cathedral.jpg after auto alignment.
Blue shift: (-3, -2); Green shift: (-6, -1).
Figure 3a
Figure 3a: Colored monastery.jpg without auto alignment.
No blue shift; No green shift.
Figure 3b
Figure 3b: Colored monastery.jpg after auto alignment.
Blue shift: (-10, -3); Green shift: (-7, -1).
Figure 4a
Figure 4a: Colored tobolsk.jpg without auto alignment.
No blue shift; No green shift.
Figure 4b
Figure 4b: Colored tobolsk.jpg after auto alignment.
Blue shift: (-6, -3); Green shift: (-4, -1).

4. Image Pyramid using Recursive Function

Although exhaustive search has proven effective on small jpg images, when adapted on larger tif images, the optimal blue and green shift may exceed the range of [-10, 10] pixels. In order to expand the range, more time is needed. Time complexity increase dramatically along with the expansion of search range. To improve time efficiency, I adapted the idea of Image Pyramid. For every large tif image, I downsample it using average pooling with a factor of 3 along each direction, with the resulting image subjected to the same procedure again until it is small enough, which in this task, having a height smaller than 500 pixels. Exhaustive search is conducted on the center square of the smallest image, and the optimal shift for blue and green layers are found and recorded. Then, the pyramid goes up a layer, using the optimal shift obtained by the lower layer to get an estimate of where the best alignment is, and conduct exhaustive search around that estimate to gain a more accurate result based on the image with higher resolution. This process is again applied on the next layer with an even bigger image, and eventually the optimal alignment is obtained from the search on the original image. I use a recursive function to implement this process. The resulting images are shown in Figure 5 ~ 15.
Figure 5a
Figure 5a: Colored cemir.tif without auto alignment.
No blue shift; No green shift.
Figure 5b
Figure 5b: Colored emir.tif after auto alignment.
Blue shift: (-107, -40); Green shift: (-58, -17).
Figure 6a
Figure 6a: Colored harvesters.tif without auto alignment.
No blue shift; No green shift.
Figure 6b
Figure 6b: Colored harvesters.tif after auto alignment.
Blue shift: (-110, -14); Green shift: (-64, -3).
Figure 7a
Figure 7a: Colored icon.tif without auto alignment.
No blue shift; No green shift.
Figure 7b
Figure 7b: Colored icon.tif after auto alignment.
Blue shift: (-90, -23); Green shift: (-48, -5).
Figure 8a
Figure 8a: Colored lady.tif without auto alignment.
No blue shift; No green shift.
Figure 8b
Figure 8b: Colored lady.tif after auto alignment.
Blue shift: (-110, -13); Green shift: (-63, -3).
Figure 9a
Figure 9a: Colored melons.tif without auto alignment.
No blue shift; No green shift.
Figure 9b
Figure 9b: Colored melons.tif after auto alignment.
Blue shift: (-109, 61); Green shift: (-96, -3).
Figure 10a
Figure 10a: Colored onion_church.tif without auto alignment.
No blue shift; No green shift.
Figure 10b
Figure 10b: Colored onion_church.tif after auto alignment.
Blue shift: (-107, -35); Green shift: (-57, -10).
Figure 11a
Figure 11a: Colored sculpture.tif without auto alignment.
No blue shift; No green shift.
Figure 11b
Figure 11b: Colored sculpture.tif after auto alignment.
Blue shift: (-110, 26); Green shift: (-106, 16).
Figure 12a
Figure 12a: Colored self_portrait.tif without auto alignment.
No blue shift; No green shift.
Figure 12b
Figure 12b: Colored self_portrait.tif after auto alignment.
Blue shift: (-110, -59); Green shift: (-98, -8).
Figure 13a
Figure 13a: Colored three_generations.tif without auto alignment.
No blue shift; No green shift.
Figure 13b
Figure 13b: Colored three_generations.tif after auto alignment.
Blue shift: (-110, -8); Green shift: (-58, -4).
Figure 14a
Figure 14a: Colored church.tif without auto alignment.
No blue shift; No green shift.
Figure 14b
Figure 14b: Colored church.tif after auto alignment.
Blue shift: (-58, 4); Green shift: (-33, 8).
Figure 15a
Figure 15a: Colored train.tif without auto alignment.
No blue shift; No green shift.
Figure 15b
Figure 15b: Colored train.tif after auto alignment.
Blue shift: (-85, -29); Green shift: (-43, -26).

* Finished on Sep 9, 2024.