Mosaics

Autostitching Photo Mosaics

Making Panoramas

A panorama is best captured by a series of pictures with slightly overlapping fields of view, which are stiched together. Using perspective projections, we can create an image mosaic (or panorama) where straight lines are kept straight, but edges may become distorted.

Nonetheless, we can take input images with varying viewing directions of a scene and stitch them together, so long as we can match reference points between the images.

Recover Homographies

Any two images can be related by a 3x3 homography matrix:

While we only need four distinct points to recover a solution, this process is prone to noise, and choosing more correspondence pairs helps in finding a better least squares regression. Here, we can solve for x to recover our homography H.

For now, I manually record coordinate pairs that appear in both images. Later on, we will automate this process.

Image Rectification & Warping

With our correspondence pairs recorded, we can warp the images. For every coordinate value, we can multiply by the inverse homography matrix and recover points to interpolate between, and this helps avoid aliasing. Image rectification allows us to take a planar surface and match its perspective to some output coordinates.

For example, we can warp the following image on the left into correspondence point pairs selected on the right to yield the following warped images of the source (left) onto the target (right) perspective.

Feature Detection

Instead of manually selecting our correspondence points, we can compute our image homogrpahy automatically using approaches outlined in the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al., but with several simplifications.

Here I implement a Harris Interest Point Detection and Adaptive Non-Maximal Suppression Algorithm, and show the points overlaid on several images. After extracting these features, we can match the pairs that look similar, and use a 4-point RANSAC algorithm to compute a robust homography estimate.

Harris Point Detection

Berkeley Psychology Building

Wildfire & Sky

Tahoe

Adaptive Non-Maximal Suppression

Berkeley Psychology Building

Wildfire & Sky

Tahoe

Autostitching

After recovering the homography transformation and warping a source image into a target image, we can stitch the results together and blend the warps. Using a simple alpha blending technique (outlined in previous projects on this website), we can seamlessly blend the two images together to create our mosaics, and crop the result.

Berkeley Psychology Building

Wildfire & Sky

The difference in contrast pictured above has to do with the difficulty focusing under low light conditions.

Tahoe

Clearly, the trees don't quite align, so let's try that again. This time, with three source images.

Tahoe Remastered

Summary

Implementing algorithms outlined in an academic paper can be challenging, but this was a fun project that revealed some of the truly powerful applications of linear algebra. Selecting all the correspondence pairs can be tough, especially if my selection is off even just slightly, which was the case for the Tahoe picture initially. Nonetheless, panoramas are cool, and with the autostitching algorithm implemented, I have a newfound appreciation for the built-in panorama feature on my smartphone.

Acknowledgements

  • CS 194-26 Course Staff at UC Berkeley
  • Professor Alexei (Alyosha) Efros