Video Stabilization

Here I have implemented an offline video stabilizer that places customizable constraints on maximum camera translation, rotation, and zoom, and enforces smooth motion by regularizing the first and second order finite differences (velocity, acceleration) of the camera motion parameters, while optimizing the smoothed camera motion to emulate the original camera motion (intentional camera movements).

A convex objective is formulated and solved using the FISTA (fast iterative shrinkage and thresholding) algorithm which has been successfully used to solve many large-scale convex optimization problems. The objective function is:

\[ G(\nu) = ||\nu - \hat{\nu}||_2^2 + \lambda_1 ||D \nu||_1^{(\mu)} + \lambda_2 ||D^2 \nu||_1^{(\mu)} \]

where $\nu$ is substituted by each of the 5 extrinisic 2D camera parameters (x-/y-translation, x-/y-scale, and rotation), $D$ and $D^2$ are the first- and second-order forward differencing operators, $\lambda_1$ and $\lambda_2$ are term-weight hyperparameters determining the trade-off between camera motion fidelity and smoothness. The $||\cdot||^{(\mu)}$ operator is the Huber loss that approximates the $L_1$ norm while remaining differentiable everywhere (thus allowing use of FISTA). Original and stabilized camera motion parameters are shown below:

After optimizing for the smoothed camera motion, it is applied to the video sequence to obtain a stabilized result, free from annoying shakes and jitters.

A note: You’ll notice black borders in my implementation that are not visible in commercial video stabilization (on your phone for example). I have not applied automatic frame cropping, because I feel that seeing the video frames in their original scale (and consequently seeing the borders) is educational for those that have never studied video stabilization before. In reality, you’ll notice that this cropping is always done automatically on consumer devices. For proof, open your phone’s camera app, frame up a photograph on a fixed object then switch to the video mode…if video stabilization is enabled (as is default at least for iPhone) then you’ll notice the visible frame is immediately cropped (or “zoomed”) to exclude these borders after stabilizatoin is performed. Pretty interesting the first time you become aware of it.