Sure, but the technique used was also very trivial. Just aligning and averaging all the video frames basically leaves a mosaic-pixel-sized blur on everything (assuming the camera movement is uncorrelated with the mosaic grid).
You can get much further by applying deconvolutions and using more math. I've been meaning to put some time into this myself but never got it off the ground.
I wonder if the author would be open to making e.g. the car data available?
First challenge is going to be figuring out the grid alignment in the stabilized frames. But I have a decent idea how to tackle that, which I'll hopefully get to tomorrow!
Yes, I've also always felt there must be ways to extract more data from a moving clip, precisely because of the effect he explains, but then it seems that just superimposing the images doesn't actually extract that information, at least not all of it.
But I wonder how to actually do it, do you have concrete ideas for a simple algorithm?
If you can figure out, for each frame, which sets of (pre-aligned) pixels have been averaged, you can create a large system of equations that captures those relations and solve it to find the unblurred pixel values.
Depending on camera movement (and whether you might get "ground truth" information from pixels entering and leaving the areas near the borders) the system will be more or less well-conditioned. I'm going to try this for the data the author graciously provided and report back!
I seem to recall that there used to be a video showing this approach in action. As input it took a video panning across a shelf full of books where the resolution was so low that the titles were illegible. And as output it produced a video with higher resolution and all the titles easily readable. Unfortunately I can't find that video any longer.
Yes, it all boils down to point spread functions. In the mosaic case, the PSF varies locally (per pixel) and temporally (in different video frames). The paper you link similarly details how they figure out the PSF. You can theoretically also do the entire thing without knowing the PSF, which is called blind deconvolution: https://en.wikipedia.org/wiki/Blind_deconvolution
> I wonder if the author would be open to making e.g. the car data available?
Interesting - this is same incorrect use of e.g. that the author made in a couple of places. Contrary to (apparently popular) belief, "i.e." and "e.g." can't simply be used as direct replacements for their English equivalents.
"e.g." is used to introduce one or more examples that satisfy a previously provided general form, for example:
I prefer fruit, e.g. apples or pears, over vegetables. Apples and pears being examples of fruit, not that an example is needed in this case, but for the sake of simplicity.
In the former example, "the car data" is not an example of "making".
"i.e." follows a similar rule. If there are exceptions for either, I'd be interested to know of them.
Interesting, thanks for bringing this to my attention! Do you have any reference that explains this rule? I noticed on Wikipedia that introducing multiple examples used to actually have a different abbreviation (ee.g. or ee.gg.), so clearly something is already lost in translation here...
In any case, what I wrote is really just a shorthand for more cumbersome formulations (like "...open to making your data, e.g. the car [data], available?" - that would hopefully be correct?), and reducing text is the whole point of using an abbreviation in the first place. But I'm open to striving for more consistent usage, so if you can refer me to some kind of authority on how to mix Latin abbreviations with English text, I'd be curious about it!
You can get much further by applying deconvolutions and using more math. I've been meaning to put some time into this myself but never got it off the ground.
I wonder if the author would be open to making e.g. the car data available?