🚀 go-pugleaf

As part of a multimedia project here at Southampton, I'd be interested
in looking into the problem of temporal segmentation of image sequences.  
By which I mean that given a video sequence, I would like to be able
to label scene and viewpoint changes (fairly) automatically.

I would be interetsed in any references to similar work, but my
initial thoughts would be along the lines of understanding what
constitutes a significant change, and how video sequences are
constructed.  

From a bottom-up viewpoint, we could categorize the principal types of 
sequences:

  static camera 	: the camera position remains fixed, objects may
		     	  move freely within the field of view.

  zooming camera  	: as above, but the camera may zoom in or out,
			  probably centered on an object.

  panning camera	: the camera pans such that most image information
			  passes in one direction, once again probably
			  while one subject is targetted.

  pan and zoom		: combination of the above two.

  new viewpoint		: there is a change in camera/viewpoint, which
			  may be a view of the same scene, or of
			  something completely different.

Clearly, one approach could involve some sort of optical flow detection,
perhaps combined with just some simple image differenceing.
However, whatever the method for performing categorization along the
lines of the above, there is also a need for some understanding of
the way clips fit together between and within viewpoints.  

Top-down, a video may have stories/programmes constructed of "acts"
in which there are "scenes" in which there may be changes in viewpoint;
at the lowest level is the single camera view, which (as above) may
be static or pan/zoom.

Note that the tracking of objects within scenes is a problem we are
addressing separately (reasonably successfully) but I think the ability
to recognise changes in scene, and perhaps in particular revisiting 
previous scenes (viewpoints) could be very useful.  Perhaps either
"skill" could help guide the other.

In some "contrived" situations, you may have knowledge to hand of the
nature of the scenes, such as with a newscast or a sporting event.
In such cases rules can be expressed that define how viewpoints
may relate to each other.  In the more general case, it would be of
interest to (say) be able to "segment" a sequence of two people
having a conversation, where perhaps the camera switches between
each person.

Anyway, it's a topic I haven't seen much on in any literature.  It was
mentioned briefly to me by a visiting researcher from Michigan, with
whom I will discuss it further, but I'd be interested in any general 
feedback ...

Cheers,
	Tim