Kinect volumetric capture

In 2015 I held the title of “Kinect Supervisor” on a very pioneering and challenging movie called Perfect World.

We used multiple Microsoft Kinect to capture an animated point cloud of nearly every scene of the movie. For every shot 3 kinects were placed in different positions around the set and their output was later combined to create a full volumetric capture of the characters. Everything was also recorded using a traditional RED camera and its high resolution output was used to assign color information to the point cloud.

The system allowed us to give a unique look to the characters.

Virtual production prototype

This is a proof of concept for a virtual production environment in the Unity engine.

In 2017, when this video was recorded, Unity was lacking any native virtual production features, but I was determined to use a high end game engine with proper lighting during the motion capture phase, especially for the virtual camera.

I ended up capturing the motion of the character and the camera work separately. The character capture and retargeting was done in OptiTrack Motive and Motion Builder and it was then imported in Unity via FBX.

For the camera movement I streamed the camera position in real time from Motive to Unity and I used a custom script to record it to an animation track.

This way I was able to see final pixels, directly in my virtual camera viewfinder and make adjustments to the lighting and blocking in real time.

Variable frame rate for dramatic effect

We are all familiar with the “soap opera effect”, which is simply video playing back at frame rates higher than 24 or 25fps. Modern cameras and displays are capable of extremely high frame rates, but the few experiments with high fps cinema have largely disappointed audiences so far (The Hobbit trilogy is especially infamous).

Movies at higher frame rates feel weird and off. It’s not clear if this is just due to audiences being used to a century of 24fps or if there is some intrinsic property of a less fluid image that makes it more suitable for narrative filmmaking.

But higher frame rates do have advantages, as demonstrated by videogames that have always chased the most fluid images, to amplify the realism and viscerality of the action and to improve visual clarity. And it’s in these situation that I believe higher frame rates have a place in traditional filmmaking. The problem with experiments like The Hobbit is that they shoot and project the entire film at say 48fps. If the action scenes might benefit from the effect, this sticks out like a sore thumb in for instance dialogue scenes, that don’t need any extra information or fluidity to work.

So my proposal is to treat frame rate like aperture or shutter speed. Something that can be changed based on artistic and dramatic requirements. Saving Private Ryan did just that with shutter speed, shooting action scenes with a narrow shutter to introduce a staccato effect that highlights the frenzy of battle.

But abruptly changing the frame rate might be jarring, so we need a way to smoothly ramp the frame rate up or down, ideally hiding it during some camera movements, kinda like an iris pull.

Here is my proposed workflow.

In my test i staged a very simple scene: an actor first sits down and reads a newspaper out loud, mimicking a dialogue scene. Then the actor stands up, and stages a fist fight with a C-stand, with the camera moving freely and frantically around him, mimicking an action scene.

I chose to stay within the 25 and 50fps because those frame rates where readily available on my camera, but everything applies to 24/48fps too.

I shot the entire take at 50fps with standard 180 degrees shutter (1/100s). The output video will be a 50fps video as well.

For the first part (the “"dialogue” scene) we want a classic filmic look, so we simply throw away half of the frames and we repeat the remaining frames twice (remember, we are still going to output a 50fps video), therefore giving the scene the feel of a 25fps movie. Yes, the motion blur will be less, since we shot at 1/100s shutter, but we can compensate for that using optical flow.

For the following “action scene” we don’t intervene and we leave the video as it was shot: 50 fluid frames per second.

To transition between the two sections we can use similar optical flow techniques to smoothly ramp up the frame rate.

Basically the output is a regular 50fps video that many displays and projectors handle easily, but the actual frame rate perceived by the audience is altered by repeating certain frames.

Here is the example.