Motion Tracking – Screen Replacement

Topics covered in this After Effects tutorial:

Planar Tracking, Screen Replacement, Planar Surface, Refining a Track

Exercise Preview

Exercise Overview

In this exercise you’ll track a device screen in Mocha AE so that you can replace the screen with an image or video file.

Previewing the Final Video

1. Let’s see a preview of what you’ll be making. If you’re in After Effects, keep it open but switch to your Desktop.
2. On the Desktop, navigate to Class Files > After Effects Advanced Class> Motion Tracking - Screen Replacement > Preview Movie and double–click Laptop Screen Replacement.mp4.
3. Also preview Person Using Mobile Phone.mp4.

4. Notice the following:

   • There are images and/or video in the screen of both devices..

5. Replay the video if you need to, and close it when done.

Setting Up the Workspace

1. In After Effects, go to Window > Workspace > Standard.

2. Choose Window > Workspace > Reset “Standard” to Saved Layout.

3. If the After Effects window does not fill the screen, maximize it:

   Mac:	At the top left of the window, click the green button (the third button).
   Windows:	At to the top right of the window, click the Maximize button (the middle button).

Getting Started

1. In After Effects, if you have a project open, choose File > Save.

2. Choose File > Open Project and:

   • Navigate to Desktop > Class Files > After Effects Advanced Class > Motion Tracking - Screen Replacement.
   • Double–click on Screen Replacement - Started.aep to open it.

3. Choose File > Save As > Save As and:

   • Name the file Your Name - Screen Replacement.aep and
   • Save it into Desktop > Class Files > After Effects Advanced Class> Motion Tracking - Screen Replacement.

Set Up

In this example we have provided you with a project and composition to start with.However, normally you’ll have to create one yourself. When tracking it is necessary to make sure that the dimensions of the video you are going to track match those of your composition.

When creating a project from scratch, follow these instructions:

1. Drag your main video (the one you want to track) into the empty Timeline panel or onto the Create a New Composition button located at the bottom of the Project panel.

2. Rename the layer so that you can remember this is the one you want to track.

Applying the MochaAE Effect

MochaAE is an effect and as such can be found in the Effects & Presets panel as well as the Effect menu. Once applied it can open a copy of the MochaAE plug–in that runs inside of After Effects.

1. Select the video layer that you want to track.

2. In the Effects & Presets panel type Mocha into the search field.

3. Double–click on the Mocha AE effect.

4. In the Effect Controls panel click on the Mocha logo to open Mocha.

Dealing with Occlusion

In motion tracking the term occlusion is used to describe a situation where some other element covers the object you are trying to track. Maybe you’re trying to someone walks in front of your

Tracking in MochaAE

Mocha is made by BorisFX not Adobe so the terminology, keyboard commands, and even concepts like layer are different from After Effects.

The basic process for tracking in MochaAE:

• Create a spline (like an AE mask) and this creates a layer.
• Track the footage and if necessary make adjustments to layer or properties while tracking.
• If necessary adjust track when complete.

1. In Mocha, locate the frame where you can best see the element you want to track.

2. Create a layer around the element you want to track.

3. Reveal the Planar Surface and make sure it’s positioned in the middle of the element you are tracking.

4. Adjust the four corners of the planar surface to line up with the four corners of the screen you want to track.

   NOTE: While the layer can and usually should be larger than the sceen you are trying to replace, the planar surface needs to match the screen exactly. The planar surface will form the shape and positioning of the replacement screen.

5. If necessary, adjust Minimum % Pixels Used.

   NOTE: The application will automatically assign a value. Higher numbers will use more pixels in the layer and will be more precise, but processing can take longer.

6. Track the layer.

   NOTE: If the layer you are tracking moves away from the content you are attempting to track this is called drift. Slight drift can be corrrected after the tracking is completed using the Adjust Track module. However if the layer moves around a lot during the tracking process it is best to stop tracking and adjust it. Usually either changing the size of the layer or Minimum % Pixels Used value will fix minor issues.

7. When tracking is complete, save the Mocha project by choosing File > Save Project.

Adjusting the Track

Minor drift that occurs when tracking can be corrected using the Adjust Track module.

1. Make sure the MochaAE application is using the Classic workspace.

2. Switch to the AdjustTrack module. It’s located at the bottom right when you are in the Classic workspace.

3. Make sure the tracked layer you want to adjust is selected.

4. Find a frame in your video that represents a good position and shape for the Planar Surface, this is going to serve as your Reference Frame.

   NOTE: You’re looking for a point where the planar surface is good, where it covers the screen you are trying to replace well and before it starts to drift.

5. Set the Transform Type and click the Set Points button. For a screen replacement you should use Perspective as your transform type.

   Transform Type

   Translation: changes in position (x & y axis) this will create a single reference point.

   Scale/Rotation: changes in the size (scale) and angle (rotation) of objects will create two reference points to measure the change between them.

   Shear: sometimes called distort in other programs this transform type sets three points which is uses to detect vertical and horizontal warp.

   Perspective: sets four points for a full 4–point corner pinning style adjustment.

   Point can be adjusted after they are created and you can also manually create point using the Add New Point button.

6. Move the Reference Point to distinct areas on the surface you are trying to track. They need to be distinct from each other because you need to find these points again and they don’t have to be inside of the planar surface.

7. Move the playhead to the end of the tracked video and Adjust any movement in the reference points.

8. Move to a point on the timeline approximately between the two keyframe and reposition the reference point to correct any drift. It is important to keep the reference points on the same pixels throughout the track.

9. Continue to to this placing keyframes in the middle of existing keys until you have corrected for the drift.

10. Test the shape and positioning of the planar surface through out this process using the Insert Clip option to see how the surface distorts.

11. When done, save your project using Cmd–S (Mac) or Ctrl–S (Windows) or File > Save Project.

12. Close the MochaAE window.

Applying Tracking Data to a Layer

Once you have tracked using MochaAE you still have to take that tracking data and apply it to a layer in your After Effects composition.

1. Back in the After Effects interface create a new Solid layer by choosing Layer > New > Solid or pressing Cmd–Y (Mac) or Ctrl–Y (Windows) on your keyboard.

2. In the Solid Settings dialog press the Make Comp Size button, set the color to whatever you like and press OK.

3. Select the new layer and Ctrl–click (Mac) or Right–click (Windows) on it and choose Pre–compose from the menu.

   NOTE: You can also pre–comp a layer by pressing Cmd–Shift–C (Mac) or Ctrl–Shift–C (Windows)

4. In the Pre–compose dialog box:

   • Rename the new composition to something descriptive, we tend to use Screen Comp
   • Click on the radio button for Leave all attributes…
   • Click OK to create the new comp.

5. Select your tracked layer and in the Effect Controls panel locate the Tracking Data properties.

6. Click the Create Tracked Data button, select the MochaAE layer with your screen tracking data, and press OK.

   NOTE: it’s important that you make sure you don’t accidentally apply the data from your occlusion layer if you created one.

7. Set Export Option to Corner Pin or Corner Pin (Support Motion Blur)

   Corner Pin vs Corner Pin (Support Motion Blur)

   Corner Pin and Corner Pin (Support Motion Blur) can both be used in screen replacement though there is a difference between the two. The corner pin option just applies to the target layer a corner pin effect taken from the corner data of the MochaAE planar surface. The Corner Pin (Support Motion Blur) option in addition to applying a corner pin effect to the target layer also applied position, scale and rotation keyframes taken from the transform properties of the planar surface.

8. Set Layer Export To property to the pre–comp you created earlier and make sure the effect is set to target the Source properties of the layer.

9. Click Apply Export

10. Preview the Timeline and you should now see the pre–comp screen following along with the movement of your device screen.

    NOTE: If you screen isn’t tracking properly, you can delete the corner pin effect, reopen the MochaAE, adjust your tracking there, and then reapply the tracking data to your layer.

Editing a Nested Comp (Precomp)

Unlike when you apply transform tracking data for screen replacement you will want to use a pre–comped layer. This will allow you to easily swap out the device screen whenever you need to.

1. Double–click on the pre–comp layer

2. Add your new screen graphic above the solid you previously created.

3. Use Layer > Transform > Fit to Comp to make sure your new layer completely fills the composition area.

   NOTE: For best results you want your screen graphic to be the same aspect ratio of the screen you are trying to replace. If it looks distorted inside the pre–comp that’s fine, it will look OK when you are in the parent composition.

4. Return to the parent composition.

5. Save your AE project.

Removing a Greenscreen from your Device

If you have something that is going to be moving in front of the screen you are trying to replace like a hand swiping or clicking on a device, it can be a massive time saver if you use a color screen on your device for easy keying.

1. In the Timeline panel, Move your new screen layer below the source video layer (the one you tracked).

2. Select the layer you want to key out.

3. In the Effects & Presets panel, type keylight into the search field.

4. Double–click on the Keylight + Key Cleaner + Advanced Spill Suppresser preset.

5. In the Effect Controls panel, use the Screen Colour pipette to select the color you want to remove.

6. Adjust Screen Gain to increase or decrease the tolerance of the color removal.

7. Adjust Screen Balance if necessary.

8. Switch to the Screen Matte or Status preview mode.

9. Open the Screen Matte properties and adjust Clip Black, Clip White, Clip Rollback as necessary. The goal is to have the foreground (what you want to keep) completely white, while the background (what you want to remove) is completely black.

10. Switch the View mode back to Intermediate Result and adjust Screen Shrink/Grow if necessary to remove any remaining fringe color.

11. Turn on the Advanced Spill Suppressor effect.

    NOTE: For a complete list of the properties of the Keylight effect see The Foundry’s help page: https://learn.foundry.com/nuke/content/reference_guide/keyer_nodes/keylight.html

12. Save your file.