Using the Roto Brush Tool

Getting the Project Files

1. Download the project files
2. After the download has finished, be sure to unzip the file if it hasn’t been done for you. You should end up with an After Effects Audio Spectrum folder.

Project Overview

In this tutorial we'll be focusing on rotoscoping with the Roto Brush tool. Rotoscoping is the process of drawing on top of your footage, and there are different reasons you will want to do this. Sometimes you want to use rotoscoping to create a cartoon effect such as films like A Scanner Darkly or Waking Life. Other times you may want to use it to isolate parts of your video footage so that you can composite or color correct them.

In this tutorial we will focus on the second use of rotoscoping. We will use it to isolate a bird so we can place an animated text graphics behind it.

Let’s get to it!

Overview: Making the Roto Brush Selection

1. Drag and drop the footage from the Project Panel onto the Timeline. 
2. Hit Return (Mac)/ Enter (PC) to rename it.
3. Double click the footage layer in the Timeline to open it in the Layer Panel. 
4. Remember: The Roto Brush tool only works in the layer panel
5. Click the Roto Brush tool, located on the top toolbar. The icon is a brush over a human figure.
6. Go to Windows > Workspace > Paint to get a side by side comparison view of the Main Comp and the footage layer.
7. Click and drag around the figure in the footage to select it.
8. To clean up the selection, hold down Option (Mac) or Alt (PC) to subtract areas.
9. Click and hold on the Roto Brush tool to access the Refine Edge tool. This is for cleaning up organic edges, like hair, fur, feathers and more.
10. Click and drag the Refine Edge tool over any areas that need it.
11. Hit Spacebar to preview the selection so the program can process it. The time will vary depending on RAM, processing power, etc.

Overview: Adding Background Effects

1. Drag and drop another copy of the footage layer from the Project Panel onto the Timeline. Make sure it’s below the Rotoscoped layer.
2. Hit Return (Mac)/ Enter (PC) to rename it.
3. In the Effects and Presets panel, enter Tritone effect. Drag and drop this onto the new background footage layer.
4. Type in Gaussian Blur into the Effects and Presets panel. Drag and drop that as well onto the background footage layer.
5. In the Effect Controls panel, change the Tritone Midtones to grey. 
6. Change the Tritone setting Blend With Original to 100%.
7. With the Playhead at the start, hit the stopwatch next to Gaussian Blur. Do the same for Tritone.
8. Move the Playhead over a few seconds.
9. Add a blur to your layer with the Gaussian blur effect.
10. Change the Tritone setting Blend With Original to 0%.
11. Any additional layers, objects, text, etc. can be placed below the Rotoscoped footage and will still remain visible.

Video Transcription

Hello, this is Tziporah Zions for Noble Desktop, and in this tutorial, I'm going to show you how to use the Roto Brush in Adobe After Effects. We'll be learning how to use the Roto Brush to easily separate elements from our footage. This allows you to place individual effects on each piece or place objects underneath video footage. Here's what our finished video is going to look like.

You can see how this parrot is separated out from the rest of the footage here. Rotoscoping is a great technique for separating foreground elements from background elements. The Roto Brush is cool tool in After Effects, that near automates and otherwise tedious manual process. By painting and adjusting with this brush, we can separate out our elements much faster and easier than if we would if we do the old school frame by frame method.

We're going to be using just this one piece of video footage over here and this precomp animation I've made for you as well. So not not much to sift through in the project. The file will be available for download in the video description below.

So let's get started. first off, let's drag and drop our footage into the empty timeline over here. Thank you. Click on the footage layer in the layer stack over here. And if you have a Mac, you're going to hit Return and a PC like mine will hit Enter and we're going to name this, which is called Roto Brushed Parrot. Now double click on the footage to go inside the layer.

This is where we're going to be doing most of our work with the Roto Brush. Now here's something of the cool. Head up to Window, Workspace. Look for Paint, and the reason why we're going to be doing this is because this is going to rearrange the interface to give us all the painting tools like brush sizes, side by side comparison, view our footage and so on and so forth. And you could check it out.

Now we can simultaneously work on a piece of footage and also see update on the main comp over here. So I'm actually going to shrink it a little bit. So we get a bit more of my layer footage over here, right? Check out this little icon up here. You see, it's like a little person with the brush on it.

So this is the Roto Brush. So with our Playhead at the origin right here at the start
we are going to click on the Roto Brush and now let's click and kind of like paint an outline, so to speak around the parrot. It really doesn't have to be exact. You know, just it's all about getting it done at this stage. So as you can see, the selection tools are pretty generous, so we're going to refine it. So hold on AKT on a PC, CMND on a Mac. And so the Roto Brush is red, we're actually going to slice up these chunks from a selection.

You know, bit by bit until we get more of an exact selection. All right. Once that's done, I take a look down here. You can see it's pretty small, but there's like a little green line over here. And what this bar is , is that this defines how many frames the Roto Brush is going to process. So we're going to want to constrain the amount so as not to overwhelm the program.

So grab the end point over here and drag it to the left, drag to the left. We're going to put it at around three seconds. Yeah. About three seconds. That's what we're going to do now, we are going to refine your selection so click and hold on the Roto Brush tool here, and this is Refine Edge and it's really what it sounds like, you refine. Some edges are pretty useful for things like fur, hair feathers, you know, fine organic edges.

And with that, I'm just going to click and drag and paint over some of the more hairy edges, so to speak. All right now, we are going to be working with the effect controls. Now they should be docked in here on next to composition. And we're, you know, composition window, we are going to actually grab and I want to pull them out so I can work with them a little more easily. So I'm going to grab this hamburger and undock it.

Let's see. I will float you over here. I think that's all right. First thing that we're going to do. Make sure that the version is up to 2.0. And then we're going to be heading through these settings now if yours is not open, toggle open. Open Roto Brush Matte. And let's be heading through our selections. So Feather over here refers to not the feathers on the bird, but, you know, kind of like soft and blurry you want edges to be

We're going to put that about nine, you know your project might be a little different. This is what's going to work for mine personally. Contrast is how much of a, you know, a hard edge you want between the values of your selection and what's around it. So it's going to make it a little more defined.

I'm going to put mine at around 75. Let's see and then shift edge really again, what it sounds like, it moves the edges of selection in and out. I'm going to try with negative 95 for this. All right. All right, we're getting there. Let's see what else. See a little bit of cleanup we're going to do over here, but for the most part, this is what I want to see.

All right now, we are going to hit spacebar or whatever you have set for preview on your machine. And we're going to have the program process our selection, give the program a minute to get through all those frames. And you should note that the speed at the Roto Brush runs at is entirely based on the computer set up RAM, processor, video card, you know, so on, so forth. All right, you can see that our parrot now isolated from the background.

So what we're going to do, we're going to finish this up with some effects. So let's set up a background as well as some cool transition effects next. So let's see. Let's go over to our project one over here. Grab another copy of the original footage. We're going to put it right underneath our Roto Brush parrot right here. And let's see. Let's, you know, hit Enter on a PC or Return on a Mac, and we're going to call this Background Video.

All right. Let's have it line up a little more with the footage. There we go. All right, then next up is the transition effects with blurring and grayscale, so we're going to head up to Window, Effects and Presets. Here it is. So mine pops into the left hand side over here. If yours appears different, you know, just dock it wherever it feeks comfortable. All right. So we are going to be looking for Gaussian Blur.

Here we go. Just let's see. Let's use this Guassian Blur, drag and drop it onto our background video footage. Nothing's going to happen just yet. And let's see. So the first thing that we're going to do with this one, we're going to be hitting Repeat Edge Pixels over here.

This option here, you could see the whole sentence over there to start with, and that's going to have the edge pixels of this blurred out effect. also be included in the blurred out effect, but you'll see it when we get to that. And then the next effect, let's see. Let's head back to effects and presets. I'm actually going to pull this out and going to undock and drag it over here so I could have access to multiple panels.

Let's put in Tritone. Drag that over to our backgrounds footage, great. So we've got two of these things. And the next thing that we're gonna be doing is click that brown box over there. Drag it over to the grey because we don't want this to have a sepia tone. We want it to really just have more of a grayscale look. And then we're going to keyfram this thing. Now with your Playhead at the origin hit the stopwatches next to Blend with Original and Blurriness.

All right, and then Blend with Original should be all the way to 100%. So basically, we're not really seeing any effect just yet. So the next thing we're going to grab our Playhead, get it to be like one second in about ish, you know, and the next thing that we're going to do is we're going to change blurriness to 20, Blend with Original to 0. And once the machine catches up we're going to be seeing those effects. All right, and then finally go over to project panel, grab this text precomp over here, drag and drop it in between those layers so it's a bit of a sandwich.

And as you can see now, the Rotoscoped parrot is hovering above the animation right below it, and it retains its own colors on, you know, own values, while the other effects go on all around it. So let's preview that.

All right. As you can see, when I scroll through the playback, I could see that the parrot is moving, the backgrounds moving and text moving. But they are all segmented and separate from each other, with different effects applied to each one. Now we're all done. So as you can see, rotoscoping is really cool. The technique can be used not only on video footage, but on still images as well. It can be repeated on a single piece of video as well to separate multiple objects through the background and all sorts of different effects can be applied individually to each rotoscripted element.

You can try combining text with totally different footage or video or still images to the back or foreground of any future rotoscoped project. All right. That's all for this tutorial. I hope you've enjoyed learning how to use the Roto Brush in Adobe After Effects. This has been Tziporah Zions from Noble Desktop.