Keying Fine Hair in After Effects Without Destroying the Edges

Every compositor has hit this: a greenscreen frame where an actor's flying hair turns into digital mush once the key is pulled. Fine hair is special because it lives at the limit of resolution: the pixels of a strand carry a mix of foreground colour (hair), green spill and background colour, and the keyer has to decide where the object ends. No keyer separates that signal perfectly — the real question is which parameters actually affect it and which are just habit. Below is only what Adobe's documentation, The Foundry's guide and camera specifications confirm.

Two things decide the pixel at a hair edge: the size of the sensor and how the camera writes colour. Here is what the specifications fix. RED KOMODO: 19.9 MP Super 35 mm global shutter CMOS sensor; effective pixels 6144 × 3240; sensor size 27.03 mm by 14.26 mm, diagonal 30.56 mm; dynamic range 16+ stops. REDCODE RAW frame rates: 6K (6144 × 3240) at 40 fps, 4K (4096 × 2160) at 60 fps, 2K (2048 × 1080) at 120 fps. R3D records 16 bits per pixel, and the monitor outputs carry 10-bit 4:2:2. Sony FX3: full-frame Exmor R CMOS sensor, 35.6 mm by 23.8 mm; 4K (QFHD) is 3840 × 2160 pixels, XAVC S-I DCI 4K is 4096 × 2160; internal recording at 10-bit depth with 4:2:2 colour sampling; dynamic range 15+ stops. Chroma and bit depth decide how much colour information survives to the hair edge. Adobe puts it directly: "Use footage with the least color subsampling possible — for example, 4:2:2 rather than 4:2:0 or 4:1:1". Apple ProRes 4444 encodes 4:4:4:4 RGBA with a lossless alpha channel of up to 16 bits; ProRes 4444 XQ supports up to 12 bits per colour channel; ProRes 422 HQ works with 4:2:2 at 10 bits, with a target data rate of about 220 Mbps for 4:2:2 at 1920×1080 and 29.97 fps, and ProRes 4444 about 330 Mbps for 4:4:4 in the same conditions. More colour samples per pixel and higher bit depth leave more information in the semi-transparent edge. But they do not replace resolution: the edge still gets only a handful of pixels, and the keyer works with what the camera recorded.

Keylight is After Effects' own keyer: Adobe ships it in the trial version and recommends trying Keylight before the built-in colour keyers, because Color Key and Luma Key are considered superseded by it. The key parameters, according to The Foundry's guide. Screen Colour: pick a representative green area of the screen with the eyedropper — this is the statistical base the whole matte is computed from. A mistake here cannot be fixed by adjusting Clip. Screen Gain: raising it keys more, but pushing it too far makes the background show through the foreground and destroys edge detail, fine hair included. It is a balance, not a maximum. Screen Balance: the value decides which of the two remaining screen-colour components saturation is measured against. At 100 % the smallest of them is used, at 0 % the largest, at 50 % their average. The Foundry guide notes: for blue screens a balance of around 95 % works best, for green screens around 50 %; if the key is not working well, try around 5 %, 50 % and 95 %. Clip Black and Clip White: an alpha value at or below Clip Black is set to zero, a value at or above Clip White becomes 100. These trim matte edges, they do not strengthen the key. Despill Bias: tells Keylight which colour to protect while removing spill; the documentation shows picking skin tones with the eyedropper for a blue screen. Edge Correction: Edge Softness softens the region considered to be the edge by a given number of pixels, Edge Grow expands it by a given number of pixels.

Primatte is Maxon's third-party keyer, not a built-in After Effects effect. Adobe's documentation does not describe its settings, so the numeric ranges we use in the studio are not published here: that is our practice, not a vendor specification, and there is no primary source to check it against. The practice can be described in words: pull the key as its own pass and do the spill suppression after it, so the suppressor does not eat the hair's own colour. Sample several areas of the screen, and do not click on hair edges — a semi-transparent pixel must not enter the solid-foreground model. If the key is fine but the edge still floats, move to alpha blur: Adobe explicitly recommends blurring the alpha channel after keying to soften the matte edges.

Supercomp is not a keyer but a third-party compositing tool: it works on an already keyed layer and mixes it with the background. It acts on the edge rather than on the alpha, so the result depends on how it builds the light wrap and the spill map. Because this is a third-party product with its own documentation, the numeric presets we use are not published here: that is a studio setting. The order of work is common for such tools: get a rough key with a separate effect and do not try to perfect hair edges straight away; lay the compositing tool on top; feed the background in as a separate input; switch spill suppression on locally, on the edge.

Every keyer ships with spill suppression: traces of the screen colour on the foreground edge, caused by light bouncing off the screen onto the subject. Adobe splits this across two effects. Advanced Spill Suppressor has two methods. Standard is simpler — it detects the dominant key colour by itself and needs less manual input; Ultra is based on the spill suppression in Premiere Pro's Ultra Key effect. The effect is off by default: you first sample the screen colour in Keylight, then switch it on if there is spill to remove. Spill Suppressor is the predecessor that Adobe moved into the Obsolete effects category in the June 2014 release and recommends against. It works with 32-bit colour. Key Cleaner sits in the chain between the key and spill suppression: it recovers alpha-channel detail lost to compression artefacts. If unwanted semi-transparency appears along edges that should be sharp, restrict Key Cleaner with an effect mask and a little mask feathering. The order Adobe proposes for well-lit footage on a colour screen: Color Difference Key → Spill Suppressor or Advanced Spill Suppressor → Matte effects. For footage with several colours or an unevenly lit screen — Color Range; for dark areas and shadows — Extract Key on the Luminance channel; for a static background — Difference Matte. If the built-in suppression is not enough, Adobe suggests another move: apply Hue/Saturation to the layer and decrease the saturation value to de-emphasise the key colour.

Light wrap is the phenomenon where background light wraps around the edges of a foreground object. Fine hair depends on it more than any other compositing element: a strand is thin enough for light to pass through it and around it, and without wrap the edge looks detached from the background. A manual light wrap in After Effects is assembled from layers and standard effects: a copy of the background above the keyed foreground, a blur of that copy, Screen or Add as the blend mode, the foreground alpha channel as a track matte, and an inverted matte so the blurred background only reaches the edge. Adobe separately notes that blurring the alpha channel after keying softens the matte edges and improves the composite — that is the technical basis of a soft edge. Testing the result is simple: switch the wrap layer on and off. If you see a jump, the hair has stopped floating above the background and the wrap works. We do not quote numeric radius or opacity presets: that is a studio setting, not a specification.

Sometimes a greenscreen is lit so badly that keying does not work at all. Adobe puts it plainly: if the background does not have a consistent and distinctive colour, you cannot remove it with keying effects, and you need rotoscoping — drawing or painting frame by frame. The thresholds worth keeping in mind: - A background that is neither consistent nor distinctive in colour means roto, not keying. - A screen mottled with shadows: Adobe advises lighting the colour screen uniformly and keeping it free of wrinkles, and shooting with the least chroma subsampling possible. - Lots of noise and compression artefacts: a slight blur before keying reduces noise and artefacts and improves the key. - Footage shot with no thought for the compositor: a high-quality key often requires several keying effects in sequence and careful adjustment of their properties, especially when the shot was acquired without the compositor's requirements in mind. - A garbage matte is not roto: mask the subject roughly, key inside the mask, then use Matte effects to fine-tune. But if a perfect garbage matte takes a lot of time, you lose the time advantage of keying over rotoscoping. The roto tools in After Effects are Roto Brush and Refine Edge. Refine Edge improves an existing matte by creating partial transparency where fine detail lives — hair and fur. Segmentation propagates 20 frames forward and 20 frames backward from the base frame; work with these tools at Full resolution, and keep the segmentation boundary within a couple of pixels of the edge you want. Switching between Roto Brush and Refine Edge is Alt+W on Windows or Option+W on macOS.