When it comes to making masks and selections, no other tool in Photoshop’s vast repository can produce the clean, graceful outlines that you can create with the Pen tool. Rather than laying down pixels, the tool allows you to define connect-the-dots paths that you can save and edit at will. You can then employ these paths as vector masks. Or combine them with alpha channels (such as hair masks) to cleanly extract complex shapes from their surroundings. That is to say, they’re wicked powerful.

Mastering this amazing tool demands a different approach from the other selection tools in Photoshop. But it offers a rich pay-off. In this tip, we’ll explore the Pen tool in thrilling detail.

Today’s tip comes from Chapter 27, “Everything About the Pen Tool,” from the course Photoshop CS5 One-on-One: Mastery on lynda.com.

When you want to precisely trace a well-defined contour, free of the kind of bumpy artifacts (what Deke refers to as “pixel dirt”) that will usually occur when you make selections with commands like Color Range, the Pen tool is the perfect option.

The Pen generates resolution-independent, reusable, scalable outlines that are stored in their own panel—the Paths panel. Paths are objects that are distinct from pixel-based layers, masks, and channels. You can use the Pen tool to trace gracefully curving edges, or make spiky outlines that change direction sharply. In fact, the Pen tool is perfect for tracing the outline of anything with a well-defined edge such as product shots, clothing, arms, legs, or the silhouette of a face. But it’s not a universal selection tool. There are much better tools for tasks like extracting hair. Later in this tip, we’ll look at how you can outline a face with the Pen tool and combine it with a hair mask.

The Pen tool works in a way that may at first seem counter-intuitive, so mastering it requires some practice. But one of its payoffs is that it can produce gracefully curved lines that are free of hand jitter. Which is impossible, for example, with the standard Lasso. If you have explored the Tools panel, you may have noticed the Freeform Pen tool nested inside the same flyout menu as the Pen tool. The Freeform Pen is essentially a version of the Lasso that generates paths instead of selection outlines, but those paths are a mess. Deke doesn’t recommend it and I won’t be covering it here.

How the Pen Tool Works

The word “pen” in the name of the feature is perhaps a bit deceptive. That evokes the notion of using the Pen tool to lay down “ink” in the same way that you make marks with a brush or a pencil. Instead, you work in a completely different way, using it to place inflection points (generally referred to as anchor or control points) on the canvas while Photoshop draws connecting lines between them to define an outline.

Rather than thinking of those connecting lines as ink or any sort of mark, it is a bit more useful to think of them as stiff wires that are fastened to the control points. In other words, the Pen tool lays down a set of points that give shape to the outline which is essentially a 2D wireframe. Once you have created that wireframe, you can use it as a vector mask, convert it into a pixel-based selection, fill the pixels inside it, or apply a stroke to the pixels that lie along its path.

To refine the shape of the outline, you move the points or manipulate handles on the points. Moving the control handles essentially torques the wires to make them flex. There’s a bit more to it than that, and folks often get mixed-up learning to operate the control handles, but we’ll flesh out those nuances soon enough.

Activating the Pen tool

Tap the P key or use the button in the tools panel to activate the Pen tool. Once the tool is active, make sure your Options bar is set as shown:

Reading from left to right, the following settings are selected.

• Pen mode is set to Paths (middle icon in the first group)
• Pen tool is selected
• Auto Add/Delete is checked
• ‘Exclude overlapping path areas’ option is selected

Creating a path exclusively with Corner points

In its simplest application, the Pen tool behaves in a fashion similar to the Polygonal Lasso tool: you click to set down corner points with straight connecting segments. Two key differences are that the Pen tool produces a path instead of a selection, and you can adjust the position of the corners after you make your initial outline.

Click to place several points on the canvas. Even though the Pen tool does not respond to pressure and tilt the way that many of the painting tools do, you will probably find the ergonomics of a graphics tablet to be more comfortable. Wherever you see references to the mouse in this tip, you could also use a stylus. Don’t worry about closing the path; we’ll do that shortly.

Key points:

• Be sure to click (press and release the mouse button), and do not drag as you place each point.
• Don’t worry about precise placement of points. You can reposition them at any time.
• To constrain the connecting lines to verticals, horizontals, and diagonals,  Shift-click as you place the next point. (Points 4-6 above were created that way.)

Making corrections:

• Use the arrow keys on your keyboard to reposition the most recently-placed control point: tap the key to nudge one pixel, or hold the shift key and tap to bump 1 pixels.
• Use Alt + Ctrl + Z (Mac: Option + Command + Z) to step backwards and undo each point in reverse order.
• Hit the Backspace key (Mac: Delete) to delete the last point you placed and select all remaining points in the path; immediately hit Backspace/Delete a second time to delete the entire remaining path.

You can select and reposition other points on the path, too. Use the White Arrow (Direct Selection) tool to select points and then drag them or use the arrow keys to reposition.

• Tap the A key to activate the Arrow tool, and use Shift + A to cycle between the White and Black Arrow tools.
• Click any hollow point with the White Arrow tool to select it (the selected point will fill in). Drag to reposition. Use the Shift key to constrain movement.
• To deselect the entire path, click outside the path with the White Arrow tool. When the path is deselected, no control points appear.
• Click the path with the White Arrow tool to activate the path. Control points will appear, but none will be selected (all points will be hollow).

You’ll use the Black Arrow (Path Selection) tool a lot less frequently. It selects all of the points on a path at once. That’s useful when you want to drag, nudge, or bump the entire path to a new location. Since you can convert the Pen tool to the White Arrow tool as needed, you can select the Black Arrow tool in the Tools panel and then switch back to the Pen tool. When you need the Black Arrow, just tap the A key to activate it.

Closing a Path

If you haven’t stopped to reposition any of the points in your line, the path will be ready to close as soon as you bring the cursor back over the first point you created. If a point somewhere in the middle of your path is currently selected, you will need to “pick up” one end point and then click the other end point to connect the two:

1. Activate the Pen tool (tap P) if it is not already active.
2. Click one end of the path. Its control point will fill in.
3. Position the cursor over the other end point. The cursor will show a circle to indicate that it’s about to close the path.
4. Click to close the path.

The Paths Panel

It’s worth noting that it doesn’t matter which layer is active while you define a path. You’re not defining the path within the layer, as would be the case with a brush; you’re defining it within the Paths panel. As you lay down points to define a new path, Photoshop updates the Work Path that appears in the panel. If you don’t see the Paths panel on your screen, you can reveal it by selecting Window > Paths from the menu bar.

The Work Path is essentially a scratch pad. If you deselect the Work Path and click in the image again with the Pen tool, the previous Work Path will be discarded and Photoshop will start a fresh one. The Pen Tool’s cursor will indicate that it is about to start a new path by displaying an X in its lower right corner.

One way to unwittingly deselect the Work Path is to hit the Enter key (Return or Enter on the Mac). The Black Arrow tool displays a checkmark in the Options bar. Clicking this will also deselect the Work Path. If you click the gray area at the bottom of the panel, you’ll deselect the path. Whenever the path is deselected, it will not be visible in the image editing area. To redisplay the path, click on it in the Paths panel. You’ll then use one of the Arrow tools to select all or part of the path for editing.

You can save a path that you want to preserve by double-clicking on the Work Path and entering a name into the dialog that appears. Saved paths are part of your document, and are saved inside all file types, even JPEGs.

Whether you’re editing the work path or one that you’ve saved, its outline is only visible when it is selected in the Paths panel. You can also hide paths with the View > Extras command.

Smoothing Squared-off Shapes

Outlining your selections with straight-line points can be useful in a number of cases, but it’s likely that you’ll quickly encounter applications where you’ll need smooth curves instead of sharp corners. You can rough-out your shape with straight-line points and then replace them with smooth points. In the example shown here, the straight lines are most problematic along the arc that curves into the crease.

You can trace the arc more precisely with smooth points instead of the corner points we’ve used so far. To do that, you can delete the problem corner points and replace them with smooth points using the White Arrow, Delete Anchor Point and Add Anchor Point tools. Rather than taking the mouse over to the Tools Panel, you won’t have to if you’ve activated auto Add/Delete in the Options bar.

1. Tap P to activate the Pen tool
2. Hold down the Ctrl key (Mac: Command key) to convert the Pen tool to the White Arrow tool on the fly.
3. Click the path with the White Arrow tool to show its control points
4. Release the Ctrl/Command key to convert back to the Pen tool.
5. Position the Pen tool over the point you want to remove, and the cursor will show a minus sign to indicate that it has converted to the Delete Anchor Point tool.
6. Click the point to delete it.
7. Position the Pen tool over the path. The cursor will show a plus sign to indicate that it has converted to the Add Point tool.
8. Click to add a new point. Do not drag, or you will reconfigure the control point.
9. Hold the Ctrl/Command key to convert to the White Arrow tool again.
10. Click the new point to select it, then press the mouse button down again and drag it into position.
11. Delete, place, and reposition additional points as needed. For the example shown, you won’t need a lot.

As an additional tweak, you can refine the arc between two control points: First, convert the Pen tool to the White Arrow tool by holding down the Ctrl/Command key. Now, position the pointer over the part of the path that you want to adjust. Drag the path, and notice how the control arms change length.

Smooth Points

Let’s take a closer look at those smooth points. Notice that the two control arms that affect the shape of a given segment extend inward toward each other from its end points. You can change the shape of the arc by dragging the handles at the ends of the arms with the White Arrow tool. The control arms of the adjoining segments on opposite sides of a smooth point are locked together. Moving the handles at the ends of the control arms has two effects:

• The segments behave as if the handles are magnetic and the segment bows toward its control handle. Changing the length or angle of the arm affects how sharply the segment bows.
• The adjacent segment completes the smooth point’s arc. Its control handle extends from the opposite side of the control point and see-saws around the control point as you move the handle with the White Arrow tool. So, changing the shape of the arc on one side of a smooth point generally changes the adjacent arc, too.

When you click directly on a smooth point with the White Arrow tool, the control handles extending from both sides of the point are revealed. When you click on a segment between two control points, only the handles affecting the segment appear.

The Shape Tools: Path Primitives

The Tools Panel has a button full of shape tools, but you can also get to those same tools via the Options bar when you activate the Pen tool. This is because the shape tools generate paths. When you activate one of the shape tools, it may default to the mode of either filling pixels in a layer or creating a shape layer. Just click on the Paths icon in the Options bar to generate paths instead.

Once you’re in the mode of creating paths, you can use the tools to create much more than simple circles and round-cornered rectangles.

• You can start with a primitive such as an ellipse, and use the Pen tool to modify its shape. That’s one way to create the outline for the water droplet shown at the beginning of this tip.  As with any other path, clicking with the White Arrow tool will reveal its control points and segments.
• You can draw a series of shape primitives such as ellipses and rectangles and then combine them to create a complex path that requires very little additional work with the Pen tool to complete. In the video on lynda.com,  Deke shows how you can cleanly outline a light bulb and surround it with a glow.

Cusp points

When outlining silhouettes such as a person’s face or fingers, you will find places where two arcs come together, and a smooth point will not suffice. In such cases, you can convert a smooth point to a cusp point.

Here, you’re looking at a close-up of the transition between the jaw line and the neck of the person’s shirt. The smooth point is creating a wide arc. We’ll convert this to a cusp point by decoupling the two control arms. To do so, activate the pen tool and hold the Alt key (Mac: Option key) while dragging the lower handle up and to the left. The Alt/Option key converts the Pen tool to the Convert Point tool on the fly.

Making Paths From Scratch and Converting Points

As you draw paths, you can set down corner, smooth, and cusp points as you go, or rough out the outline with just corner and smooth points, then go back and add or convert points as needed. Don’t go crazy, though. The temptation is to get the path outline exactly right, and that’s unlikely to happen. Err on the side of selecting too little.

• Tap P to activate the Pen tool and be sure it’s set to create paths.
• Be sure Auto Add/Delete is checked in the Options bar.
• To start the path, pick a direction and outline in either a clockwise or counter-clockwise direction.
• Optional: turn on the Rubber Band feature. This shows a preview of the path that is sometimes helpful. However, it is not universally popular.

  

  Click the triangle menu icon in the Options bar to reveal the Pen Options control and check the Rubber Band box. Hit Enter (Mac: Return) to put the dialog away.
• Click to place corner points.
• Shift-click to constrain angles.
• Drag to create smooth points.
• Drag to set the first arm of a cusp point, then hold the Alt key (Option on Mac) and drag to set the length and angle of the other arm. Release the mouse button before you release the Alt/Option key.
• Hold the Ctrl key (Mac: Command key) to convert to the White Arrow tool on the fly. Click to select and drag to reposition anchor points.
• Ctrl-drag (Mac: Command-drag) on the handle of a control point to change its length or angle.
• Alt-drag (Mac: Option-drag) any control handle to convert to a cusp point
• Alt-click (Mac: Option-click) on a control point to convert from a curve or cusp to a corner point.
• Alt-drag (Mac: Option-drag) outward from a control point to convert a corner or cusp point to a smooth point.
• Click on the path to add a new smooth point. (Do not drag.)
• Click an existing control point to delete it.
• To close an open path, click one end to pick up the path and then click on the other end. When the path closes, you may need to select the control point and adjust the angle and length of the control handle on the completing end.
• Optional: Double-click on the Work Path and assign a name to save the path.

Where to put curve points: In most cases, you want to place your points on the sides of your curves, not at the ends. These are things that you will get the feel of as you trace more with the Pen tool.

When adjusting shapes, you generally don’t want the control handles to cross. That creates sharp pinches or loops.

Paths to Masks

Once you have created a path that you’re satisfied with, you can convert the path to a mask. To use the path as a vector mask, do the following:

1. In the Paths panel, click on the path to activate it.
2. In the Layers panel, click on the layer you want to add the mask to.
3. Ctrl-click (Mac: Command-click) on the Add Mask icon at the bottom of the layers panel.

   

One advantage of using vector masks is that the mask remains an editable path. You can edit the mask directly, and you don’t have to re-save it in the Paths panel.

When working with vector masks, it’s sometimes helpful to activate the Masks panel and temporarily decrease the density of the mask so that you can see how the mask is lining up.

When you’re done editing the mask, return the density to 1% and click the mask thumbnail to hide the path outline and review your work.

Combining Paths With Alpha Masks

If you want to extract people from their surroundings to replace the background, you’ll often want to outline the face or body with the pen tool and use some other technique to create a hair mask. Then you can combine the two to create the final mask. The example here comes from Masking Essentials, Chapter 26 of the Photoshop CS5 One-on-One: Mastery course on lynda.com.

The high-quality Hair Mask alpha channel was created with advanced techniques, including the Calculations command. If you’re not familiar with the technique, a simpler (though less effective) approach might be to make a Color Range selection followed by the Refine Edge command.

The body outline was created with the pen tool and includes the gathers in the cloth along the torso. Those details will help to make the new composite believable.

When outlining the figure, here’s one important subtlety: Be sure to allow your path to extend slightly beyond the image area and into the pasteboard. In this example, you can see how the outline of the arm goes cleanly into the pasteboard.

To combine both elements into a complete hair and body mask, do the following:

1. Optional: duplicate the Hair Mask channel. This will allow you to alter a copy and keep the original version.
2. Click in the Channels panel to activate the target Hair Mask channel.
3. In the Paths panel, click on the body outline to activate it.
4. Tap D to set the default foreground and background colors.
5. Select Fill Path from the fly-out menu at the upper right corner of the Paths panel.
6. Choose Foreground Color from the menu at the top of the Fill Path dialog and then click OK.
7. Click the gray area at the bottom of the Paths panel to deselect the path and hide its outline.

Armed with this information and a little practice, you’ll be amazed at what you can do with the Pen tool. It is mightier than the sword, indeed.