Anti-aliasing in pixel art

Anti-aliasing places a few in-between colours where a line steps, so a curve looks smoother than the grid allows. Used sparingly it polishes a sprite; used everywhere it turns it soft.

A ring spinning, rendered with Pixor

Sprite with no anti-aliasing on its lines — Off

Sprite with anti-aliasing on the silhouette — Silhouette

Sprite with anti-aliasing on all line corners — All line corners

What it is

Pixels are square, so any line that is not perfectly straight or at a clean angle becomes a staircase. On a gentle curve, such as the rim of a shield or the top of a head, those steps can look jagged at 1× zoom.

Manual anti-aliasing (often written AA) softens a step by placing one or two pixels of an in-between colour in the corner where the line jumps. The eye blends them, and the curve reads as smoother than the pixels really are. The colour is usually picked from the existing ramp, somewhere between the line and what is next to it.

It is a hand-made effect: a few chosen pixels, not a filter over the whole sprite.

When it helps

Long, gentle curves. A circle or a slope that steps two or three pixels at a time benefits most.
Large sprites. At 64 px and up there is room for in-between pixels without losing detail.
Inner lines. Lines between parts of a sprite sit on known colours, so the in-between shade is easy to choose and always correct.

When to leave it out

Small sprites. At 16 or 24 px every pixel carries shape. An in-between pixel there often reads as a smudge or a stray detail.
The outer edge, on unknown backgrounds. An anti-aliased silhouette is mixed for one background colour. Put the sprite on another and the soft pixels show as a faint halo.
Very limited palettes. With four or eight colours there may be no shade that sits between two others. Then the steps stay hard, and that is part of the look.
Bold, chunky styles. Some games want every step visible. That is a choice, not a mistake.

Common mistakes

Too much. Anti-aliasing every step makes the whole sprite look blurred, as if scaled up with filtering on.
Wrong length. The in-between run should match the step it softens. A single pixel at the end of a long step does little; a long run next to a short step bulges.
Banding. Lining up in-between pixels into a second, parallel line makes the edge look doubled.
Engine smoothing. Hand-made anti-aliasing is undone by bilinear filtering. Set the texture filter to nearest.

How Pixor does it

Pixor draws lines one pixel wide without L-corners, and has an anti-aliasing setting with three levels. 0 leaves every line hard. 1 softens the silhouette only. 2 softens the corners of every line, not only the outer edge. Try 0 for small sprites and busy backgrounds, and 1 or 2 for larger sprites where curves have room. The setting is part of the look in the Style graph, and the settings manual lists it with the other line options.

Outlines: the lines anti-aliasing softens
Selout: coloured outlines that already soften an edge
Sprite sizes: how much room a sprite has for in-between pixels

Questions

Should I anti-alias the outside edge of a sprite?

Only if you know the background. An in-between colour on the outer edge is mixed for one background; on a darker or lighter one it shows as a halo. Many artists keep the outer edge hard for that reason.

Is anti-aliasing the same as the smoothing a game engine does?

No. Manual anti-aliasing is a handful of pixels the artist chooses. Engine smoothing filters the whole image when it scales, which blurs pixel art. Keep texture filtering on nearest.