Recently I have been having some issues with sprites in the game I'm currently working on. It's a 2D game and I'm using OpenGL to render textured triangle pairs for my sprites. Let me explain some of the caveats I've run into with regard to mipmaps.

Mipmaps

First, for the uninitiated (or if, like me, you regularly need a reminder about what all the various 3D-graphics terms mean), I'll cover what mip-maps are and why they're used.

A computer graphic consists of a 2-dimensional array of pixels, and so does the screen that you view it on. If we want to display a graphic at its original size, the pixels of the graphic are simply copied to the pixels of the screen and no information from the graphic is lost.

Rendering without scaling

Rendering without scaling

However, if we want to display a graphic at a smaller size to the original, we have to scale down the graphic. A naive approach to this would be, for each pixel in the target area of the screen, to look up the pixel at the same location, proportionally, in the graphic. As we look up each pixel to draw on the screen, we skip out pixels in-between in the graphic.

Scaled down by skipping pixels

Scaled down by skipping pixels

A better approach is to use bilinear interpolation to calculate the colour values between pixels in the source graphic. For example, if our scale lookup lands 3/4 of of the way between a blue pixel and a white pixel, we draw a pixel on the screen which mixes 3/4 white and 1/4 blue, giving us a light blue colour. The result is a smoother-looking scaled image.

Scaled down using bilinear interpolation

Scaled down using bilinear interpolation

Smooth scaling is important when rendering textured surfaces in 3D. A surface which disappears into the screen, such as a wall or floor, shows the effect of the chosen scaling approach very obviously in the distance. Using the naive pixel-skipping approach, the texture quickly becomes a mess of undecipherable pixels and we begin to see moire patterns at the furthest points.

Scaling on a 3D surface

Scaling on a 3D surface

By contrast, interpolated scaling gives a much better result. The overall shape of the texture can still be made out in the distance, even if the details can not, and the dizzying moire patterns are gone.

Scaling on a 3D surface with interpolation

Scaling on a 3D surface with interpolation

The problem with scaling like this is that it is a much more costly operation to perform in real-time. This is where mipmaps come in. Mipmaps are collections of pre-scaled textures. Rather than performing the slow, interpolated scaling every time a texture is rendered, we create several copies of the texture once, smooth-scaled to various sizes, and use these to look up the pixels to render each time. The texture size that we read from is based upon the scale that we're rendering.

Scaled down using mipmaps

Scaled down using mipmaps

Stretching and Un-Stretching

So, back to my 2D sprites. I had generated mipmaps for all my sprite textures, because why not? However, I soon ran into my first caveat.

I had a rectangular sprite which, when converting to a texture, I had stretched to be square (a requirement of textures). I had assumed I could just render a rectangular triangle pair with the square texture on it, and this would result in the sprite having the correct aspect ratio. It did have the correct aspect ratio, but it looked terrible. The visual quality of the image was awful; it was far too blurry to make out any of the fine detail.

Low-quality image when squashing the texture back to its original height

Low-quality image when squashing the texture back to its original height

The reason was down to the mipmaps. Because the texture was being squashed, even just on one axis, OpenGL's mipmapping was kicking in. It was helpfully trying to smooth out the result by switching to a pre-smoothed version of the image. The result was a loss of visual fidelity on both axes.

Stretching the texture was a bad idea. Instead, expanding the empty space around the sprite to make a square texture was a much more sensible alternative and avoided the need for unnecessary scaling. An even better idea would be for me to use a texture atlas anyway, but I've yet to implement that.

Colour Bleed

The second mipmap-related caveat I hit was an ugly darkened border that I noticed around my sprites. This was particularly noticeable where the edge of one sprite overlayed another of the same colour, where there should have been no visible transition between them. At first I thought this might have been related to pre-multiplied alpha, but this turned out to not be the case. The reason actually turned out to be to do with the way the sprite images were saved from my paint application.

Noticeable darkened border when same colour overlaps

Noticeable darkened border when same colour overlaps

When OpenGL creates mipmaps, it does so by taking the average colour values over multiple adjacent pixels. This was fine for the opaque parts of the sprite, where colours simply mixed with other colours. But at the edge of the sprite, the alpha channel got involved. What happens, you might wonder, when semi-transparent coloured pixels are averaged with fully-transparent pixels? The result would be more transparent but not quite fully-transparent, but what about the colour?

The answer is that the average uses whatever colour value those fully-transparent pixels happen to have in their red, green and blue channels. You can't usually see this colour value, because it's completely see-through, but there is a colour stored there. The graphics application I'm using, GIMP, happens to use black (zero red, green and blue) for those pixels when it exports an image, and so averaging the sprite's edge pixels with black was resulting in visibly darker colours around the edge.

Black in transparent pixels darkening colours in mipmap

Black in transparent pixels darkening colours in mipmap

A solution to this issue is to explicitly define the colour values of the transparent pixels so that they don't default to black. In GIMP, I had some degree of success using the following technique:

  1. With layers collapsed, right click on the image's remaining layer and select "Add Layer Mask"
  2. From the list of options, choose "Layer's Alpha Channel"
  3. Right click on the layer mask in the layer view and choose "Disable Layer Mask" so that we can see the image without alpha applied
  4. Left click on the layer in the layer view to reselect it
  5. Select the magic wand tool and select the empty region around the sprite
  6. From the menu, choose "Select" > "Grow" to include a bit of the sprite's edge in the selection
  7. From the menu, choose "Filter" > "Distorts" > "Value Propagate".
  8. Select the "More opaque" radio button and click OK
  9. Continue to repeat the filter with Ctl+F until the edge of the sprite has bled out to the edges of the image
  10. Right click on the layer in the layer view and choose "Remove Alpha Channel"
  11. Right click on the layer mask in the layer view and re-enable the layer mask to see the image with alpha applied. The sprite should look unchanged from its original state, but now has appropriate colour values in the transparent pixels
  12. Export the image as normal
Bleeding out colours using Value Propagate

Bleeding out colours using Value Propagate

I found that the Value Propagate filter wasn't entirely reliable, so I had to do some value propagation by hand to prevent corrupting the sprite itself.

Do I Even Need Mipmaps?

At this point, I've concluded that I can probably manage without mipmaps at all, at least for the time being. The hoops I'm having to jump through by processing each of my sprite graphics, for the sake of nicer scaling, doesn't seem like such a great trade-off. I've put that feature on the low-priority, "nice-to-have" pile, for now.