Dynamic Rendering Masks

Mike Dailly
5th September 2017

So this is a quick one, but incredibly cool and valuable. When drawing to the screen (or a surface), most people only think about the colour, that is the red, green and blue channels, but actually there's also an alpha channel. Now it has a whole heap of issues if you're going to start using it to build sprites or generate screens or whatever, but one thing that's its REALLY good at, is being a mask.

Because the user only ever sees the final image, it means we can use that alpha channel for effects if we want to. There's a little known command called gpu_set_colorwriteenable() that lets you switch on and off channel writing, and this means we can build up a colour image, and then mess around with this invisible one. So, lets take a pixel of the screen, it has 4 channels; Red,Green,Blue and Alpha and you can enable and disable the writing to any or all of these channels. Then say we wanted to define an "odd" mask and have a sprite go inside this odd mask only. The RGB channels might be - for example, a funny shaped button, and what we can do is render the button graphic normally, then use the alpha channel on the screen to "mask out" bits outside the button, so we only draw inside the button shape.

So lets take 2 sprites, A button, and a "mask" where we want to allow drawing.


Now the idea is we'll draw everything as normal including the background, button and a button shadow, and then restrict all further drawing to INSIDE the green area of the button. We can do all this without resorting to render targets, or CPU processing, shaders - or in fact anything especially complex. Next we need a background and a sprite we'll use to draw inside the button.


Now we're all set! If we create a test object and a draw event we can then set about actually rendering everything. First off, we'll draw the the background, the button shadow and then the button itself, all as normal.

draw_sprite(sBackground,0, 0,0);
draw_sprite_ext(sButton,0, 32+4,32+4,1,1,0,c_black,0.5);
draw_sprite(sButton,0, 32,32);

Now it gets a little funky! We disable blending and all the colour channels as we want to fill just the alpha channel with 0 and if blending was on, normal blend maths would apply, and nothing would get written in - or at the very least, not the values we're after. By switching off blending, 0 is copied into the alpha channel, while disabling the other channels protects the actual image.

draw_rectangle(0,0, room_width,room_height, false);

So above you can see we've disabled blending and the colour channels, set the alpha to 0, and then drawn a full screen rectangle - although we know this will ONLY fill the alpha channel. Next we'll write in the "mask", the area of the screen we want to allow writing to. This could literally be anything, but in this case, its a mask specifying the green area of the button.

draw_sprite(sMask,0, 32,32);

Alpha is set back to 1 and the mask is drawn - again with no blending so that it stores the whole thing directly into the alpha channel. We now have a colour screen with the button, and an alpha channel that's 0, except for where the green area of the button, which is now 1. Now we switch on destination alpha mode. What this does is instead of reading alpha from the sprite as it would normally do, it'll read it from the screen. Since we have no alpha in our sprite anymore, we'll to switch on alpha testing. This will still read the alpha from the sprite, and then drop out pixels that are below a certain value before it gets to the actual screen blending.

draw_sprite(sPlayer,0, mouse_x,mouse_y);

So, here we set the blend, then we enable alpha testing, then we draw the sprite using the mouse as the centre, and then finally restore everything back to normal. If we now run this, we'll get the image below...


There are a huge number of things this can be used for, from effects like the one above, to using it to highlight in game elements that have been rendered normally during the game. I've used it in the past to simulate the ZX Spectrum colour clash, for doing shadows that only cover the areas I want it to cover, to full on reflections in roads in a 3D F1 game. It's also important to remember that it doesn't have to be a simple 0 or 255 value that's stored there. It's an normal alpha value, so you and vary it to be solid in some areas, to faint or not there in others. It can be a great way to do water in a game, allowing you to apply a single water sprite over the whole scene yet have if change as it draws.

You should also remember that these are just sprites we're drawing there, so you could be moving things around procedural in the alpha channel. How about applying some blobby dynamics into the alpha channel then using it for fog for example? This would give a really nice sensation of clumps of fog as it moves around the world for example. The list really is endless! This is also a very cheap effect, and will work on just about every platform - barring HTML5 Canvas mode (although there are other ways to do it in canvas mode).

Hope you enjoyed this one, you can download this project HERE

Written by Mike Dailly
Mike Dailly was the previous head of development at GameMaker, helping us to transform the game engine during his tenure. He was also the lead game programmer and creator of Lemmings and Grand Theft Auto.
Back to blogs