Paper Goomba

Nov 11, 2023

technical rust graphics bevy camera text-gem

Alright, alright. I know this isn’t rendering text to the grid. I know this isn’t “according to plan”! But… We need to walk with sprites in a 3d world before we can run with beautiful ASCII art in a 3d world. It turns out that Bevy does not make it easy to render text or sprites using a 3d camera. So we’ll need to get that working, and we’ll add a little extra embellishment as well. At the end of this post, we’ll be rendering some paper goombas that look like this. We are going to start with rendering basic images because ASCII text can be rendered to a texture and used in the same way as a sprite sheet. It turns out that rendering text is really hard though! So we’ll take the smaller step of rendering sprites in our 3d world first.

Code Refactor

I refactored the code a bit between the last post and this one. Normally you can follow along by making edits to the code with the post, but in this case I thought it would be easier to refactor the code and just include the important snippets in the post. If you want to follow along and try things out, get the code for this post.

2D Mesh for a 3D World

Our world is 3D, its glorious grid lines zooming off into the distance. But we want to render 2D characters to represent NPCs and players in our world. To do this, we will need to represent our characters using 3D meshes. In 2D graphics programming, we would use sprites and blit them to the screen. When rendering with a GPU, using a quad (2 triangles that form a rectangle) with a texture is a great way to accomplish the same thing as blitting. So, let’s create a mesh that will let us render our sprites in a 3D world. The mesh is just two quads back to back. One quad for the front one quad for the back. We’ll want to integrate our mesh and material into the physically-based rendering (PBR) pipeline, and so we will call our module pbr_sprite. We’ll create a struct that we use to build our mesh, let’s call it QuadSprite. We can base this code on the code for bevy::shape::Quad.

pbr_sprite.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
use bevy::{
    pbr::{ExtendedMaterial, MaterialExtension},
    prelude::*,
    render::{
        mesh::Indices,
        render_resource::{AsBindGroup, PrimitiveTopology, ShaderRef},
    },
};

pub struct QuadSprite {
    size: Vec2,
    flip: bool,
}

impl QuadSprite {
    pub fn new(size: Vec2) -> Self {
        Self { size, flip: false }
    }

    pub fn flipped(mut self) -> Self {
        self.flip = true;
        self
    }
}

impl From<QuadSprite> for Mesh {
    fn from(quad: QuadSprite) -> Self {
        let extent_x = quad.size.x / 2.0;
        let extent_y = quad.size.y / 2.0;

        let (u_left, u_right) = if quad.flip { (1.0, 0.0) } else { (0.0, 1.0) };
        let vertices = [
            // Front Face
            ([-extent_x, -extent_y, 0.0], [0.0, 0.0, 1.0], [u_left, 1.0]),
            ([-extent_x, extent_y, 0.0], [0.0, 0.0, 1.0], [u_left, 0.0]),
            ([extent_x, extent_y, 0.0], [0.0, 0.0, 1.0], [u_right, 0.0]),
            ([extent_x, -extent_y, 0.0], [0.0, 0.0, 1.0], [u_right, 1.0]),
            // Back Face
            ([-extent_x, -extent_y, 0.0], [0.0, 0.0, -1.0], [u_left, 1.0]),
            ([-extent_x, extent_y, 0.0], [0.0, 0.0, -1.0], [u_left, 0.0]),
            ([extent_x, extent_y, 0.0], [0.0, 0.0, -1.0], [u_right, 0.0]),
            ([extent_x, -extent_y, 0.0], [0.0, 0.0, -1.0], [u_right, 1.0]),
        ];

        let indices = Indices::U32(vec![0, 2, 1, 0, 3, 2, 6, 4, 5, 7, 4, 6]);

        let positions: Vec<_> = vertices.iter().map(|(p, _, _)| *p).collect();
        let normals: Vec<_> = vertices.iter().map(|(_, n, _)| *n).collect();
        let uvs: Vec<_> = vertices.iter().map(|(_, _, uv)| *uv).collect();

        Mesh::new(PrimitiveTopology::TriangleList)
            .with_indices(Some(indices))
            .with_inserted_attribute(Mesh::ATTRIBUTE_POSITION, positions)
            .with_inserted_attribute(Mesh::ATTRIBUTE_NORMAL, normals)
            .with_inserted_attribute(Mesh::ATTRIBUTE_UV_0, uvs)
    }
}

Rendering a Sprite in 3D

We’ll use this little goomba sprite that I hacked together in Aseprite.

main.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
fn init_scene(
    mut commands: Commands,
    asset_server: Res<AssetServer>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut standard_materials: ResMut<Assets<StandardMaterial>>,
    mut grid_materials: ResMut<Assets<ExtendedMaterial<StandardMaterial, GridMaterial>>>,
    mut pbr_sprite_materials: ResMut<
        Assets<ExtendedMaterial<StandardMaterial, pbr_sprite::PbrSpriteMaterial>>,
    >,
) {
    ...

    // PBR Sprite
    let image: Handle<Image> =
        asset_server.load_with_settings("goomba.png", |settings: &mut ImageLoaderSettings| {
            settings.sampler = ImageSampler::nearest();
        });
    commands.spawn(MaterialMeshBundle {
        mesh: meshes.add(pbr_sprite::QuadSprite::new(Vec2::new(32.0, 32.0)).into()),
        material: standard_materials.add(StandardMaterial {
            base_color: Color::WHITE,
            base_color_texture: Some(image),
            alpha_mode: AlphaMode::Mask(0.2),
            ..Default::default()
        }),
        transform: Transform::from_xyz(0.0, 30.0, 0.0),
        ..Default::default()
    });
}

Running our application with cargo run will produce a result that looks like this.

If we rotate to the back of the goomba, it looks like this.

Consistent Lighting

This is certainly an aesthetic. But it’s not one that I think works very well with the paper look. The goomba should be in a consistent light regardless of if we are looking at the front or back face. In order to accomplish this, we need to modify the normals of our mesh. Let’s point all of the normals in the up direction, on both faces. This will produce nice, consistent lighting. Se let’s create a new Mesh from our QuadSprite, we’ll call this new shape a PaperSprite.

pbr_sprite.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
...

pub struct PaperSprite(pub QuadSprite);

impl From<PaperSprite> for Mesh {
    fn from(paper: PaperSprite) -> Self {
        let quad = paper.0;

        let extent_x = quad.size.x / 2.0;
        let extent_y = quad.size.y / 2.0;

        let (u_left, u_right) = if quad.flip { (1.0, 0.0) } else { (0.0, 1.0) };
        let vertices = [
            // Front Face
            ([-extent_x, -extent_y, 0.0], [0.0, 1.0, 0.0], [u_left, 1.0]),
            ([-extent_x, extent_y, 0.0], [0.0, 1.0, 0.0], [u_left, 0.0]),
            ([extent_x, extent_y, 0.0], [0.0, 1.0, 0.0], [u_right, 0.0]),
            ([extent_x, -extent_y, 0.0], [0.0, 1.0, 0.0], [u_right, 1.0]),
            // Back Face
            ([-extent_x, -extent_y, 0.0], [0.0, 1.0, 0.0], [u_left, 1.0]),
            ([-extent_x, extent_y, 0.0], [0.0, 1.0, 0.0], [u_left, 0.0]),
            ([extent_x, extent_y, 0.0], [0.0, 1.0, 0.0], [u_right, 0.0]),
            ([extent_x, -extent_y, 0.0], [0.0, 1.0, 0.0], [u_right, 1.0]),
        ];

        let indices = Indices::U32(vec![0, 2, 1, 0, 3, 2, 6, 4, 5, 7, 4, 6]);

        let positions: Vec<_> = vertices.iter().map(|(p, _, _)| *p).collect();
        let normals: Vec<_> = vertices.iter().map(|(_, n, _)| *n).collect();
        let uvs: Vec<_> = vertices.iter().map(|(_, _, uv)| *uv).collect();

        Mesh::new(PrimitiveTopology::TriangleList)
            .with_indices(Some(indices))
            .with_inserted_attribute(Mesh::ATTRIBUTE_POSITION, positions)
            .with_inserted_attribute(Mesh::ATTRIBUTE_NORMAL, normals)
            .with_inserted_attribute(Mesh::ATTRIBUTE_UV_0, uvs)
    }
}

main.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
fn init_scene(
    mut commands: Commands,
    asset_server: Res<AssetServer>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut standard_materials: ResMut<Assets<StandardMaterial>>,
    mut grid_materials: ResMut<Assets<ExtendedMaterial<StandardMaterial, GridMaterial>>>,
    mut pbr_sprite_materials: ResMut<
        Assets<ExtendedMaterial<StandardMaterial, pbr_sprite::PbrSpriteMaterial>>,
    >,
) {
    ...

    // PBR Sprite
    let image: Handle<Image> =
        asset_server.load_with_settings("goomba.png", |settings: &mut ImageLoaderSettings| {
            settings.sampler = ImageSampler::nearest();
        });
    commands.spawn(MaterialMeshBundle {
        mesh: meshes.add(pbr_sprite::QuadSprite::new(Vec2::new(32.0, 32.0)).into()),
        material: standard_materials.add(StandardMaterial {
            base_color: Color::WHITE,
            base_color_texture: Some(image),
            alpha_mode: AlphaMode::Mask(0.2),
            ..Default::default()
        }),
        transform: Transform::from_xyz(0.0, 30.0, 0.0),
        ..Default::default()
    });
}

Rerun the scene with cargo run and see the rewards that we have reaped.

Paper Material

This is cool, we’ve got a 2D goomba that is rendering in 3D space, and it’s well-lit! Let’s make it even more 2D, more papery, more better by adding an outline to the sprite. This will give it kind of a paper cutout look to it. We’ll start by adding a new PBR material extension called PbrPaperMaterial.

See the shader code this is based on at: https://github.com/cukiakimani/shaders-yo/blob/master/Assets/Sprite%20Outline/Shaders/SpriteOutline.shader.

pbr_sprite.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
pub struct PbrSpritePlugin;

impl Plugin for PbrSpritePlugin {
    fn build(&self, app: &mut App) {
        let mut material_plugin =
            MaterialPlugin::<ExtendedMaterial<StandardMaterial, PbrPaperMaterial>>::default();
        material_plugin.prepass_enabled = true;

        app.add_plugins(material_plugin);
    }
}

#[derive(Debug, Default, Clone, Asset, TypePath, AsBindGroup)]
pub struct PbrPaperMaterial {
    #[uniform(200)]
    pub uv_scale: Vec2,
    #[uniform(201)]
    pub uv_translate: Vec2,
    #[uniform(202)]
    pub outline_thickness: f32,
    #[uniform(203)]
    pub outline_color: Color,
}

impl MaterialExtension for PbrPaperMaterial {
    fn fragment_shader() -> ShaderRef {
        "pbr_paper.wgsl".into()
    }
}

pbr_paper.wgsl

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
// Adapted from https://github.com/cukiakimani/shaders-yo/blob/master/Assets/Sprite%20Outline/Shaders/SpriteOutline.shader

#import bevy_pbr::{
    pbr_fragment::pbr_input_from_standard_material,
    pbr_functions::alpha_discard,
    mesh_view_bindings::view,
    pbr_types::STANDARD_MATERIAL_FLAGS_UNLIT_BIT,
    pbr_bindings,
}


#ifdef PREPASS_PIPELINE
#import bevy_pbr::{
    prepass_io::{VertexOutput, FragmentOutput},
    pbr_deferred_functions::deferred_output,
}
#else
#import bevy_pbr::{
    forward_io::{VertexOutput, FragmentOutput},
    pbr_functions::{apply_pbr_lighting, main_pass_post_lighting_processing},
}
#endif

@group(1) @binding(200) var<uniform> uv_scale: vec2<f32>;
@group(1) @binding(201) var<uniform> uv_translate: vec2<f32>;
@group(1) @binding(202) var<uniform> outline_thickness: f32;
@group(1) @binding(203) var<uniform> outline_color: vec4<f32>;

fn outline_alpha(uv: vec2<f32>) -> f32 {
    let outline_thickness_x = outline_thickness * uv_scale.x;
    let outline_thickness_y = outline_thickness * uv_scale.y;

    var alpha = 0.0;
    // fixed upAlpha = SampleSpriteTexture ( IN.texcoord + fixed2(0, _OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv + vec2<f32>(0.0, outline_thickness_y), view.mip_bias).a;

    // fixed downAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(0, _OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(0.0, outline_thickness_y), view.mip_bias).a;

    // fixed rightAlpha = SampleSpriteTexture ( IN.texcoord + fixed2(_OutlineThickness, 0)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv + vec2<f32>(outline_thickness_x, 0.0), view.mip_bias).a;

    // fixed leftAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(_OutlineThickness, 0)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(outline_thickness_x, 0.0), view.mip_bias).a;
    
    // fixed upRightAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(_OutlineThickness, _OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(outline_thickness_x, outline_thickness_y), view.mip_bias).a;

    // fixed upLeftAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(_OutlineThickness, -_OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(outline_thickness_x, -outline_thickness_y), view.mip_bias).a;

    // fixed downRightAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(-_OutlineThickness, _OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(-outline_thickness_x, outline_thickness_y), view.mip_bias).a;

    // fixed downLeftAlpha = SampleSpriteTexture ( IN.texcoord - fixed2(-_OutlineThickness, -_OutlineThickness)).a;
    alpha += textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv - vec2<f32>(-outline_thickness_x, -outline_thickness_y), view.mip_bias).a;

    return saturate(alpha);
}

fn outline(uv: vec2<f32>) -> f32 {
    let c = textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);

    if (c.a == 0.0) {
        return outline_alpha(uv);
    } else {
        return 0.0;
    }
}

@fragment
fn fragment(
    in: VertexOutput,
    @builtin(front_facing) is_front: bool,
) -> FragmentOutput {
    // generate a PbrInput struct from the StandardMaterial bindings
    var pbr_input = pbr_input_from_standard_material(in, is_front);
    
    // translate and scale UVs
    let uv = (in.uv * uv_scale) + uv_translate;
    pbr_input.material.base_color = textureSampleBias(pbr_bindings::base_color_texture, pbr_bindings::base_color_sampler, uv, view.mip_bias);

    // apply outline
    let outline_alpha = outline(uv);
    if (outline_alpha > 0.0) {
        pbr_input.material.base_color = outline_color;
    }

    // alpha discard
    pbr_input.material.base_color = alpha_discard(pbr_input.material, pbr_input.material.base_color);

#ifdef PREPASS_PIPELINE
    // write the gbuffer, lighting pass id, and optionally normal and motion_vector textures
    let out = deferred_output(in, pbr_input);
#else
    // in forward mode, we calculate the lit color immediately, and then apply some post-lighting effects here.
    // in deferred mode the lit color and these effects will be calculated in the deferred lighting shader
    var out: FragmentOutput;
    if (pbr_input.material.flags & STANDARD_MATERIAL_FLAGS_UNLIT_BIT) == 0u {
        out.color = apply_pbr_lighting(pbr_input);
    } else {
        out.color = pbr_input.material.base_color;
    }

    // apply in-shader post processing (fog, alpha-premultiply, and also tonemapping, debanding if the camera is non-hdr)
    // note this does not include fullscreen postprocessing effects like bloom.
    out.color = main_pass_post_lighting_processing(pbr_input, out.color);
#endif

    return out;
}

We’ll use this new material in our scene to render the goomba with an outline.

main.rs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
fn init_scene(
    mut commands: Commands,
    asset_server: Res<AssetServer>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut standard_materials: ResMut<Assets<StandardMaterial>>,
    mut grid_materials: ResMut<Assets<ExtendedMaterial<StandardMaterial, GridMaterial>>>,
    mut pbr_sprite_materials: ResMut<
        Assets<ExtendedMaterial<StandardMaterial, pbr_sprite::PbrSpriteMaterial>>,
    >,
) {
    ...

    // PBR Sprite
    let image: Handle<Image> =
        asset_server.load_with_settings("goomba.png", |settings: &mut ImageLoaderSettings| {
            settings.sampler = ImageSampler::nearest();
        });
    commands.spawn(MaterialMeshBundle {
        mesh: meshes.add(
            pbr_sprite::PaperSprite(pbr_sprite::QuadSprite::new(Vec2::new(32.0, 32.0))).into(),
        ),
        material: pbr_sprite_materials.add(ExtendedMaterial {
            base: StandardMaterial {
                base_color: Color::WHITE,
                base_color_texture: Some(image),
                alpha_mode: AlphaMode::Mask(0.2),
                ..Default::default()
            },
            extension: pbr_sprite::PbrSpriteMaterial {
                uv_scale: Vec2::new(1.0, 1.0),
                uv_translate: Vec2::new(0.0, 0.0),
                outline_thickness: 0.05,
                outline_color: Color::WHITE,
            },
        }),
        transform: Transform::from_xyz(0.0, 30.0, 0.0),
        ..Default::default()
    });
}

We should now see our paperized goomba when we run our application with cargo run.

Paper Goomba

Next Up…

Alright, we are rendering a paperized goomba into our scene and it looks pretty good. We’ll want to start rendering our ASCII characters next. In order to do that we’ll have to make textures from our character glyphs and use them to render our paper material.

Fine the full code for this post here: TextGem Post 6.

« Camera Frame Smoothing