74. Feelings

Getting work done is almost entirely about managing one’s own psychology. It’s always true throughout the development of any long-term project, but perhaps even more true when one is getting closer to the finish line; You have to maintain a state of mind where productive work is possible before any work can get done.

Recently, I’ve been feeling a bit depressed, both because of the protracted amount of time the endgame design is taking, and the slow dawning realization that the runway is getting shorter and I soon will have to, to a certain extent, abandon this project. I don’t mean to suggest that I’m giving up, but instead that no project of this magnitude is ever truly finished, it is just abandoned, and perhaps released on Steam at the same time.

It’s an interesting thing, since I should probably feel proud of how the game has come along. I have had several playtesters talk with me at length about how much they enjoyed the game, even in its various incomplete forms. As the game takes a more developed shape, these conversations have become more frequent. However, Taiji will always exist for me more as an idea than an actual thing I’ve created. An aspiration and a hope more than an artifact.

In that sense, I am disappointed in the game and I see myself as a failure. I have seen infinite potential and I have only managed to grasp at what I could reach. The limitations of both my ability and my time creep up on me as the project approaches something like a finish line. I still want so much more for it, but it isn’t practical to keep working on it forever.

With that said, there’s still runway left.

73. Website Updates/Ending Puzzle Games

Since you’re reading this on the development blog, you’re probably already aware; but I freshened up the home page to be cleaner and more informative for newcomers. There’s also some nice links at the bottom to all of the places you can find Taiji around the internet.

Speaking of which, you can now join the Discord group and chat with other folks who are excited about the game. We’ve got room to grow, so I hope you’ll consider joining our little community!

If you have any suggestions or complaints about the website or the discord group, then you can leave a comment on this post or talk to me on Discord and I’ll see what I can do!

Working on The Endgame

Taiji’s ending involves some puzzle concepts that I don’t want to spoil, so this article will be spoiler-free as far as Taiji is concerned. However, I will need to discuss some other game’s endings for context. Spoiler sections will be marked if you want to skip them, you may be able to pick up enough with context.

Because I want to keep the ending a surprise, I haven’t been able to work on it on my livestreams. But bouncing between working on art for the livestreams and working on the endgame prevented me from being able to do the type of Deep Work necessary to get the ending right.

However, as I’m heading into what should be the final year of development, it is urgent that I have this part of the game design worked out, so I’ve been taking a bit of a sabbatical from livestreams so that I can focus on getting the ball rolling here. This past week has involved dusting off some code that I haven’t touched in almost six months.

What Makes a Good Ending?

Making a satisfying ending for any video game is a challenge, but I think I puzzle games are a particular enigma. With action games it can often suffice to cap things off with a particularly difficult boss battle. But what is the equivalent of a boss battle for a puzzle video game?

The most straightforward interpretation might be to simply have a really hard puzzle to cap things off, but this is a fundamental misunderstanding of the role that boss battles serve. Boss battles are not simply “the normal gameplay, but harder.” Instead they offer a different type of gameplay than the rest of the game. The boss enemy will usually have complex movement patterns that require memorization, and may change over the course of the fight, or perhaps the environment will become involved in the battle in a way that it previously hadn’t.

The important takeaway is that satisfying endings require a change-of-pace. There ideally needs to be a shift in the gameplay style, context, and intensity. How you choose to accomplish this is really up to you, but the point is that although a hard puzzle may suffice for the ending to a smaller subsection of your puzzle game, it cannot be relied upon on its own to create a satisfying final coda.

So, let’s take a look at some of the methods that other games have chosen in order to achieve this change of pace. One of the most common methods that I’ve seen is to introduce a timer of some kind.

(mechanical spoilers for the end of Portal and Portal 2 follow)

Both Portal games, for instance, end with boss battles where the player must solve a series of portalling challenges in a limited amount of time. Apart from a bit of a forced mechanical trick at the end of Portal 2, both games don’t introduce any new mechanics at the end and instead have the player exercising basic puzzle skills that they have mastered earlier in the game, with most of the challenge coming from the time constraint.

(end Portal spoilers)

The problem with puzzle games having time-constrained challenges is that they can devolve into a sort of worst-case scenario if you’re not very careful about designing them. If the puzzles are too much of a challenge or the time limit is too aggressive, players may find themselves repeating one of the least replay-able forms of gameplay besides horror.

(I mean, I’m a huge fan of puzzle games, but it’s an obvious fact that you often get much more out of replaying action games due to the dynamics of the gameplay than you get out of replaying a fixed set of puzzles, even in a fantastically well-designed puzzle game.)

(mechanical spoilers for The Witness follow)

One possible way of resolving this problem of repeatability is to introduce dynamics into the puzzle gameplay. The Witness does this with procedurally generated puzzles both in a small subsection of the Mountain ending, and in large form in its hidden Challenge area. Although the player must repeat the Challenge many times in order to succeed, because the puzzles are different each time, the player is never doing the exact same set of gameplay events through their many attempts.

(As a side note, if we consider The Challenge to be one of the endings, The Witness actually has three different endings: The meta-puzzle gauntlet inside the mountain, the time-limited challenge, and the sky lounge ending. Perhaps The Witness is “covering all the bases”, having multiple endings which might satisfy different players in different ways.)

(end The Witness spoilers)

There are lots of different approaches that have been attempted with puzzle game endings, and I won’t rule out incorporating some aspects of what other games have done into the ending for Taiji, but it is my plan right now to attempt something that I have not seen done before.

With that said, it is a large design and tech challenge and it is still in the nascent stages as of the writing of this article. But if I can pull it off, I think I will have created something that I personally can be satisfied with.

Oh, there’s also a another ending. 😉

72. Gotta Save Fast

Been a little busy with various things, including dealing with my girlfriend’s dog having gotten run over. He’s okay but with a broken hip. Somehow, getting a dev log post out in time for the end of November totally slipped my mind. In any case, if you haven’t been keeping up with the video dev logs. Now would be a good time. The latest one is a bit beefy:

With that said, I’d like to dive a bit more deeply into what was involved in getting the save game hitch down to an acceptable level. As of writing this post, There’s still a hitch doing the actual saving, so the job is really only halfway done. However, I did do some work to get screenshot saving down to an acceptable level, and I think that’s a problem that many other games have, so it may be more useful to cover it in detail.

If you just want a decent solution to the problem, then feel free to skip to the end of the article to the section labeled “The Solution” and take a look at the code posted there. For the rest of you interested in hearing my journey to get there, read on!

The Problem

So why are we wanting to save screenshots as part of our save game routine? Well, the answer here is pretty simple, which is that we want the load game menu to show a screenshot of where the player was when they last saved. That way they can easily tell whether they are picking the right file to load:

Unity’s API really only gives you a few functions to handle taking screenshots. The first one and the easiest to use is ScreenCapture.CaptureScreenshot. It’s a simple to use function that takes a screenshot and saves it to a png file. This is great, right?

The problem is that it causes a massive lag spike. The peak of this single frame spike is close to 66 milliseconds. Needless to say this is unacceptable for a game that is targeting under 16.66ms per frame (and don’t even get me started about 8.33ms 120FPS which is apparently so impossible that Unity didn’t see fit to label it in this graph)

A Million and One Ways to Save a Frame

Okay okay, so lets look at some of our other alternatives. We also have CaptureScreenshotAsTexture and CaptureScreenshotIntoRenderTexture. The second one will come back much later, but for now lets take a look at the first one:

That’s an improvement, however we are still exceeding 33ms, which is slow enough to cause a noticable hitch even if the game were locked at 30fps. Additionally, this doesn’t even include the hit from writing out the file, as we only have a texture in memory.

Alright, so we should look into writing out this screenshot into a file. The easiest way to do this is the following:

System.IO.File.WriteAllBytes("test1.png", ImageConversion.EncodeToJPG(ScreenCapture.CaptureScreenshotAsTexture()));

Here’s the performance graph for that:

So, obviously this is even worse than ScreenCapture.CaptureScreenshot but we should probably have expected that because the fine folks at Unity HQ know what they’re doing (perhaps), and because we’re just taking a function designed for putting a screenshot into a Texture2D and then we’re writing that out to disk.

However this isn’t actually the approach that I used, because I was actually unaware of both CaptureScreenshotAsTexture and CaptureScreenshotIntoRenderTexture when I initially went about writing this code. I don’t know if they are new API additions or if I am just a bit slow. Either way they both got ignored for no good reason. They are perfectly fine approaches for certain cases, but with that said I probably wouldn’t have stumbled upon my ultimate solution if I had known about them. Instead, my approach was using Texture.ReadPixels.

ReadPixels requires implementing in a coroutine delayed to WaitForEndOfFrame(). This seems alright as far as performance goes, not quite under the 60FPS threshold here, but an improvement. However, this graph is simply calling ReadPixels on a 3840×2160 framebuffer, without even writing that out to a file. If we take a similar approach to the way we did earlier, then we’d get the following code and the following performance characteristics:

    IEnumerator CoWriteScreenshotTemp(int width, int height)
    {
        yield return new WaitForEndOfFrame();
        if(screenshot==null) screenshot = new Texture2D(width, height, TextureFormat.RGB24, false);
        screenshot.ReadPixels(new Rect(0, 0, width-1, height-1), 0, 0, false);
        screenshot.Apply();
        screenshot_stored = true;
        System.IO.File.WriteAllBytes("test2.jpg", ImageConversion.EncodeToJPG(screenshot));
    }

Damn, so now we’re almost as bad as CaptureScreenshotAsTexture, while having made the code significantly more complicated. However, because we already need to do this ReadPixels process in a Coroutine, perhaps we can do it over multiple frames, so what might happen if we tossed in a few yield return‘s in there and spread that out over a couple frames, and while we’re at it, why don’t we put the file writing on its own frame?

Okay, performance implications are not always obvious in these high-level languages. We have almost an equivalent hitch across two frames now. Still, it seems like there has to be some fertile ground in this direction, even with all the obfuscation of a high-level language. So perhaps the issue is that we’re saving out the file on the main thread. Perhaps if we spin up a thread to write out the file?

Thread _encodeThread;

IEnumerator CoWriteScreenshotTempMultiFrameWithFileWriteThread(int width, int height)
    {
        if(screenshot==null) screenshot = new Texture2D(width, height, TextureFormat.RGB24, false);
        yield return new WaitForEndOfFrame();
        screenshot.ReadPixels(new Rect(0, 0, width/2, height-1), 0, 0, false);
        yield return new WaitForEndOfFrame();
        screenshot.ReadPixels(new Rect(width/2, 0, (width/2)-1, height-1), width/2, 0, false);
        screenshot.Apply();
        yield return 0;
        rawBytes = screenshot.GetRawTextureData();
        QueuedPath = "test4.jpg";
        _encodeThread = new Thread(EncodeAndWriteFileWithArray);
        _encodeThread.Start();
    }

    void EncodeAndWriteFileWithArray()
    {
        byte[] jpgBytes = ImageConversion.EncodeArrayToJPG(rawBytes,
                          UnityEngine.Experimental.Rendering.GraphicsFormat.R8G8B8_UInt,
                          (uint)Screen.width, (uint)Screen.height, 0, 75);
        File.WriteAllBytes(QueuedPath, jpgBytes);
        Globals.updateLoadGameMenu = true;
    }

Well, in spite of the fact that the above graph looks very similar, the scale on the left has changed, and now we are peaking a bit past 33ms rather than all the way to 66ms. So this is definitely an large improvement. However, we’re still not even halfway to 60FPS… One thing we could do pretty easily is to just add in even more yield return new WaitForEndOfFrame()s and read from the framebuffer over even more frames. However, because we are just reading from whatever the most recent frame was, if the scene changes much, this pretty easily can result in problems such as below:

Notice the visible seam down the middle of the frame. The issue is that the framebuffer is changing while we are trying to read from it. The performance of spreading this out over 5 different slices would look like the following:

Admittedly, I was a bit stuck here, but then someone suggested that I take a look into doing a copy GPU side so that I could simply read from the same framebuffer and not have any of these seams. I apologize for not remember who exactly it was who suggested it. In retrospect it seems pretty obvious, but as you might imagine I was pretty lost in the details at this point.

The easiest way to do this would be to use our earlier friend CaptureScreenshotIntoRenderTexture (I told you it’d come back!). As this will do exactly what the doctor ordered, capture the framebuffer into a RenderTexture. Unfortunately (or perhaps fortunately as the case may be), I didn’t know about this function and so I proceeded to dive into several ways of doing this type of thing, ultimately settling on CommandBuffers. (In the future, this will be the most outdated part of this article, as CommandBuffers are part of the legacy graphics pipeline for Unity and will most likely be removed.)

Okay, I’m going to save the code breakdown for a bit later, because otherwise I’d be putting a bunch of code in here that’s marginally different from the final code. But we’ll just say that we set up a CommandBuffer to copy the contents of the screen over to a second RenderTexture (which is really just a GPU-side texture). As part of that process, we have to blit it to a temporary buffer so that we can flip the image vertically. Due to some vagaries in how Unity handles its graphics pipeline, the image gets flipped after it is post-processed but before it is fed into the final display. We come out the other end with a RenderTexture, only it’s actually vertically oriented correctly, unlike CaptureScreenshotIntoRenderTexture, that just leaves it upside down.

This means that, unlike with our earlier staggered read, we can split the read from this RenderTexture across as many frames as we want without having any seams show up. The performance characteristics of doing this across 5 frames looks like the following:

Okay, so another improvement, but there still seems to be a large hitch at the end associated with getting the raw bytes from a Texture2D, and even the 5-frame-staggered readback from the framebuffer into the Texture2D is exceeding our 16ms frame boundary….enter:

The Solution

So, the ideal way to avoid this readback and conversion hitch is…maybe you can guess?

Well, if you didn’t want to guess, I’m gonna tell you anyway; tell the GPU to do the conversion and readback for us in an asynchronous way. The fact is that we will always end up with a bit of a hitch if we are using the CPU to tell the GPU to give us some texture data right now, because that means it has to stop or finish what it’s doing and focus entirely on that task. Instead, we can tell the GPU to give us that texture data “whenever it has a moment”, this is called a asynchronous readback request and can be done in Unity using the AsyncGPUReadback.Request function.

Because this generates garbage, I instead chose to use the variant that uses NativeArray: AsyncGPUReadback.RequestIntoNativeArray. Note that this does mean that we will want to make sure we dispose of the NativeArray when we are done with it. For my purposes, I really only dispose of the array and reallocate it if the resolution changes.

First the performance characteristics of this approach:

Ah, we are finally at something acceptable. You may notice the peak shows slightly above the 16ms line, so it’s clear there is still some hit from the async readback, however in practice the stutter is not noticeable. I may revisit this further before release to see if I can squeeze out a bit more performance here. But for now I am happy enough with this part of the equation. For Taiji, I still have to do some work to improve the actual writing of the save file, but the screenshot taking is “good enough to ship”, as far as I’m concerned.

The actual save file performance is the left spike, and the right is the screenshot taking

So, here’s a snippet of something resembling the final code that I used. This is not exactly a plug and play class, but should be a good basis for getting something working in your own projects. I tried to at least have everything necessary in the snippet:

string QueuedPath;
NativeArray<byte> imageBytes;
byte[] rawBytes;
Thread _encodeThread;

public Material screenshotFlipperMat;

Vector2 previousScreenSize;

CommandBuffer cBuff;
public RenderTexture screenRT, tempRT;
    

//Call this function whenever you want to take a screenshot
public void TakeScreenshot(string fileOutPath)
{
    StopAllCoroutines();
    StartCoroutine(SaveScreenshot(fileOutPath, Screen.width, Screen.height));
}

//Helper function that will be called later
void InitializeBuffers()
{
    //We do some checks to see if the resolution changed and we need to recreate our RenderTextures
    bool resChanged = (previousScreenSize.x != Screen.width) || (previousScreenSize.y != Screen.height);
    previousScreenSize = new Vector2(Screen.width, Screen.height);
    
    //The array is Screen width*height*3 because we are going to use a 24bit RGB format (TextureFormat.RGB24)
    if(imageBytes.IsCreated == false) imageBytes = new NativeArray<byte>(Screen.width*Screen.height*3, Allocator.Persistent);
    else if(resChanged)
    {
        imageBytes.Dispose();
        imageBytes = new NativeArray<byte>(Screen.width*Screen.height*3, Allocator.Persistent);
    }
    
    if(tempRT == null || resChanged) tempRT = new RenderTexture(Screen.width, Screen.height, 24);
    if(screenRT == null || resChanged) screenRT = new RenderTexture(Screen.width, Screen.height, 24);
    
    //We build our command buffer, which includes a double blit using a special 
    //material (shader in article) so that we can flip the output from the backbuffer
    
    //This double blit seems to be necessary because CommandBuffer.Blit will not allow us to blit using a material
    //if we are blitting from the backbuffer (BuiltinRenderTextureType.CurrentActive)
    if(cBuff == null || resChanged)
    {
        cBuff = new CommandBuffer();
        cBuff.name = "ScreenshotCapture";
        cBuff.Blit(BuiltinRenderTextureType.CurrentActive, tempRT);
        cBuff.Blit(tempRT, screenRT, screenshotFlipperMat);
    }
    GetComponent<Camera>().AddCommandBuffer(CameraEvent.AfterImageEffects, cBuff);
}


//Function to dispose of our imageBytes from external classes
public void Dispose()
{
    imageBytes.Dispose();
}

//It may be possible to do this in one frame instead of as a coroutine, but I have not tested
IEnumerator SaveScreenshot(string fileOutPath, int width, int height)
{
    InitializeBuffers();
    yield return 0;
    Camera.main.RemoveCommandBuffer(CameraEvent.AfterImageEffects, cBuff);
    QueuedPath = fileOutPath;
    AsyncGPUReadback.RequestIntoNativeArray(ref imageBytes, screenRT, 0, TextureFormat.RGB24, ReadbackCompleted);
}

void ReadbackCompleted(AsyncGPUReadbackRequest request)
{
    if(request.hasError) return; //We just won't write out a screenshot, not a huge deal
    _encodeThread = new Thread(EncodeAndWriteFile);
    _encodeThread.Start();
}

void EncodeAndWriteFile()
{
    rawBytes = imageBytes.ToArray();
    byte[] jpgBytes = ImageConversion.EncodeArrayToJPG(rawBytes, UnityEngine.Experimental.Rendering.GraphicsFormat.R8G8B8_UInt, (uint)Screen.width, (uint)Screen.height, 0, 75);
    File.WriteAllBytes(QueuedPath, jpgBytes);
    Globals.updateLoadGameMenu = true;
}

Additionally, you’ll need a shader to do the backbuffer flipping, so here’s something for that. Better could perhaps be done, but this is based off of an internal Unity copy shader, so maybe not much better:

Shader "Custom/ScreenshotFlipper" {
    Properties{ _MainTex("Texture", any) = "" {} }
    SubShader {
        Pass {
            ZTest Always Cull Off ZWrite Off
 
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma target 2.0
 
            #include "UnityCG.cginc"
 
            sampler2D _MainTex;
            uniform float4 _MainTex_ST;
 
            struct appdata_t {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
            };
 
            struct v2f {
                float4 vertex : SV_POSITION;
                float2 texcoord : TEXCOORD0;
            };
 
            v2f vert(appdata_t v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                float2 uv = v.texcoord.xy;
                uv.y = 1-uv.y;
                o.texcoord = TRANSFORM_TEX(uv, _MainTex);
                return o;
            }
 
            fixed4 frag(v2f i) : SV_Target
            {
                return tex2D(_MainTex, i.texcoord);
            }
            ENDCG
 
        }
    }
    Fallback Off
}

Hopefully this has been a fun deep dive for you more technically-minded folks. Lord knows it was a lot of work to write it, but I had fun too. 🙂

71. Trees in the Wind

I’ve wanted to get animation for the trees into the game for a while, but could never quite manage it. I’ve tried many different approaches to this, but always the main issue is that the game renders at a fixed low-resolution pixel grid.

Some other pixel art games choose to render the game at a higher internal resolution than that of the actual pixel art, which means that you can rotate sprites without much aliasing. You can see a comparison between a high-res rotation-based animation (left) and how the effect breaks down when you render at a low resolution (right):

I have never liked the look of pixel art games that mix different resolutions, so I chose to render Taiji in a way that would force all effects in the game to be rendered at the same resolution of the base pixel art. But as you can see above, this means that rotating pixel art tends to cause strange artifacts that sort of look like edges sliding across the image. Obviously, this is very unaesthetic looking, and we need to try something else.

One possibility that I tried was to add in some noise to attempt to jitter out the sampling and create a smoother look. This removes the “sliding edges” appearance, but ends up adding in a lot of noise along edges. The effect could perhaps work well with a game that has a more forgiving art-style with a lot of noise built into the graphics.

So, with a couple of failures under my belt, I decided to rule out large motions such as rotating the entire tree, and instead I focused my efforts on animating the leaves on their own. This type of effect can be done fairly easily in a shader by simply adding in a noise offset when you sample the texture for the leaves.

This is certainly an improvement, but the effect is a bit too strong. Also if you look at it closely, it feels more like the tree is underwater than being effected in the wind. We could tone the strength of the distortion down, but then the motion becomes so subtle that it’s almost not worth having.

Another possibility that I attempted was to custom author a secondary texture which would control how the distortion was applied. I tried using a noise texture with leaf pattern built into it. I even did some tests pre-rendering leaves with Blender so that I could use the scene normals of the leaves to modulate the distortion.

I didn’t save this iteration of the shader, but suffice to say that it did not work much better than the purely random noise I was using earlier.

However, I started to think that an approach similar to how I animated the grass would be effective. The grass is essentially just a flat texture on the inside, with all the distortion happening along the outside edges.

So what would it look like if I did the same for the trees?

We’re getting close! This effect is even more pleasing, with a better balance between the details of the original pixel art and significant enough motion to be worthwhile. However, the motion feels a bit unnatural because it is confined completely to the outside edges.

What I chose to do to resolve this was to re-incorporate the idea of having a secondary texture control where the distortion effect can be applied. When used to highlight the internal edges, this forms the final effect. The wind map texture is below on the left. You can see that some interior pixels are colored red, those are the ones that are allowed to be distorted in the final effect on the right:

Overall, I’m pretty happy with how this came out. It adds some much needed motion to the trees, giving those scenes a more dynamic feel, and it doesn’t distort the base pixel art so much that it feels unnatural.

For a fun bonus, remember when I said that the unconstrained effect looked like water? I ended up using the same effect for this reflection in the water:

70. It’s All Mine

The past month has been spent working on the art for the Mine area. At this point it’s complete apart from the indirect lighting pass. I’ve already started implementing the art into the game, as this is a time-consuming process in its own right. There are lots of details that you have to figure out at that point which are easy to overlook when just doing art in a paint program. Some of the obvious ones though, are sorting order issues and putting trigger volumes all over the place.

I’m overall pretty happy with how the art for this area has turned out. A pattern that I’ve noticed is that each new area comes out a little bit closer than the last area to what my original vision was. It’s been a big learning experience working on this game.

That brings me to another point which I’ve been ruminating a bit on lately. As I get closer to the finish line for the game (there’s only two areas left), it becomes much clearer what the final form of the game is likely to be. Although this is exciting, it also ends up being a bit depressing, because it becomes clear that the game will not quite live up to all of my own hopes for it.

Don’t get me wrong, I think the game is in many ways much better than any of my initial ideas suggested, but there’s still this sense of wistful potential. I left many of my early ideas on the cutting room floor in order to make a more focused game, and I still feel a longing for a game that doesn’t exist. Perhaps I’m too close to the project and I can only see it’s flaws. Perhaps this is just a general feeling that I will always have, that will continue to drive me to make more games in the future.

Thinking about the future, I’m not really sure what I will end up doing after Taiji is finished. At the moment, I try not to think about it too much, because I still have a long road to finish walking down, and I don’t want to lose focus by finding distraction in some new shiny idea.

Part of me wants to quit games and never do this again. I can’t lie, this has become quite a grueling process. Working on a game solo has its upsides, but its very easy to lose motivation and feel isolated and lonely.

Oh well, we’ll see what the future holds when we get there. For now, I’m trying to decide how much I will be able to afford going back to areas that I’ve already done and improving the artwork there. I’m glad that many of you have had nice things to say about the look of the game, since I don’t consider myself a particularly great artist. Still, I have made most of the progress on the game by trying to keep a mantra of “good enough for now”, and part of that was consoling myself with the possibility that I could go back and revise stuff that I don’t like “later”.

The fact is though, I don’t have an infinite amount of time, money, or energy to work on this game, so at some point: “later” has to become “forever”.

This is standing out in my mind more because part of the Mine area connects with the Shrine area, and while working on the art for the Mine, I availed myself of the opportunity to improve that connecting part of the Shrine. This was something that has been on my list of “go back to later hopefully”, and I found it made me really happy to bring that small area more in line with my original vision.

Again, we’ll see how much of that I can actually afford to do before I have to ship. At this point the focus has to be on making a mad dash towards having something “shippable”. I suppose that’s typically called a beta; where you have all the features in the game and could theoretically put the game out, but you are taking time to polish and fix bugs and such. I honestly am looking forward to that point much more than I am looking forward to actually shipping the game.

Gallery!

For the last couple months (!), I’ve been working on doing the art for one of the areas in the game. This area is called the Gallery, and is a large manor which has been repurposed to house many works of art and puzzles. This area marks the largest and most complex single interior that I’ve made for the game so far. Below you’ll see the exterior on the left, as well as a diagonal cutaway to show all the different interior floor layers.

Needless to say, this was a time consuming effort, and I’m glad to finally be finished with the area. I have also been working on tying it into the surrounding environment, which will take some more time. Below you can see the area near the entrance, which features a winding path down a cliff-side.

68. In The Mountains of Madness (Win32 Wrangling Across the API Boundary)

As I mentioned in the previous blog post, I ran into some issues with mapping the mouse movement from the screen into the game. One of those issues required venturing outside the bounds of the Unity API.

So, for the longest time, I had a certain number of playtesters who were reporting that the mouse movement in the game was very sluggish for them. Unfortunately, I was not able to reproduce this issue until I purchased another monitor. It’s possible that the issue still exists in other circumstances and for other reasons, but in this blog post I will be covering what I did to at least resolve the issue that I could reproduce.

The Issue

“…Unity does not always properly tell you the resolution information for the monitor that the game is running on, and instead shows you the resolution of the “main monitor” as it is set in the display settings in Windows.”

–Me, earlier

Although Unity provides us with a Screen structure that contains information corresponding to the resolution of the screen and the size of the game window, this information is not always correct for our purposes. I was wrong however, in the previous post, in asserting that it always pulls info from the main monitor; what it actually does depends on whether the game is running in windowed mode or fullscreen mode.

If the game is running in windowed mode, then Screen.currentResolution is the native resolution of the monitor that the game is currently running on. However, if the game is running in fullscreen mode, then currentResolution will always be the same as the game’s internal rendering resolution, regardless of the final output resolution.

For example, if we are running the game fullscreen at 1920×1080 but the display is 3840×2160, even though Unity upscales the game to 3840×2160 in order to make it fullscreen, currentResolution will be 1920×1080.

This is a big problem, because in order to scale our mouse movement from screen space into world space, we need to know the ratio between the size of the screen pixels and game pixels. In fullscreen mode, if the player is running below their native monitor resolution, a single pixel in the game’s internal resolution will correspond to multiple pixels on the screen because of upscaling.

Funnily enough, even though we can get this ratio in windowed mode, it is totally unnecessary there, as the game is be rendered unscaled in windowed mode.

The Solution

Because I couldn’t reproduce the issue for the longest time, my initial hunch here was that solving this problem would involve coming up with some other way to handle mouse input appropriately.

I hoped that the new input system for Unity would allow me to solve this issue, however my initial experiments showed some kind of cumulative performance issue which would cause the game to become unplayable after about 30 minutes or so. (I am not sure if this issue has been resolved at this point, but in any case, I decided to pursue other possibilities for fixing the mouse movement.)

Upon finally reproducing the issue, and coming to the diagnosis that I mentioned in the previous section of this article, I set about trying to get the information about the monitor in some other way.

There are other functions in the Unity API, but none of them were very helpful. For example, you can easily find information about all the monitors using the Display class, but there is no information about which monitor the game is currently running on. Display.main is simply the main system monitor according to Windows (this was the cause of my earlier confused reporting about Screen)

So I did what any confused programmer would do at this point; I googled the problem. This led me to this thread on the Unity forums, and some very confusing code.

There’s no real good way around saying it, I just copied and pasted that code and tried to work from there. This was after locating the only thing I could find written officially by Unity about this type of functionality, which was really no help at all.

(I also found a Unity Answers thread with some similar code to the thread on the forums.)

So, in hopes of explaining what I have learned in a better way, and adding one more random thing to the internet about how to handle calls to the Win32 api from Unity. I will post the entire class I wrote and we’ll go through the code as far as I can explain it.

First, the code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System;
using System.Runtime.InteropServices;

public static class MonitorInfo 
{
    [DllImport("user32.dll")]
    private static extern IntPtr GetActiveWindow();
    [DllImport("user32.dll")]
    private static extern IntPtr MonitorFromWindow(IntPtr hwnd, int flags);
    
    [StructLayout(LayoutKind.Sequential)]
    public struct RECT
    {
        public int left;
        public int top;
        public int right;
        public int bottom;
    }
    
    [StructLayout(LayoutKind.Sequential)]
    public class MONITORINFO
    {
        public int cbSize = Marshal.SizeOf(typeof(MONITORINFO));
        public RECT rcMonitor = new RECT();
        public RECT rcWork = new RECT();
        public int dwFlags = 0;
    }
    
    [DllImport("user32.dll", CharSet = CharSet.Auto)] [return: MarshalAs( UnmanagedType.Bool )]
    private static extern bool GetMonitorInfo(IntPtr hmonitor, [In, Out] MONITORINFO info);
    
    
    public class monitor
    {
        public int width, height;
    }
    
    public static monitor current;
    static MONITORINFO info;
    public static bool isValid;
    
    public static void Update()
    {
        if(info == null) info = new MONITORINFO();
        isValid = GetMonitorInfo(MonitorFromWindow(GetActiveWindow(), 0), info);
        if(isValid)
        {
            if(current == null) current = new monitor();
            current.width = info.rcMonitor.right - info.rcMonitor.left;
            current.height = info.rcMonitor.bottom - info.rcMonitor.top;
        }
    }
    
    
}

Looking at the boilerplate at the top of the file, it’s pretty standard fare, however we need the following in order to interface between Unity’s API and the Win32 API layer:

using System.Runtime.InteropServices;

Next, in our class we need to declare prototypes for the external functions that we plan on calling. We also tell Unity with the [DllImport] attribute to import these functions at runtime from the user32.dll file.

[DllImport("user32.dll")]
private static extern IntPtr GetActiveWindow();
[DllImport("user32.dll")]
private static extern IntPtr MonitorFromWindow(IntPtr hwnd, int flags);

These function definitions are based off the interfaces specified for these functions on MSDN:

GetActiveWindow()

HWND GetActiveWindow();

MonitorFromWindow()

HMONITOR MonitorFromWindow(
  HWND  hwnd,
  DWORD dwFlags
);

A big part of the annoyance here is these types that Microsoft uses for their API calls. What the heck is an HWND? Well, unfortunately I do have some experience with the terrible windows API, so I know that HWND is a Handle to a WiNDow. Similarly for HMONITOR, which is a handle to a monitor. And by handle, they just mean a 32-bit pointer.

God knows how you’re supposed to find this out, but the type that we’re supposed to use in C# to deal with 32-bit pointers is IntPtr.

Okay, so that just leaves DWORD to figure out.

Well, a DWORD, or Double WORD is just a 32-bit integer. This is essentially based off the idea that the processor word size is 16 bits (which it is not, but probably was at the time the Windows API was designed).

Anyway, moving on, we can just use int in our C# code, as the C# integer size is 32 bits. That gives us the definitions below:

private static extern IntPtr GetActiveWindow();
private static extern IntPtr MonitorFromWindow(IntPtr hwnd, int flags);

private and static may not be necessary for you, but in my case this is part of a static class so that it can be easily accessed at global scope from all my other classes in Unity. Static classes in C# require all members to be declared static (and unfortunately it doesn’t do this automatically, we actually have to type static a billion times). I also chose to make these members private because this makes this all a bit less worry-prone as a globally scoped class.

So, next we have a couple of structure definitions:

[StructLayout(LayoutKind.Sequential)]
public struct RECT
{
    public int left;
    public int top;
    public int right;
    public int bottom;
}
[StructLayout(LayoutKind.Sequential)]
public class MONITORINFO
{
    public int cbSize = Marshal.SizeOf(typeof(MONITORINFO));
    public RECT rcMonitor = new RECT();
    public RECT rcWork = new RECT();
    public int dwFlags = 0;
}

These are again, based on the MSDN documentation for these structures.

Rect

typedef struct tagRECT {
  LONG left;
  LONG top;
  LONG right;
  LONG bottom;
} RECT, *PRECT, *NPRECT, *LPRECT;

MonitorInfo

typedef struct tagMONITORINFO {
  DWORD cbSize;
  RECT  rcMonitor;
  RECT  rcWork;
  DWORD dwFlags;
} MONITORINFO, *LPMONITORINFO;

We could use long in the C# code for the RECT members, and it would technically be more correct, however integers work just fine here. This may be some legacy feature of the API though.

We also need to use the [StructLayout] attributes, because normally, in C# the compiler is free to re-order the elements of a struct or class for memory efficiency purposes, however in our case we need the elements of these structures to be in the exact order that they are in the Win32 API.

One strange detail is this line:

public int cbSize = Marshal.SizeOf(typeof(MONITORINFO));

This is essentially code that I copied from the forum post I mentioned earlier, and as such I can’t entirely explain why it works this way, but the Marshal class is part of the System.Runtime.InteropServices that we included earlier. Specifically, it is a class that is used in order to convert between managed and unmanaged memory types.

Based on that, I can hazard a guess that what we’re doing here is actually pulling the size of the MONITORINFO struct from the Win32 API side of the the code, and not the size of the MONITORINFO class that we are currently in the process of defining in our own code. This seems reinforced by the fact that if we change this line to….

public int cbSize = sizeof(typeof(MONITORINFO));

…then Unity will complain that MONITORINFO is undefined.

Okay, moving on. Now we have this whopper of a function prototype definition.

[DllImport("user32.dll", CharSet = CharSet.Auto)] [return: MarshalAs( UnmanagedType.Bool )]
private static extern bool GetMonitorInfo(IntPtr hmonitor, [In, Out] MONITORINFO info);

This is, of course, still based on the definition on the MSDN page:

GetMonitorInfoA()

BOOL GetMonitorInfoA(
  HMONITOR      hMonitor,
  LPMONITORINFO lpmi
);

“Wait, hold on, why is this called GetMonitorInfoA?”

Answering that will also answer why we need to have this CharSet = CharSet.Auto attribute as part of the dll import.

There are two versions of GetMonitorInfo in the Win32 api, one for ASCII characters (GetMonitorInfoA, which is the legacy version) and one for UTF-32 characters (GetMonitorInfoW).

“But wait, why on earth are we worried about text?”

We only have to care about this because this function could potentially return a MONITORINFOEX structure, which contains a string that is a name of the monitor. In our case, we are just throwing that data away and using the smaller MONITORINFO struct, but we still have to support it as part of our function prototype definition.

*sigh*

Another oddity as part of the Attribute definition is this:

[return: MarshalAs( UnmanagedType.Bool )]

Why do we have to marshal bools? Don’t ask me, but the function returns a bool specifying whether or not a monitor could successfully be found, and if you actually want to know that, you’ll need to marshal the bool across the API boundary, because managed and unmanaged bools are not compatible.

The only detail that might be confusing about the actual definition of the function is the [In, Out] attribute. This seems to be the way to specify that a parameter is passed by reference across the API boundary here. Changing it to ref does not work.

At this point, the rest of the code should be fairly understandable, if you have any experience with Unity C# coding:

public class monitor
{
    public int width, height;
}
    
public static monitor current;
static MONITORINFO info;
public static bool isValid;
    
public static void Update()
{
    if(info == null) info = new MONITORINFO();
    isValid = GetMonitorInfo(MonitorFromWindow(GetActiveWindow(), 0), info);
    if(isValid)
    {
        if(current == null) current = new monitor();
        current.width = info.rcMonitor.right - info.rcMonitor.left;
        current.height = info.rcMonitor.bottom - info.rcMonitor.top;
    }
}

One thing that’s worth noting, is that I keep track of a isValid bool publicly, so that I can always check if the calls to the Win32 api returned valid data before I go around using it.

Implementation

So with all that done, we can now change the code that handles the mouse scaling to the following:

Vector2 ScreenRes = new Vector2(Screen.width, Screen.height);
if(MonitorInfo.isValid) ScreenRes = new Vector2(MonitorInfo.current.width, MonitorInfo.current.height);

This means, that as long as we are able to return some valid information from the Win32 API, we will use that. If not, we will fall back to the Screen structure that Unity provides for us, which may be wrong in some cases.

Hope you learned something!

67. Mouse Wrangling

Okay, so I’m a bit late on this devlog entry, but in one of my recent video dev logs (Which I do weekly over on my YouTube channel; if you haven’t been checking them out) I promised that I would write about how mouse movement is handled in-depth, and so I’m going to pay off on my promise.

Mouse movement in games is almost always a bit more of a tricky thing to get right than it seems when playing a finished product. For one, the mouse is a direct input method, so when you move the mouse you expect an equivalent motion in-game, either of the camera or a cursor. This means that latency is much more noticeable here than it is on something indirect like a thumbstick.

Latency can be handled in any number of ways and is mostly outside the scope of this article, but I mention it because it’s a very common case where the technical details of how you handle input can have a large effect on how the game feels.

Mouse motion in Taiji is handled in a way that I haven’t seen any other games use. Most top-down or 2d games with mouse controls (i.e. Diablo, Civilization, Starcraft), just have you move the mouse around on the screen like it were your desktop, and the camera moves very rigidly and only under direct player control. However, in Taiji, camera movement is much more dynamic and loosely follows the position of your player avatar. This causes a bit of an issue, in that the camera may still be moving when the player is trying to use the mouse cursor to target a tile on a puzzle panel. Trying to hit a moving target with the mouse is a bit of a challenge that I am not interested in introducing into the game.

There are a few ways that this problem could have been resolved. One possible solution is to just never move the camera when a puzzle is visible and interact-able. However, this causes some discontinuities in the camera movement which can be, at best, irritating or at worst, nauseating. It also doesn’t work for some panels that are always interact-able if they are on screen. Another possibility is to just give up and lock the camera to the player as many other games do (Diablo, Nuclear Throne). This approach didn’t appeal to me for aesthetic reasons. I want the game to have a relaxed feel, and the slower camera movement is part of that.

The approach I chose instead was to treat the mouse cursor as though it is character existing in the world of the game, and allow the player to control that character directly with the mouse. Another way of thinking about this is that the mouse behaves as though the entire world of the game was your computer desktop, and we are just seeing a small view into that “desktop” at any one time. The view into this “desktop” can move around, but the mouse cursor would stay in the same place relative to everything else on the “desktop”. Technically, this is to say that the cursor is in world-space rather than screen space. This can all seem a bit abstract though, so to help make this concept of “world-space” vs “screen-space” mouse cursors a bit more clear, I recommend watching the video below, which I excerpted from one of my recent video devlogs.

Great! So this fixes our issue of having the mouse cursor drift around as the camera moves, and of the player trying to hit a moving target sometimes when they are clicking on puzzle panel tiles. However, we now have introduced another problem, which is that, since the fairy cursor character only moves if the player moves the mouse; when the player walks across the map, they might forget and leave the cursor behind. Luckily, for this game, in particular, the player is seldom needing to move both the player avatar and the cursor at the same time, so if the player walks more than a certain distance without touching the mouse, we can just put the little fairy character into a separate mode where they follow the player’s avatar around automatically. This might seems like it would get annoying, but in practice, it never really gets in the way of what you’re doing.

So how does this work?

So far, I’ve mostly been recapping the same ground that I covered in my recent devlog video about the mouse movement, but in this written devlog I’ve got much more space to dive into some of the technical details about how this functions.

Fundamentally, the work that we need to do here is to translate a certain amount of motion of a physical mouse into an equivalent motion in the game world. Taiji is being developed using Unity, so in this case, I use Unity’s input system (which Unity is currently in the process of replacing). In Unity’s input system, there are a few ways that I can access information about the actual mouse device.

One possibility is to simply look at the screen position of the mouse, and then project that into the game world in some way. However, we don’t use this method, as we want to make sure the OS mouse cursor is confined to the window while the game is running (you can free up the mouse to use other applications by pausing the game, of course). So we lock the OS mouse cursor to the center of the game window and just ask Unity to tell us how much it tried to move each frame.

mouseDelta = new Vector3(Input.GetAxisRaw("Mouse X"),Input.GetAxisRaw("Mouse Y"));

In the above code, mouseDelta represents the delta (amount of change) in the position of the mouse from the previous time we polled (the last frame). We get the horizontal (X) and vertical (Y), components of this delta, using the GetAxisRaw functions to avoid any time smoothing that Unity might otherwise apply.

Now we can get an amount that the mouse has moved, but there’s one problem; if the OS mouse cursor moved 10 units horizontally, we don’t know how far that really is. We don’t have any idea what the units are. Unfortunately, Unity’s documentation is no real help here, but through experimentation, I have determined that these values are in “pixels at 96dpi“. This might seem the same as pixels on the screen, however, because of screen scaling in Windows, these values may not correspond 1:1 with pixels on the screen.

In any case, correcting for this is fairly easy, as we can simply ask unity for the DPI of the screen. Then we normalize this value to 96dpi and multiply the mouse movement by this value:

float DPI_Scale=(Screen.dpi/96.0f);
mouseDelta *= DPI_Scale;

This now means our mouseDelta is in actual screen pixels.

So…at this point we can just take the movement and project it into the world of the game…right?

Well, unfortunately, no, and this is part of the reason that I had to go down an annoying rabbit-hole that involved calling into the Win32 API. For now let’s just continue onward and ignore that problem, as the explanation of how the mouse gets transformed into world space is already quite complicated on its own. But just make a mental note and we’ll come back here in a bit.

So, we have another issue that we have to resolve, which is that the game can be running in its own scaled resolution. This happens at two levels. The first is that the player can run the game at a sub-native resolution but in fullscreen mode. The game runs in a borderless fullscreen window, which means that if, for example, the game is running fullscreen at 1920×1080 on a 4k monitor, one pixel in the game’s output resolution will correspond to 4 screen pixels.

There are unfortunately some issues here, in that Unity does not always properly tell you the resolution information for the monitor that the game is running on, and instead shows you the resolution of the “main monitor” as it is set in the display settings in Windows. This will be the cause of the Win32 API madness later (interestingly enough, the DPI value is always correct, even with the screen resolution information is not). In any case, we will pretend for now that the Unity API call returns the correct value in all cases, and so the code to resolve this possible mismatch in resolution is as follows:

Vector2 ScreenRes = new Vector2(Screen.width, Screen.height);
float renderScale = (ScreenRes.x / ScreenRes.y < GameRes.x / GameRes.y) ? ScreenRes.x/GameRes.x : ScreenRes.y/GameRes.y;
if(!Screen.fullScreen) renderScale = 1.0f;
mouseDelta *= renderScale;

You might notice that this is a bit more complicated than the code accounting for screen dpi scaling. This is because the game runs at a forced 16:9 aspect ratio in order to tightly control what the player can and cannot see at any given time. This means that if the player is running the game fullscreen on a monitor of a different aspect ratio, it will be either letterboxed or pillarboxed, depending on whether the monitor’s native aspect is wider or narrower than the games. The final rendering scale will, therefore, depend on which side of the game’s view fully spans the monitor (horizontal in the case of letterboxing, and vertical in the case of pillar boxing)

Also of course, if the game is not fullscreen, we don’t have to worry about this render scaling at all.

Omigosh, we’re almost done with this and then we can go home. So the next and final thing that we have to do to get the mouse movement into world space is to account for the degree to which the in-game camera is zoomed in. Luckily, Unity provides us with a fairly straightforward function to map a position from the game’s rendered viewport into a world position. We use that to figure out a scaling value between screen and world space:

float mouseScale = (mainCamera.ScreenToWorldPoint(Vector3.one).x - mainCamera.ScreenToWorldPoint(Vector3.zero).x)/2.0f;
mouseDelta *= mouseScale;

ScreenToWorldPoint takes an input coordinate in pixels and returns a coordinate in “Unity Units”, so we are taking the horizontal distance from one screen pixel to a pixel immediately up and to the right of it, then dividing that horizontal distance by 2 to find the zoom factor. The reason for the division by 2 is actually a bit of a mystery to me at the time of writing this, which is why writing these deep tech dives can be a bit more useful for development than they might otherwise seem. I initially thought that perhaps this was because I was somehow returning the diagonal distance here. However, changing the code to use a pixel directly to the right does not produce a different result. So I guess it remains a mystery to me. However, without the division, the scaling will be wrong. Perhaps someone will comment on this devlog and tell me where I’ve obviously messed up the math somewhere earlier that would require this division or some other reason why it needs to happen here.

At this point, other than multiplying the mouse movement by an additional sensitivity value, we are done, and we can now apply mouseDelta as a translation to the actual in-game object representing the mouse cursor.

To Be Continued

I know, I know, I said I would get into the API nonsense, but this piece has gone on long enough on its own, so I’m gonna leave this on a cliffhanger and get back to fill in the details on that later this week. Till now, thanks for reading, and remember to wishlist the game on steam if you haven’t!

66. Burnout and Walking Animations

I plan on posting these video development logs on a weekly basis over at my YouTube channel. I may post some occasional reminders here going forward, but I’d rather keep this written devlog as it’s own separate thing rather than simply as a cross-posting of the videos. So, if you don’t want to miss any of the video dev logs, I recommend you subscribe to my YouTube channel.

However, since you’re here instead of at YouTube, I’ll reward you with a few sneak peeks into what I left out of the video log.

This past week, I’ve been working on relatively small polish features, which is a bit of a continuation of the work I did last week with footstep sounds (still have some more of those to do actually). I think this is partly as a way to give myself a break from tearing up major chunks of the game to add artwork. But even if it feels like less progress, these small things propagate across the entire game and affect your interactions throughout.

One of these small improvements is to the walking animations. The previous animations were serviceable, however when I added running animations, the running animations looked significantly better in comparison. So I added more frames to the walking animation and made some small tweaks. You can see the comparison between the old (left) and new (right) animations below:

I still want to add animations for when you move diagonally, and hope to get to that next. But I think even this goes some ways towards giving the game a more polished feeling.

I did a few other fun things, but I’ll save those for next week’s video. Hope to see you then. 🙂