77. One in a Mill-ion

Gosh, time sure does fly when you’re making a game. I seem to have let June slip by without an update here. Although I have kept up with the video devlog series over on the YouTubes, I recognize that some folks just like to read a little more than they like to look at my beautiful face and unkempt mane.

I’ve been fixing up small things around the game that have been annoying me for a long time. There’s still bunches left undone on the “nice to have” list, but I’ve tidied up some important ones. Mostly I’ve been doing this because the progression on the artwork for the Mill area has been going a bit slower than anticipated (even though every time I do an area I anticipate it will take more time than the last!)

Part of the delay has been from doing some design revisions to the overall flow of the area. I mentioned these plans in the previous post, but I wasn’t expecting that the structural changes to the area would require so many puzzle design changes as well. It turns out that when you are planning on having an area be mostly knowledge-gated, you have to design puzzles that are specific to this purpose. Good knowledge-gates put up an initial barrier of complexity, but then fall rather quickly if the player has the pre-requisite knowledge.

Who’d have thunk that a good knowledge gate is one that gates mostly on knowledge!

In addition to this, I felt like there were a few small ideas in the area that could use a bit more polish in how they are delivered to the player. One of my favorite things in puzzle games is when I’m put in what seems like an impossible situation, based upon everything I’ve come to assume about the mechanics of the game. But the design or layout of the puzzle allows me to deduce that some move that was previously unknown to me must be possible. So, I have been doing some redesigns to create that type of experience. Requiring the player to guess that there is some emergent behavior of the mechanics that is non-obvious rather than giving them a simple situation which just illustrates it.

At this point, I’m fairly happy with the puzzle set for this area, and I think that the flow of ideas will work well. So I’ve spent the past week or so sketching out some rough ideas for the visuals. This has required me to do a lot of research on different types of machinery so that I can hopefully create something a bit plausible as a functional space.

Concepting is still a work in progress and things are subject to large changes, but here are some things that I’ve drawn along the way so far.

76. Endings and New Beginnings

Things are finally progressing! I finished up the endgame area a week ago and shipped that out to a small group of testers. I believe there will still be some changes/improvements there, but overall I’m happy with the initial feedback I’ve seen.

Experience has shown me that when I take this long between builds of the game, major things will end up broken. Because of that, I’ve been taking a little downtime between shipping this build and implementing the next major area for the game. I didn’t want testers finding game-breaking bugs while I’m ripping up so much that I can’t ship a build until I finish the art for the next area.

Thus far, The bugs have definitely started to flow in. In fact, I’ve not been able to address the endgame area feedback as much as I’d like because there were so many other bugs. But it’s good to have my decision to postpone major changes validated.

In the meantime, as part of my break from game-breaking work, I’ve completed one of those “nice-to-have” improvements on the list; something I thought I might not be able to get done before ship: I’ve been re-doing some of the art for the starting area. You can see a couple comparisons below:

It’s hard to believe it’s already been two years since I originally did some of this artwork, but it’s been in desperate need of improvement for a while. I think the revision has much stronger details while also having a cleaner look that fits better with the rest of the game’s aesthetics.

It’s also a bit of a testament to how much I’ve improved as an artist over the course of this project. When I first started doing the art, I really didn’t know if I’d be up to the task. I’ve done some dabbling in visual art through my life, but I’ve never considered it something that I was particularly strong at. But through a combination of thoughtful art direction choices and just churning away for years on end, I think I’ve achieved something that looks nice and works well for the game.

So, What’s Next?

As I mentioned in the previous post, the next major task on the list is the artwork for the Mill area. I also need to re-think the structure of the area significantly. The current implementation is too straightforward and boring:

As you can see, most of the area is just a straight line. Although there is a bit of branching in the middle, the puzzles on the left side there are completely optional, so the main thrust of the area is totally linear. This structure, combined with the sheer number of puzzles in this area (44 essential, 25 optional), can be pretty exhausting for players to go through, and although adding artwork to the area can help significantly with the fatiguing aspect of the experience, I also want to try some other things.

I’ve experimented a bit with the structural designs for each area as I go along. I always try to do something a little different. Some areas are very linear, with a straight line of puzzles you have to all do in order. Some are mostly linear but with a few splits that you can do out of order. Some are almost entirely non-linear, with many options that are equally challenging.

One of the things I’ve thought about doing with the Mill area is designing it a bit “backwards”, with the player finding the hardest puzzles in the area before the introductory panels. This could backfire horribly, since some players may assume they should be able to solve anything that’s accessible to them, but it fits into my plan to go for a much more non-linear and knowledge-gated approach to this area than I have achieved with any of the others.

By “knowledge-gating”, I’m referring to the concept that the only barrier to progress is that you don’t understand how to do something, rather than requiring the player to obtain some item, or solve a long set of puzzles elsewhere.

Perhaps the area will be split into several buildings which can each be entered by solving a difficult puzzle. Each of these difficult puzzles will require some knowledge gained elsewhere in the area. This means that the player will have to explore around a bit when they first enter the area before they find a puzzle they can gain some traction on, but hopefully it will create that fun experience where the player realizes, after learning some new concept: “Aha! I know how to do that other puzzle now!”

I always try to push myself to do better with every new thing I add to the game. It’s what keeps this project engaging and enjoyable for me over such a long development cycle. (coming up on 6 years now!) Unfortunately this does mean that each new area has tended to take longer than the previous one, but I’m excited to get started on the Mill, and I think it has potential to be one of the better areas in the game.

75. Putting the Ends Together

I am still working on the Endgame area. I had hoped to be done with it by the end of last year, if you can believe that. So I would say things are progressing quite a bit slower than anticipated. With that said, the puzzle design for this area is now complete, as is most of the art, so I’ve just been assembling the pieces that I’ve created. Wiring up the bits and the bobs, so to speak. I’ve been doing a lot of this on stream, compromising a bit on my desire to keep everything about the Endgame unspoiled, but I think it’s more important for me to be able to focus and get the Endgame area done as quickly as I can.

Here’s a spoiler-free screenshot of some of that:

Once I finish the Endgame, there’s really only one large task left on the list, which is the Mill. So, in spite of being behind schedule, there’s still a decent chance that I can ship the game roughly on time. As I regretted in the previous devlog post, I do wish there were more time for me to polish up stuff and re-do bits I’m not as happy with as I could be. But alas…

One major thing that I’ve had to cut is simultaneous console ports of the game. The current plan is for the initial release of the game to be PC only, and then I will do one unannounced console port as soon as is reasonable. It would help me a lot with determining where best to focus if you would answer the following poll.

74. Feelings

Getting work done is almost entirely about managing one’s own psychology. It’s always true throughout the development of any long-term project, but perhaps even more true when one is getting closer to the finish line; You have to maintain a state of mind where productive work is possible before any work can get done.

Recently, I’ve been feeling a bit depressed, both because of the protracted amount of time the endgame design is taking, and the slow dawning realization that the runway is getting shorter and I soon will have to, to a certain extent, abandon this project. I don’t mean to suggest that I’m giving up, but instead that no project of this magnitude is ever truly finished, it is just abandoned, and perhaps released on Steam at the same time.

It’s an interesting thing, since I should probably feel proud of how the game has come along. I have had several playtesters talk with me at length about how much they enjoyed the game, even in its various incomplete forms. As the game takes a more developed shape, these conversations have become more frequent. However, Taiji will always exist for me more as an idea than an actual thing I’ve created. An aspiration and a hope more than an artifact.

In that sense, I am disappointed in the game and I see myself as a failure. I have seen infinite potential and I have only managed to grasp at what I could reach. The limitations of both my ability and my time creep up on me as the project approaches something like a finish line. I still want so much more for it, but it isn’t practical to keep working on it forever.

With that said, there’s still runway left.

73. Website Updates/Ending Puzzle Games

Since you’re reading this on the development blog, you’re probably already aware; but I freshened up the home page to be cleaner and more informative for newcomers. There’s also some nice links at the bottom to all of the places you can find Taiji around the internet.

Speaking of which, you can now join the Discord group and chat with other folks who are excited about the game. We’ve got room to grow, so I hope you’ll consider joining our little community!

If you have any suggestions or complaints about the website or the discord group, then you can leave a comment on this post or talk to me on Discord and I’ll see what I can do!

Working on The Endgame

Taiji’s ending involves some puzzle concepts that I don’t want to spoil, so this article will be spoiler-free as far as Taiji is concerned. However, I will need to discuss some other game’s endings for context. Spoiler sections will be marked if you want to skip them, you may be able to pick up enough with context.

Because I want to keep the ending a surprise, I haven’t been able to work on it on my livestreams. But bouncing between working on art for the livestreams and working on the endgame prevented me from being able to do the type of Deep Work necessary to get the ending right.

However, as I’m heading into what should be the final year of development, it is urgent that I have this part of the game design worked out, so I’ve been taking a bit of a sabbatical from livestreams so that I can focus on getting the ball rolling here. This past week has involved dusting off some code that I haven’t touched in almost six months.

What Makes a Good Ending?

Making a satisfying ending for any video game is a challenge, but I think I puzzle games are a particular enigma. With action games it can often suffice to cap things off with a particularly difficult boss battle. But what is the equivalent of a boss battle for a puzzle video game?

The most straightforward interpretation might be to simply have a really hard puzzle to cap things off, but this is a fundamental misunderstanding of the role that boss battles serve. Boss battles are not simply “the normal gameplay, but harder.” Instead they offer a different type of gameplay than the rest of the game. The boss enemy will usually have complex movement patterns that require memorization, and may change over the course of the fight, or perhaps the environment will become involved in the battle in a way that it previously hadn’t.

The important takeaway is that satisfying endings require a change-of-pace. There ideally needs to be a shift in the gameplay style, context, and intensity. How you choose to accomplish this is really up to you, but the point is that although a hard puzzle may suffice for the ending to a smaller subsection of your puzzle game, it cannot be relied upon on its own to create a satisfying final coda.

So, let’s take a look at some of the methods that other games have chosen in order to achieve this change of pace. One of the most common methods that I’ve seen is to introduce a timer of some kind.

(mechanical spoilers for the end of Portal and Portal 2 follow)

Both Portal games, for instance, end with boss battles where the player must solve a series of portalling challenges in a limited amount of time. Apart from a bit of a forced mechanical trick at the end of Portal 2, both games don’t introduce any new mechanics at the end and instead have the player exercising basic puzzle skills that they have mastered earlier in the game, with most of the challenge coming from the time constraint.

(end Portal spoilers)

The problem with puzzle games having time-constrained challenges is that they can devolve into a sort of worst-case scenario if you’re not very careful about designing them. If the puzzles are too much of a challenge or the time limit is too aggressive, players may find themselves repeating one of the least replay-able forms of gameplay besides horror.

(I mean, I’m a huge fan of puzzle games, but it’s an obvious fact that you often get much more out of replaying action games due to the dynamics of the gameplay than you get out of replaying a fixed set of puzzles, even in a fantastically well-designed puzzle game.)

(mechanical spoilers for The Witness follow)

One possible way of resolving this problem of repeatability is to introduce dynamics into the puzzle gameplay. The Witness does this with procedurally generated puzzles both in a small subsection of the Mountain ending, and in large form in its hidden Challenge area. Although the player must repeat the Challenge many times in order to succeed, because the puzzles are different each time, the player is never doing the exact same set of gameplay events through their many attempts.

(As a side note, if we consider The Challenge to be one of the endings, The Witness actually has three different endings: The meta-puzzle gauntlet inside the mountain, the time-limited challenge, and the sky lounge ending. Perhaps The Witness is “covering all the bases”, having multiple endings which might satisfy different players in different ways.)

(end The Witness spoilers)

There are lots of different approaches that have been attempted with puzzle game endings, and I won’t rule out incorporating some aspects of what other games have done into the ending for Taiji, but it is my plan right now to attempt something that I have not seen done before.

With that said, it is a large design and tech challenge and it is still in the nascent stages as of the writing of this article. But if I can pull it off, I think I will have created something that I personally can be satisfied with.

Oh, there’s also a another ending. 😉

72. Gotta Save Fast

Been a little busy with various things, including dealing with my girlfriend’s dog having gotten run over. He’s okay but with a broken hip. Somehow, getting a dev log post out in time for the end of November totally slipped my mind. In any case, if you haven’t been keeping up with the video dev logs. Now would be a good time. The latest one is a bit beefy:

With that said, I’d like to dive a bit more deeply into what was involved in getting the save game hitch down to an acceptable level. As of writing this post, There’s still a hitch doing the actual saving, so the job is really only halfway done. However, I did do some work to get screenshot saving down to an acceptable level, and I think that’s a problem that many other games have, so it may be more useful to cover it in detail.

If you just want a decent solution to the problem, then feel free to skip to the end of the article to the section labeled “The Solution” and take a look at the code posted there. For the rest of you interested in hearing my journey to get there, read on!

The Problem

So why are we wanting to save screenshots as part of our save game routine? Well, the answer here is pretty simple, which is that we want the load game menu to show a screenshot of where the player was when they last saved. That way they can easily tell whether they are picking the right file to load:

Unity’s API really only gives you a few functions to handle taking screenshots. The first one and the easiest to use is ScreenCapture.CaptureScreenshot. It’s a simple to use function that takes a screenshot and saves it to a png file. This is great, right?

The problem is that it causes a massive lag spike. The peak of this single frame spike is close to 66 milliseconds. Needless to say this is unacceptable for a game that is targeting under 16.66ms per frame (and don’t even get me started about 8.33ms 120FPS which is apparently so impossible that Unity didn’t see fit to label it in this graph)

A Million and One Ways to Save a Frame

Okay okay, so lets look at some of our other alternatives. We also have CaptureScreenshotAsTexture and CaptureScreenshotIntoRenderTexture. The second one will come back much later, but for now lets take a look at the first one:

That’s an improvement, however we are still exceeding 33ms, which is slow enough to cause a noticable hitch even if the game were locked at 30fps. Additionally, this doesn’t even include the hit from writing out the file, as we only have a texture in memory.

Alright, so we should look into writing out this screenshot into a file. The easiest way to do this is the following:

System.IO.File.WriteAllBytes("test1.png", ImageConversion.EncodeToJPG(ScreenCapture.CaptureScreenshotAsTexture()));

Here’s the performance graph for that:

So, obviously this is even worse than ScreenCapture.CaptureScreenshot but we should probably have expected that because the fine folks at Unity HQ know what they’re doing (perhaps), and because we’re just taking a function designed for putting a screenshot into a Texture2D and then we’re writing that out to disk.

However this isn’t actually the approach that I used, because I was actually unaware of both CaptureScreenshotAsTexture and CaptureScreenshotIntoRenderTexture when I initially went about writing this code. I don’t know if they are new API additions or if I am just a bit slow. Either way they both got ignored for no good reason. They are perfectly fine approaches for certain cases, but with that said I probably wouldn’t have stumbled upon my ultimate solution if I had known about them. Instead, my approach was using Texture.ReadPixels.

ReadPixels requires implementing in a coroutine delayed to WaitForEndOfFrame(). This seems alright as far as performance goes, not quite under the 60FPS threshold here, but an improvement. However, this graph is simply calling ReadPixels on a 3840×2160 framebuffer, without even writing that out to a file. If we take a similar approach to the way we did earlier, then we’d get the following code and the following performance characteristics:

    IEnumerator CoWriteScreenshotTemp(int width, int height)
    {
        yield return new WaitForEndOfFrame();
        if(screenshot==null) screenshot = new Texture2D(width, height, TextureFormat.RGB24, false);
        screenshot.ReadPixels(new Rect(0, 0, width-1, height-1), 0, 0, false);
        screenshot.Apply();
        screenshot_stored = true;
        System.IO.File.WriteAllBytes("test2.jpg", ImageConversion.EncodeToJPG(screenshot));
    }

Damn, so now we’re almost as bad as CaptureScreenshotAsTexture, while having made the code significantly more complicated. However, because we already need to do this ReadPixels process in a Coroutine, perhaps we can do it over multiple frames, so what might happen if we tossed in a few yield return‘s in there and spread that out over a couple frames, and while we’re at it, why don’t we put the file writing on its own frame?

Okay, performance implications are not always obvious in these high-level languages. We have almost an equivalent hitch across two frames now. Still, it seems like there has to be some fertile ground in this direction, even with all the obfuscation of a high-level language. So perhaps the issue is that we’re saving out the file on the main thread. Perhaps if we spin up a thread to write out the file?

Thread _encodeThread;

IEnumerator CoWriteScreenshotTempMultiFrameWithFileWriteThread(int width, int height)
    {
        if(screenshot==null) screenshot = new Texture2D(width, height, TextureFormat.RGB24, false);
        yield return new WaitForEndOfFrame();
        screenshot.ReadPixels(new Rect(0, 0, width/2, height-1), 0, 0, false);
        yield return new WaitForEndOfFrame();
        screenshot.ReadPixels(new Rect(width/2, 0, (width/2)-1, height-1), width/2, 0, false);
        screenshot.Apply();
        yield return 0;
        rawBytes = screenshot.GetRawTextureData();
        QueuedPath = "test4.jpg";
        _encodeThread = new Thread(EncodeAndWriteFileWithArray);
        _encodeThread.Start();
    }

    void EncodeAndWriteFileWithArray()
    {
        byte[] jpgBytes = ImageConversion.EncodeArrayToJPG(rawBytes,
                          UnityEngine.Experimental.Rendering.GraphicsFormat.R8G8B8_UInt,
                          (uint)Screen.width, (uint)Screen.height, 0, 75);
        File.WriteAllBytes(QueuedPath, jpgBytes);
        Globals.updateLoadGameMenu = true;
    }

Well, in spite of the fact that the above graph looks very similar, the scale on the left has changed, and now we are peaking a bit past 33ms rather than all the way to 66ms. So this is definitely an large improvement. However, we’re still not even halfway to 60FPS… One thing we could do pretty easily is to just add in even more yield return new WaitForEndOfFrame()s and read from the framebuffer over even more frames. However, because we are just reading from whatever the most recent frame was, if the scene changes much, this pretty easily can result in problems such as below:

Notice the visible seam down the middle of the frame. The issue is that the framebuffer is changing while we are trying to read from it. The performance of spreading this out over 5 different slices would look like the following:

Admittedly, I was a bit stuck here, but then someone suggested that I take a look into doing a copy GPU side so that I could simply read from the same framebuffer and not have any of these seams. I apologize for not remember who exactly it was who suggested it. In retrospect it seems pretty obvious, but as you might imagine I was pretty lost in the details at this point.

The easiest way to do this would be to use our earlier friend CaptureScreenshotIntoRenderTexture (I told you it’d come back!). As this will do exactly what the doctor ordered, capture the framebuffer into a RenderTexture. Unfortunately (or perhaps fortunately as the case may be), I didn’t know about this function and so I proceeded to dive into several ways of doing this type of thing, ultimately settling on CommandBuffers. (In the future, this will be the most outdated part of this article, as CommandBuffers are part of the legacy graphics pipeline for Unity and will most likely be removed.)

Okay, I’m going to save the code breakdown for a bit later, because otherwise I’d be putting a bunch of code in here that’s marginally different from the final code. But we’ll just say that we set up a CommandBuffer to copy the contents of the screen over to a second RenderTexture (which is really just a GPU-side texture). As part of that process, we have to blit it to a temporary buffer so that we can flip the image vertically. Due to some vagaries in how Unity handles its graphics pipeline, the image gets flipped after it is post-processed but before it is fed into the final display. We come out the other end with a RenderTexture, only it’s actually vertically oriented correctly, unlike CaptureScreenshotIntoRenderTexture, that just leaves it upside down.

This means that, unlike with our earlier staggered read, we can split the read from this RenderTexture across as many frames as we want without having any seams show up. The performance characteristics of doing this across 5 frames looks like the following:

Okay, so another improvement, but there still seems to be a large hitch at the end associated with getting the raw bytes from a Texture2D, and even the 5-frame-staggered readback from the framebuffer into the Texture2D is exceeding our 16ms frame boundary….enter:

The Solution

So, the ideal way to avoid this readback and conversion hitch is…maybe you can guess?

Well, if you didn’t want to guess, I’m gonna tell you anyway; tell the GPU to do the conversion and readback for us in an asynchronous way. The fact is that we will always end up with a bit of a hitch if we are using the CPU to tell the GPU to give us some texture data right now, because that means it has to stop or finish what it’s doing and focus entirely on that task. Instead, we can tell the GPU to give us that texture data “whenever it has a moment”, this is called a asynchronous readback request and can be done in Unity using the AsyncGPUReadback.Request function.

Because this generates garbage, I instead chose to use the variant that uses NativeArray: AsyncGPUReadback.RequestIntoNativeArray. Note that this does mean that we will want to make sure we dispose of the NativeArray when we are done with it. For my purposes, I really only dispose of the array and reallocate it if the resolution changes.

First the performance characteristics of this approach:

Ah, we are finally at something acceptable. You may notice the peak shows slightly above the 16ms line, so it’s clear there is still some hit from the async readback, however in practice the stutter is not noticeable. I may revisit this further before release to see if I can squeeze out a bit more performance here. But for now I am happy enough with this part of the equation. For Taiji, I still have to do some work to improve the actual writing of the save file, but the screenshot taking is “good enough to ship”, as far as I’m concerned.

The actual save file performance is the left spike, and the right is the screenshot taking

So, here’s a snippet of something resembling the final code that I used. This is not exactly a plug and play class, but should be a good basis for getting something working in your own projects. I tried to at least have everything necessary in the snippet:

string QueuedPath;
NativeArray<byte> imageBytes;
byte[] rawBytes;
Thread _encodeThread;

public Material screenshotFlipperMat;

Vector2 previousScreenSize;

CommandBuffer cBuff;
public RenderTexture screenRT, tempRT;
    

//Call this function whenever you want to take a screenshot
public void TakeScreenshot(string fileOutPath)
{
    StopAllCoroutines();
    StartCoroutine(SaveScreenshot(fileOutPath, Screen.width, Screen.height));
}

//Helper function that will be called later
void InitializeBuffers()
{
    //We do some checks to see if the resolution changed and we need to recreate our RenderTextures
    bool resChanged = (previousScreenSize.x != Screen.width) || (previousScreenSize.y != Screen.height);
    previousScreenSize = new Vector2(Screen.width, Screen.height);
    
    //The array is Screen width*height*3 because we are going to use a 24bit RGB format (TextureFormat.RGB24)
    if(imageBytes.IsCreated == false) imageBytes = new NativeArray<byte>(Screen.width*Screen.height*3, Allocator.Persistent);
    else if(resChanged)
    {
        imageBytes.Dispose();
        imageBytes = new NativeArray<byte>(Screen.width*Screen.height*3, Allocator.Persistent);
    }
    
    if(tempRT == null || resChanged) tempRT = new RenderTexture(Screen.width, Screen.height, 24);
    if(screenRT == null || resChanged) screenRT = new RenderTexture(Screen.width, Screen.height, 24);
    
    //We build our command buffer, which includes a double blit using a special 
    //material (shader in article) so that we can flip the output from the backbuffer
    
    //This double blit seems to be necessary because CommandBuffer.Blit will not allow us to blit using a material
    //if we are blitting from the backbuffer (BuiltinRenderTextureType.CurrentActive)
    if(cBuff == null || resChanged)
    {
        cBuff = new CommandBuffer();
        cBuff.name = "ScreenshotCapture";
        cBuff.Blit(BuiltinRenderTextureType.CurrentActive, tempRT);
        cBuff.Blit(tempRT, screenRT, screenshotFlipperMat);
    }
    GetComponent<Camera>().AddCommandBuffer(CameraEvent.AfterImageEffects, cBuff);
}


//Function to dispose of our imageBytes from external classes
public void Dispose()
{
    imageBytes.Dispose();
}

//It may be possible to do this in one frame instead of as a coroutine, but I have not tested
IEnumerator SaveScreenshot(string fileOutPath, int width, int height)
{
    InitializeBuffers();
    yield return 0;
    Camera.main.RemoveCommandBuffer(CameraEvent.AfterImageEffects, cBuff);
    QueuedPath = fileOutPath;
    AsyncGPUReadback.RequestIntoNativeArray(ref imageBytes, screenRT, 0, TextureFormat.RGB24, ReadbackCompleted);
}

void ReadbackCompleted(AsyncGPUReadbackRequest request)
{
    if(request.hasError) return; //We just won't write out a screenshot, not a huge deal
    _encodeThread = new Thread(EncodeAndWriteFile);
    _encodeThread.Start();
}

void EncodeAndWriteFile()
{
    rawBytes = imageBytes.ToArray();
    byte[] jpgBytes = ImageConversion.EncodeArrayToJPG(rawBytes, UnityEngine.Experimental.Rendering.GraphicsFormat.R8G8B8_UInt, (uint)Screen.width, (uint)Screen.height, 0, 75);
    File.WriteAllBytes(QueuedPath, jpgBytes);
    Globals.updateLoadGameMenu = true;
}

Additionally, you’ll need a shader to do the backbuffer flipping, so here’s something for that. Better could perhaps be done, but this is based off of an internal Unity copy shader, so maybe not much better:

Shader "Custom/ScreenshotFlipper" {
    Properties{ _MainTex("Texture", any) = "" {} }
    SubShader {
        Pass {
            ZTest Always Cull Off ZWrite Off
 
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #pragma target 2.0
 
            #include "UnityCG.cginc"
 
            sampler2D _MainTex;
            uniform float4 _MainTex_ST;
 
            struct appdata_t {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
            };
 
            struct v2f {
                float4 vertex : SV_POSITION;
                float2 texcoord : TEXCOORD0;
            };
 
            v2f vert(appdata_t v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                float2 uv = v.texcoord.xy;
                uv.y = 1-uv.y;
                o.texcoord = TRANSFORM_TEX(uv, _MainTex);
                return o;
            }
 
            fixed4 frag(v2f i) : SV_Target
            {
                return tex2D(_MainTex, i.texcoord);
            }
            ENDCG
 
        }
    }
    Fallback Off
}

Hopefully this has been a fun deep dive for you more technically-minded folks. Lord knows it was a lot of work to write it, but I had fun too. 🙂

71. Trees in the Wind

I’ve wanted to get animation for the trees into the game for a while, but could never quite manage it. I’ve tried many different approaches to this, but always the main issue is that the game renders at a fixed low-resolution pixel grid.

Some other pixel art games choose to render the game at a higher internal resolution than that of the actual pixel art, which means that you can rotate sprites without much aliasing. You can see a comparison between a high-res rotation-based animation (left) and how the effect breaks down when you render at a low resolution (right):

I have never liked the look of pixel art games that mix different resolutions, so I chose to render Taiji in a way that would force all effects in the game to be rendered at the same resolution of the base pixel art. But as you can see above, this means that rotating pixel art tends to cause strange artifacts that sort of look like edges sliding across the image. Obviously, this is very unaesthetic looking, and we need to try something else.

One possibility that I tried was to add in some noise to attempt to jitter out the sampling and create a smoother look. This removes the “sliding edges” appearance, but ends up adding in a lot of noise along edges. The effect could perhaps work well with a game that has a more forgiving art-style with a lot of noise built into the graphics.

So, with a couple of failures under my belt, I decided to rule out large motions such as rotating the entire tree, and instead I focused my efforts on animating the leaves on their own. This type of effect can be done fairly easily in a shader by simply adding in a noise offset when you sample the texture for the leaves.

This is certainly an improvement, but the effect is a bit too strong. Also if you look at it closely, it feels more like the tree is underwater than being effected in the wind. We could tone the strength of the distortion down, but then the motion becomes so subtle that it’s almost not worth having.

Another possibility that I attempted was to custom author a secondary texture which would control how the distortion was applied. I tried using a noise texture with leaf pattern built into it. I even did some tests pre-rendering leaves with Blender so that I could use the scene normals of the leaves to modulate the distortion.

I didn’t save this iteration of the shader, but suffice to say that it did not work much better than the purely random noise I was using earlier.

However, I started to think that an approach similar to how I animated the grass would be effective. The grass is essentially just a flat texture on the inside, with all the distortion happening along the outside edges.

So what would it look like if I did the same for the trees?

We’re getting close! This effect is even more pleasing, with a better balance between the details of the original pixel art and significant enough motion to be worthwhile. However, the motion feels a bit unnatural because it is confined completely to the outside edges.

What I chose to do to resolve this was to re-incorporate the idea of having a secondary texture control where the distortion effect can be applied. When used to highlight the internal edges, this forms the final effect. The wind map texture is below on the left. You can see that some interior pixels are colored red, those are the ones that are allowed to be distorted in the final effect on the right:

Overall, I’m pretty happy with how this came out. It adds some much needed motion to the trees, giving those scenes a more dynamic feel, and it doesn’t distort the base pixel art so much that it feels unnatural.

For a fun bonus, remember when I said that the unconstrained effect looked like water? I ended up using the same effect for this reflection in the water:

70. It’s All Mine

The past month has been spent working on the art for the Mine area. At this point it’s complete apart from the indirect lighting pass. I’ve already started implementing the art into the game, as this is a time-consuming process in its own right. There are lots of details that you have to figure out at that point which are easy to overlook when just doing art in a paint program. Some of the obvious ones though, are sorting order issues and putting trigger volumes all over the place.

I’m overall pretty happy with how the art for this area has turned out. A pattern that I’ve noticed is that each new area comes out a little bit closer than the last area to what my original vision was. It’s been a big learning experience working on this game.

That brings me to another point which I’ve been ruminating a bit on lately. As I get closer to the finish line for the game (there’s only two areas left), it becomes much clearer what the final form of the game is likely to be. Although this is exciting, it also ends up being a bit depressing, because it becomes clear that the game will not quite live up to all of my own hopes for it.

Don’t get me wrong, I think the game is in many ways much better than any of my initial ideas suggested, but there’s still this sense of wistful potential. I left many of my early ideas on the cutting room floor in order to make a more focused game, and I still feel a longing for a game that doesn’t exist. Perhaps I’m too close to the project and I can only see it’s flaws. Perhaps this is just a general feeling that I will always have, that will continue to drive me to make more games in the future.

Thinking about the future, I’m not really sure what I will end up doing after Taiji is finished. At the moment, I try not to think about it too much, because I still have a long road to finish walking down, and I don’t want to lose focus by finding distraction in some new shiny idea.

Part of me wants to quit games and never do this again. I can’t lie, this has become quite a grueling process. Working on a game solo has its upsides, but its very easy to lose motivation and feel isolated and lonely.

Oh well, we’ll see what the future holds when we get there. For now, I’m trying to decide how much I will be able to afford going back to areas that I’ve already done and improving the artwork there. I’m glad that many of you have had nice things to say about the look of the game, since I don’t consider myself a particularly great artist. Still, I have made most of the progress on the game by trying to keep a mantra of “good enough for now”, and part of that was consoling myself with the possibility that I could go back and revise stuff that I don’t like “later”.

The fact is though, I don’t have an infinite amount of time, money, or energy to work on this game, so at some point: “later” has to become “forever”.

This is standing out in my mind more because part of the Mine area connects with the Shrine area, and while working on the art for the Mine, I availed myself of the opportunity to improve that connecting part of the Shrine. This was something that has been on my list of “go back to later hopefully”, and I found it made me really happy to bring that small area more in line with my original vision.

Again, we’ll see how much of that I can actually afford to do before I have to ship. At this point the focus has to be on making a mad dash towards having something “shippable”. I suppose that’s typically called a beta; where you have all the features in the game and could theoretically put the game out, but you are taking time to polish and fix bugs and such. I honestly am looking forward to that point much more than I am looking forward to actually shipping the game.

Gallery!

For the last couple months (!), I’ve been working on doing the art for one of the areas in the game. This area is called the Gallery, and is a large manor which has been repurposed to house many works of art and puzzles. This area marks the largest and most complex single interior that I’ve made for the game so far. Below you’ll see the exterior on the left, as well as a diagonal cutaway to show all the different interior floor layers.

Needless to say, this was a time consuming effort, and I’m glad to finally be finished with the area. I have also been working on tying it into the surrounding environment, which will take some more time. Below you can see the area near the entrance, which features a winding path down a cliff-side.