85. Delays, New Music, and Parsing

Today I finally finished up the ending related stuff I’ve been talking about for months. This was the last major task before I could ship the game. Except that I’ve just decided to add a bunch more tasks to the list.

It’s a bit of a long story, but one of my testers, Gregory Bednorz, has gone from writing a bit of guest music for the ending to being heavily involved in helping me improve and expand the rest of the music in the game. You can see a bit more about that and hear some samples of the new music (and the old!) in the latest video devlog:

I don’t want to rehash everything that I discussed in the above video, although I will say that I am still weighing different options as far as whether the game should release in some way before the new music is finished. There are reasons to release early and reasons to wait, and I am not 100% decided yet.

I was hoping to be a little less “all tech deep dives all day erry day”, but with that said, I think it could be enlightening to some to show how simple it was for me to implement a composer-editable configuration file format for the new music system.

I felt that the results would be better if Gregory was not only composing music tracks, but was also involved in the implementation of the tracks into the game. But since I didn’t want to send the whole Unity project back and forth or attempt to institute some type of version control system this late in development, I felt it would be good to have a configuration file system and dynamic asset importing.

The dynamic asset importing was actually more of a challenge than the parsing, so perhaps I will come back to that another time. Suffice to say that the development version of the game pulls in .wav files and generates a set of configurable audio layers based on the filenames. Then these filenames can be referenced and set up in a configuration file as follows:

zone Orchard1
{
    osc_frequency_low 0.03
    osc_frequency_high 0.07
    osc_transition_interval 10
    osc_transition_duration 3
    
    add WindLow
    add Orchard_FlutePadsLow
    add Orchard_MatthewFluteSolo
    
    tweak Orchard_MatthewFluteSolo offset_volume 0.5
    tweak WindLow osc_low_volume 0.1
    tweak WindLow osc_high_volume 0.4
}

layer WindLow
{
    osc_low_volume 0.03
    osc_high_volume 0.25
    offset_volume 0
    wind_edge_influence 0.5
}

layer Orchard_MatthewFluteSolo
{
    osc_low_volume 0
    osc_high_volume 0.1
    offset_volume 0.4
    wind_edge_influence 0.2
}

layer Orchard_FlutePadsLow
{
    osc_low_volume 0.31
    osc_high_volume 0.44
    offset_volume 0
}

The way that this works, is that you set either a target zone or layer and then you configure settings for the target. Zones are areas in the world that contain only certain layers of the music and oscillate the volume of those layers up and down over time. Layers are just looping music tracks that have a volume that is tied to the high and low points of the current zone’s oscillator. The oscillator itself is just a sine wave which periodically blends into a new random frequency. This allows the music to change dynamically over time in a somewhat unpredictable way.

So, now let’s talk about parsing.

The main way that I kept parsing simple was that I decided that all state would be confined to individual lines, with only two exceptions: the current zone and the current layer. The brackets and indentation are purely for readability purposes and are ignored by the parser.

Because the inline code viewer on wordpress is horrible, and the pastebin one is slightly less horrible, here is a link to the entire relevant bit of code on pastebin.

The first thing that we do in terms of parsing is to point the C# standard StreamReader class at the file. I’m definitely not claiming this is the best or the most efficient way to parse a text file, but it’s what I’m familiar with from the save system, so I chose to reuse it here:

StreamReader sr = new StreamReader(file);
string line = "";
int active_line_index = 1;
while(sr.EndOfStream == false)
{

Once we get past this, we are now in the main loop, which is repeated for each line. First we create a cleaned version of the line by trimming any leading or trailing tabs and spaces.

stored_identifier = "";
line = sr.ReadLine();
line = line.TrimStart(' ', '\t'); //trim tabs and spaces
line = line.TrimEnd(' ', '\t');
int remainingChars = line.Length;
bool already_added_tweak = false;

Now we begin parsing through the line itself and breaking it down into tokens. A token is essentially just any single “word” in the file, so its just the next bit of text until there is a space. So we take the subset of the line until the next space and store that in a token string so we can interpret it later.

while(remainingChars > 0)
{
    string token;
    int tokenEnd = line.IndexOf(' ');
    if(tokenEnd == -1) 
    {
        token = line;
        tokenEnd = line.Length;
    }
    else token = line.Remove(line.IndexOf(' '));
    remainingChars -= token.Length;
    if(tokenEnd+1 < line.Length)
    {
        line = line.Substring(tokenEnd+1);
        remainingChars = line.Length;
    }
    else remainingChars = 0;

Now, we match the string against a set of keywords: #, zone, layer, add, tweak. The # is just a comment marker and is set to discard the rest of the line past it. The others all manipulate a line-level state variable which will be used in order to interpret the next token. If the token matches one of these keywords then we set the line-level state and parse out the next token on the line based on that state.

switch(token)
{
    case "#":
        remainingChars = 0;
    break;
    case "zone":
        state = states.SET_ZONE;
    break;
    case "layer":
        state = states.SET_LAYER;
    break;
    case "add":
        state = states.ADD_LAYER;
    break;
    case "tweak":
        state = states.TWEAK;
    break;
    default: //identifier, action depends on current state

When the token doesn’t match any of the keywords, this means it must be parsed differently depending on the current state. If we’ve already parsed a token on this line, and for example, that first token was zone, then the state will be SET_ZONE and we will parse the next token as though it was the identifier of a music zone that we want to set to be the target for future modification. This target, c_ActiveMusicZone, along with c_ActiveMusicLayer are the only state that is allowed to persist across multiple lines.

switch(state)
{
    case states.SET_ZONE:
        MusicZone targetZone = FindMusicZone(token);
        bool created_new_music_zone = false;
        if(targetZone == null) 
        {
            targetZone = new MusicZone();
            targetZone.name = token;
            created_new_music_zone = true;
        }
        if(created_new_music_zone) musicZones.Add(targetZone);
        else
        {
            targetZone.tweaks = new List<Tweak>();
            targetZone.layers = new List<MusicLayer>();
        }
        c_ActiveMusicZone = targetZone;
    break;

And we do the same if the first token was layer or add, setting the active music layer or adding a music layer to the active music zone, respectively.

case states.SET_LAYER:
    MusicLayer targetLayer = FindMusicLayer(token);
    if(targetLayer == null) 
    {
        Debug.LogError("File: "+file+" Line: "+active_line_index+" Attempted to set a Music Layer active that does not exist! Please verify that the names match.");
    }
    else c_ActiveMusicLayer = targetLayer;
break;
case states.ADD_LAYER:
    MusicLayer targetLayer2 = FindMusicLayer(token);
    if(targetLayer2 == null) 
    {
        Debug.LogError("File: "+file+" Line: "+active_line_index+" Attempted to add a music layer that does not exist! Please verify that the names match.");
    }
    if(c_ActiveMusicZone != null) c_ActiveMusicZone.layers.Add(targetLayer2);
break;

The next possible keyword and state would be tweak, which is a way for a zone to override the settings of a specific layer. This makes things a bit complex, because we have to parse out an identifier for tweaking as well as a value, which somewhat duplicates the default path when the token doesn’t match any keywords. I’ll confine the explanation of identifiers to that path, but here’s the tweak-related code anyhow:

case states.TWEAK:
    if(already_added_tweak == false)
    {
        c_ActiveTweak = new Tweak();
        c_ActiveTweak.target = FindMusicLayer(token);
        if(c_ActiveTweak.target == null)
        {
            Debug.Log("File: "+file+" Line: "+active_line_index+" Unable to find target for tweaking \""+token+"\", perhaps it is misspelled?");
        }
        already_added_tweak = true;
    }
    else
    {
        if(c_ActiveTweak != null) //we only parse the rest of the line if we successfully found a tweak target
        {
                
            float value1;
            if(float.TryParse(token, out value1))
            {
                c_ActiveTweak.value = value1;
                switch(stored_identifier)
                {
                    case "osc_high_volume":
                        c_ActiveTweak.type = tweakType.OSC_HIGH_VOLUME;
                    break;
                    case "osc_low_volume":
                        c_ActiveTweak.type = tweakType.OSC_LOW_VOLUME;;
                    break;
                    case "offset_volume":
                        c_ActiveTweak.type = tweakType.OFFSET_VOLUME;
                    break;
                    case "blend_duration":
                        c_ActiveTweak.type = tweakType.BLEND_DURATION;
                    break;
                    case "wind_edge_influence":
                        c_ActiveTweak.type = tweakType.WIND_EDGE_INFLUENCE;
                    break;
                }
                if(c_ActiveMusicZone != null) c_ActiveMusicZone.tweaks.Add(c_ActiveTweak);
            }
            else stored_identifier = token;
        }
    }
break;

Next we have the general case, which is when we didn’t match a keyword like layer, tweak, or add. In this case, it’s assumed that the token is an identifier for a variable or the value that we want to set a variable to. So first we attempt to parse the token as a value, and if that fails then we store the token as the identifier of a variable which we will set after we parse the next token on the line. There is no overlap between layers and zones when it comes to the names of variables, so as long as there is a current active zone or active layer, we can set the appropriate values just based on the identifier.

default:
    //we must assume that the token is intended to identify a variable or a value, so we will check against a list of known variable names
    float value;
    if(float.TryParse(token, out value))
    {
        switch(stored_identifier)
        {
            case "osc_frequency_low":
                if (c_ActiveMusicZone != null) c_ActiveMusicZone.osc_frequency_low = value;
            break;
            case "osc_frequency_high":
                if (c_ActiveMusicZone != null) c_ActiveMusicZone.osc_frequency_high = value;
            break;
            case "osc_transition_interval":
                if (c_ActiveMusicZone != null) c_ActiveMusicZone.osc_transition_interval = value;
            break;
            case "osc_transition_duration":
                if (c_ActiveMusicZone != null) c_ActiveMusicZone.osc_transition_duration = value;
            break;
            case "osc_low_volume":
                if(c_ActiveMusicLayer != null) c_ActiveMusicLayer.osc_low_volume = value;
            break;
            case "osc_high_volume":
                if(c_ActiveMusicLayer != null) c_ActiveMusicLayer.osc_high_volume = value;
            break;
            case "offset_volume":
                if(c_ActiveMusicLayer != null) c_ActiveMusicLayer.offset_volume = value;
            break;
            case "blend_duration":
                if(c_ActiveMusicLayer != null) c_ActiveMusicLayer.blend_duration = value;
            break;
            case "wind_edge_influence":
                if(c_ActiveMusicLayer != null) c_ActiveMusicLayer.wind_edge_influence = value;
            break;
            default:
                Debug.LogError("File: "+file+" Line: "+active_line_index+" Attempting to modify an unknown variable, please verify the name!");
            break;
        }
    }
    else stored_identifier = token;
break;

And that pretty much covers the parser, apart from that we reset the state between lines and increment the line counter so we can report errors a bit more helpfully.

Is it even remotely bulletproof? No. But for something quick and dirty that was mostly written in a day, I think it gets the job done, and hopefully can serve as somewhat of an inspiration against the idea that one should always “just use JSON” or that “parsing is hard”, or whatever other nonsense excuse you might have to not just write your own format.

If I were to think of some things that definitely could be improved, the main would be making sure that all error cases are handled better. For the most part, the format is simple enough that even if there is a major error, the file should still parse for the most part and just report the error without putting the game into a broken state. However, there may be some error states which I am not reporting, which could bite us in the butt on down the line.

Another avenue for improvement would be to have a file version marker. Since I expected these files to be human edited, I didn’t want to mark up all of the variables with some type of version information, but at least a file level version marker could be useful if there are some changes to the format that I want to make. This could facilitate automatic conversions of older files to a new format. With that said, this is a relatively minor concern, since there’s really only Gregory and myself working with these files, and I can easily just convert the files and pass them back to Gregory.

49. Slow Speed: Deep Owls

For the past several weeks, besides working a bunch of Saturdays at the day job, and chopping up a bunch of firewood (I rely on wood heat in the winter), I’ve been busy with some logistical behind-the-scenes stuff on the game. Unfortunately, that means not a ton of progress has happened on the game proper.

The project is reaching the stage where the list of planned tasks is ballooning faster than I can keep up with it. Although I have quite a few new ideas and mechanical changes that I’m excited to implement, I need to address the problems that I already know about before I add any new ones to the list.

Therefore, the next step will be to focus on finishing a test build. It’s been over a year(!) since I last sent the game out to testers. I’ve done a tiny bit of spot testing here and there, but the game just hadn’t changed enough to make a full test worthwhile. There are still large sections of the game that I know how to improve but I just haven’t gotten around to yet.

Music

I also started working on some music sketches, and I’m relatively happy with the initial direction there. I’ve attempted music for the game a few times already but never found anything that really clicked. There are also some pretty harsh constraints on the things I am even allowed to do with the music. I don’t want to spoil things too much, but Taiji can be very subtle with the clues to puzzles so it’s quite easy to accidentally create red herrings while just trying to make things look or sound interesting.

Now I have a few pieces in various stages of completion, but it will be a while before I want to share anything publicly yet.

17. Woah, Dave, working on Sounds

I actually cracked open Unity and worked on the game project yesterday. My thinking was that I had some people who were interested in the game remark about wanting to test it, and I don’t really have a properly playable build of the game at the moment.

So, I opened it up and proceeded to work a bit more on the interface transition, although I am still not entirely sure what the final approach should be. I also found a few bugs and quirks along the way.

One bug was due to the way in which I was determining what tile the player was standing on. This was due to a decision, which seemed innocuous at the time, to test a rectangle instead of a single point. This normally causes no problems, but there are certain degenerate cases when the player walks across a field of tiles that the tile which is currently highlighted will sort of ping-pong back and forth between the tile below and also above the player. I am again, not entirely sure why this is the case, but switching the test to a point test seems to have resolved the issue, and feels much better from an interaction standpoint.

The other thing that I tried out, was adding some more feedback to the puzzles in the game in the form of audio. Thus far, playing the game has been entirely silent, and I started to wonder if some of the complaints about the feel of interacting with the panels in the game had to do with the fact that there was no audio feedback to what the player was doing. I unfortunately do not have a properly functioning version of the cursor interface to see if the audio feedback would be sufficient there. I sort of broke it when  I implemented the direct interface. I was hoping to keep them side-by-side, but that was somewhat more challenging than it initially appeared it was going to be. I may still decide to re-implement a cursor just to see if it is better, but I’m not sure. Main priority is just to get the gameplay up and running again.