Join our community

From Rollback with Love – IKEMEN Go

...

By Andrea "Jens" Demetrio | February 2, 2023

First off, happy 2023! Yes, I know I am a bit late with celebrations, as we are already deep in February, but – guess what? It’s time for a new year of SuperCombo articles, delving into indie and less indie titles, with the usual mix of interviews and reviews. The first article of 2023 is about a welcome, yet unexpected development in the IKEMEN Go community: rollback netcode is finally becoming a reality. If you have ever wondered how this technical marvel is actually coming to pass, don’t miss the interview with the main developer of the IKEMEN Go rollback branch: Fantasma!

Fighting game engines are an incredibly niche, yet interesting topic. On one side, you get developers writing their own contraptions in unholy combinations of languages and engines, like Hot Soup Processor, Löve 2D or even something that is not easy to point out. On the other, you have other developers, struggling to build general purpose engines for the masses to use and abuse. Such is the case of Unity Universal Fighting Engine 2, 2D Fighter Maker 2002, Castagne, M.U.G.E.N. and – last but absolutely not least – IKEMEN Go. We have already met this gentleman and its associated GitHub repository when talking about Battle Craze!! and TMNT x JL Turbo, two games that made ample use of its features.

In short, IKEMEN Go can be seen as a spiritual successor to M.U.G.E.N., written with the programming language Go. IKEMEN Go is almost completely back-compatible with stages and characters created for its predecessor and adds even more options to the mix – such as Lua scripting for finer control and online delay netcode. The delay nature of the online multiplayer, however, made it hard to play around the world and – somehow – stifled its appeal a bit: in a post-pandemic world, publishing a fighting game with delay netcode might be a commercial death sentence.

Wait, what is rollback netcode?

If you are new to fighting games in general, you might have asked yourself this question, while browsing through the opening paragraphs of this article. Volumes have been written on the topic, but let me give you a short summary: fighting games, contrary to most sport games and competitive shooters, are played online with a peer-to-peer connection. Which means that two players are connected to each other and exchange inputs, without a server in the middle running the game and checking that everything is going as planned.

Until around ten years ago, the standard implementation of online play for fighting games was the so-called delay netcode, which consists in the following:

  • The game runs on the machines of both players independently;
  • Every frame, the game waits for inputs from both players before proceeding;
  • Once the game receives all inputs, it proceeds to the next frame.

This rather simple implementation works relatively fine, but has several issues: first and foremost, the game must wait for the inputs of the opponent to step forward. Given that there is a physical cap to the speed at which information can travel, this means that the game must artificially add variable input delay to the local player to compensate for it. If the compensation isn’t long enough, the match can quickly become slower and clunkier than a PowerPoint presentation.

Rollback netcode tries to solve the problem of artificial input delay by operating in a slightly different manner: instead of waiting for the inputs of the opponent, it moves the game forward using their last known input as soon as the inputs of the local player are registered. Then, when the actual input arrives, the game compares it with what was used to advance the state. If there is a mismatch, the game “rolls back” to the last known common state (hence the name) and re-simulates the outcome based on the new information, immediately “jumping forward” to the new reality. The forebearer of this technique was the framework known as GGPO, recently released as open source code by Tony Cannon.

The advantage of rollback netcode is that it is easier for it to “mask” the delay, especially in games with slower pacing and animations, while keeping the input delay constant. Not zero, mind me, but at least fixed to a constant value, making it possible to keep consistency even while playing online. Compared to delay netcode, rollback allows people even farther away from each other to play with acceptable quality: a New Zealand – US connection on delay netcode would be almost impossible, while with rollback netcode it is at least playable.

Of course, rollback is not a panacea and still suffers from bad network connections and jittery Wi-Fi, but it’s an incredible advancement in terms of the amount of players one can have decent matches with, when implemented correctly.

Since a video is better than a thousand words, I’ll leave you a very good explanation by development studio Code Mystics, known for having implemented this kind of netcode in older games made by SNK.

Adding rollback is easier than you think (no, it isn’t. Not at all)

Suffice to say that implementing rollback netcode is not easy, especially if one needs to retrofit it into an existing engine. The developers of Mortal Kombat X needed to invest a considerable amount of time and manpower to do that, as outlined in their GDC talk, and it’s not something that can be done without a lot of planning and hard, sweaty code work.

One of the challenges is allowing your engine to “save” and “load” the current state of the game at arbitrary times, especially if said game wasn’t designed with this feature in mind. This is necessary to perform the eponymous “rollbacks” when the inputs received don’t match the prediction. IKEMEN Go couldn’t do this, as the game state of an engine with limitless possibilities is very hard to pinpoint and isolate. In order to tackle this, developer BWDYeti tried to build an embryo of a save state mechanism for it, but despite some heartwarming early success, they left it incomplete.

To summarize the whole ordeal in layman’s terms: adding rollback netcode to an existing engine not built for it in the first place is a gigantic pain in the neck – and IKEMEN was NOT designed for it AT ALL. Therefore, it was almost considered a lost cause in the IKEMEN community for a long while.

That is, until Fantasma joined the chat.

The GGPO connection

Thanks to a serendipitous set of circumstances and a Discord server dedicated to GGPO, a developer simply known as Fantasma decided to take matters in his own hands and take on the titanic task of integrating rollback netcode into IKEMEN Go. The development is still ongoing, but as of today, there has already been some promising progress, especially after the publication of the 0.1.7 rollback alpha build. I had the pleasure of witnessing the unfolding developments in the IKEMEN Discord server, which prompted me to have a chat with Fantasma about his current endeavor. Here’s the full interview text (slightly edited for clarity, emphasis and notes mine).

 

Q: How did it start? And why precisely IKEMEN?

A: I worked at a local ISP in college in their research department. I didn’t do any hardcore networking stuff at this time (mainly frontend intern work), but I received heavy exposure to it and this is where I got the idea of myself as a “networking guy”. I always loved fighting games, so naturally I became obsessed with GGPO, because it combined two things I was passionate about. When Tony Cannon open sourced GGPO, I immediately became a part of the GGPO open source community. I submitted a few pull requests about documentation about whatnot. I learned Rust specifically for GGPO [Note: GGPO is written in a programming language known as C++. Rust is another programming language].

I attempted to make Rust bindings for it. This never quite worked out, as Rust wasn’t the language for me. However, Go was a language I really liked and was able to actually build things in. So several years later, I get this idea to write GGPO in Go. This started out as a toy project solely because I was interested in writing a “decentralized P2P network protocol” (whatever that means) in Go, like a few that I had done prior. I never expected or believed anyone would care about it, as from what I knew at the time, nobody made games in Go, let alone fighting games. I did not even want to make a two-squares-prototype for it, believing that I didn’t have the game development skills to do even that.

At some point (I believe June 2022) a user on the GGPO Discord named ValorZard said “oh, you’re making a rollback library for Go? That could be used for IKEMEN rollback!” and invited me to the IKEMEN Discord’s dev room. From there I was off and running. I got a lot of enthusiasm and encouragement (from people who I now know are prolific and legendary MUGEN/IKEMEN creators) which meant a lot as someone who’d never made anything that got that sort of attention before, so I stuck with it. I saw IKEMEN as sort of “just a small open source project” until about 6 months in, when I announced rollback to the public, when I was told how significant this could be.

 

Q: Retrofitting rollback in an existing engine is usually thought to be a nightmare by developers. What were the biggest challenges in interacting with the IKEMEN code base?

A: GGPO requires discrete functions for reading inputs, processing the frame, and rendering. IKEMEN’s normal game loop isn’t (explicitly) organized in that way. It’s kind of a rat’s nest of game logic really (I say that lovingly), a lot of which happens in anonymous functions.

With regards to the retrofitting of rollback, figuring out how to organize the code for rollback in the beginning was the hardest thing. I was unsure if putting rollback in IKEMEN was even possible without a full refactoring of the engine. I did multiple rollback prototypes where I tried to rig up rollback within the existing systems using replays, before I actually integrated my library. At some point I just decided to throw my library in there, wrap things up into ‘RunFrame’/’Render’/’ReadInputs’ functions, wrap the anonymous functions up into methods, and it just sort of “worked”. I try not to think too hard about it.

Kirby Battle Blitz rollback netcode tests

 

Q: IKEMEN supports an additional Lua scripting system, compared to the original MUGEN. How did this feature influence your approach to the problem?

A: Well, it didn’t influence it initially. I went in totally naïve about the ramifications of that. When I was finally able to reach the end of matches in the Desync Test, I’d always get desync errors and after more than a week spent trying to understand what was happening, I realized it was because Lua code was not being run and Lua state was being restored in rollback [note: a desync (short for desynchronization) happens when the state of the game for one players becomes different from the state of the game for the other player, effectively making it impossible to go on with the match. Examples of desyncs include character landing in different positions, health values not matching, hits not registering only for one player and so on].

From that point forward, I tried many things. From trying to “clone” the entire Lua state object we use for interacting with Lua (didn’t work), to forking the Lua virtual machine we use (gopher-lua) , to add snapshotting (didn’t work), to ostensibly adding hooks that allow scripters to register certain Lua data to be rolled back (didn’t “fix” things but could help things to a certain extent).

I eventually realized that the Lua game loop works a little differently, and that I had to rollback and update the Userdata provided to the scripts. That’s what’s gotten me as far as I’ve gotten, but these are still open issues.

 

Q: IKEMEN characters have little to no limits on what can be done with them, in terms of variables and resources allocation. However, this comes out as a detriment when trying to determine the game state. How did you handle this specific aspect?

My mentality has always been to copy the entire game state first and ask questions later, for better or worse. It’s only recently hitting me that optimizing the rollback is so hard because the engine has no limitations. I never thought about that, I always just wanted the rollback to be as accurate as possible to the engine.

Right now, I’m using an experimental Golang feature called Go arenas [note: Fantasma wrote an informative post about Arenas and its application to IKEMEN, You can find at the following link]. Before, I’d have issues with other optimization methods. With pools, even if you can reuse the objects later, the initial frames wherein there were heavy allocations would cause slowdowns (especially when you consider that you have to keep data around for several frames before returning it back to the pool). Also, the fact that there’s no generic pooling system means that you’d have to create a pool for every type of objects (because any one object could destroy the GC [note: GC stands for Garbage Collector, a feature of some programming languages that automatically disposes of unused objects and frees memory]). With preallocations, it’s impossible to come up with a reasonable minimum or upper bound for how much data you’d need for any possible 8 frames of an IKEMEN round. I’ve tried many times but all it takes is a few supers in a row to wreck any of your expectations.

Go arena’s solve these problems. They allow you to allocate many different kinds of objects, completely off the garbage collector, dynamically. They present their own issues though, for example I’ve noticed myself that they can create strangely high memory usage on lower end machines, and are very experimental. But I hope this becomes a feature added to the language soon.

The IKEMEN port of Hyper Dragon Ball Z running on the IKEMEN rollback alpha

 

Q: Adding to this, are you planning to keep using Go Arenas even after Google decided to deprecate it out of the blue?

A: That’s an interesting part of the story that I think should be told. I ultimately decided to go forward with using them despite that because I’d set a deadline for myself to release the rollback alpha and couldn’t take them out in time. My understanding is that they are keeping them in there at least for the life of Go 1.20. There are more pertinent issues I have to focus on in the interim.

If/when Go 1.21 rolls around and they’re taken out, I’ll replace them with a hybrid pooling-preallocation system and just make it work. I still have maybe 1% hope that it shows up as a feature in some form, but I definitely have a backup plan. It’s just there’s other work to do right now.

 

Q: With the new rollback module, will existing IKEMEN characters need to be adjusted or are they going to work without major modifications?

A: That’s a good question. I’ll say that I have no current plans to ask creators to adjust their characters to be compatible with rollback. I will always prioritize making the engine compatible with all characters first. That being said, I am always shocked as to what characters work; work well in fact; with no modification. I think that’s tantamount to my approach of accuracy to the engine.

Technical tests of Battle Craze!! rollback on stream [channel: MisterMKL – link to the video]

 

Q: I have already seen some prominent games jumping on the rollback alpha. Can you tell me a bit more about how the feedback process worked?

A: There are some creators who I’ve worked very closely with. Kamekaze of the TMNTxJL team helped me figure out the issues with Lua, for example. Or Kokorz of the Kirby Battle Blitz team, who’s been very hands-on with me and providing feedback, having their players test new builds and reporting their experiences. I also interact with a lot of creators in the IKEMEN dev room, people like RollinJay who work on the (unofficial) Hyper Dragon Ball Z IKEMEN build. For the most part however, things have been pretty hands off. I find the nature of IKEMEN is just giving people a build and letting them work with it. There’s something magical about that that I’d definitely like to retain, though I’ll probably work more closely with people as I try to get 100% compatibility with the full games.

 

Q: What was the response of the IKEMEN community at large to the rollback alpha?

A: To be honest, I’ve been too busy working on bugs to pay too much attention to the response, unfortunately. I think the general response is that “when it works, it’s really good“, so I’ve been trying hard to make it work 100% of the time.

The picture shows the IKEMEN Kung Fu Gopher, a potential mascot for the engine. It's a blue gopher with a blue headband, wearing a karate gi, similar to MUGEN mascot Kung Fu Man.

“The answer lies in the heart of the battle!” – Meet the Kung Fu Gopher, IKEMEN’s potential new mascot, The design is inspired by the MUGEN staple unofficial mascot Kung Fu Man and by the Go gopher.

 

Q: What is your personal wishlist for the future of IKEMEN and your work on it?

A: I’m very passionate about IKEMEN and I want so much for it. In the broader scope of things, I want IKEMEN games to be at major tournaments, I want IKEMEN games to get platform distribution. I want IKEMEN creators to get deals. I want IKEMEN to get the recognition that I feel it deserves from the broader game community (and the Go community). I want there to be IKEMEN streamers, and IKEMEN YouTubers. I want IKEMEN to be at conventions. The whole nine yards.

As far as my future in the project and what I personally believe I can contribute post rollback? Spectators, lobbies, matchmaking. Nothing is in the works as I’m currently working on making rollback the best it can be, but that’s my goal

 

Q: If someone wanted to support the project (economically or as additional manpower), where could I direct them?

A: I have a Kofi link. Right now I’m working on IKEMEN Go’s rollback full time. If people would like that to continue, please support me on Kofi https://ko-fi.com/fant4sma.

Other technical tests of rollback, this time in a The King of Fighters compilation

The next steps

IKEMEN Go’s rollback implementation still needs to be worked on, but there are already a lot of positive takes around it. If this project ends up working as intended (and I wish Fantasma and the IKEMEN team all the best in this regard), we might enter an era where IKEMEN becomes the de facto standard for 2D indie fighting game development. Which would also mean witnessing wild online matches between characters such as Omega Tom Hanks and Ronald McDonald – now with rollback.

But this is a small price to pay for such an impressive breakthrough, isn’t it?

Engine summary

More from Community