Hey all, welcome to this week’s Thursdoid!
Our weekly Wednesday test went a teeny bit worse than last week, which was a little disheartening at first – however don’t fear too much, as it transpired this was mainly down to the last of the new features required for the MP release going in this week that were still a little untested and broken, and not down to any new issues with the net code. The initial version of the stealth system that will allow players who are hidden in dark corners or behind fences to have a chance to remain hidden to other players went in, and there are still many goblins to exorcise, leading to players being unable to spot players next to eachother. Likewise, changes to the safehouse system had various issues that slowed things down. And yes, bloody teleporting zombies! We’ll get into those in a minute.
Here are a couple of highlight videos from the test session!
Andrei and Yuri are still working on the systems that we hope will vastly improve the zombie experience, and eliminate all zombie teleporting that could potentially impact the player, however these did not make it into the Wednesday test and its possible may not make it into next weeks yet.
As we draw closer to the finish line, it’s becoming clear that just saying ‘we’re still fixing teleporting zombies’ isn’t enough to get across the work ahead, why it’s taking so long and so on. We hear the frustration and so thought we’d go into a little more detail.
This week we thought we’d go on a bit of a deeper dive into what our goals are for fixing the zombie movement and providing a near single player experience in zombie combat, as well as the general problems we and any multiplayer game face. Hopefully after it you will appreciate why this is such a tricky puzzle to solve and why ‘teleporting zombies’ are currently a thing, and how we plan to mitigate it, as well as some comfort that this effort will lead to something pretty special when we finally pull it off.
ZOMBIE SYNCRONIZATION CONUNDRUM AND CHALLENGES
The first thing you need to know about with latency, is it’s impossible to know where a character is in a game-world NOW on a PC that is running in another country. The latency means there’s always an inherent delay and thus often in games there needs to be a bit of illusionary duct tape applied to ‘make it all make sense’. You take a headshot at a character in an FPS, well the other player is NOT at that location by the point you take the shot.
But you have to pretend they WERE still at that location, to make the experience make sense for the shooter. This is why for example you have peeker’s advantage where a player can run around a corner and shoot someone before they even see that player run around the corner. It’s also why you may run around a corner out of line of sight of an opponent and still get shot. It’s a pain in the ass but its just an immutable law of online gaming.
There’s this idea of ‘truth’, where things are truly said to ‘be’ within your game world, and while this is often the server’s responsibility, sometimes it needs to be determined on the client. Of course this is also where hack vulnerabilities come from, since clients can be altered.
Things get a bit worse when you consider the scenario where you have hundreds of entities moving around. In old B40 multiplayer, the server was always the source of truth for where zombies were, and clients were always looking at out of date information. They never attempted to ‘guess’, using client side prediction, where zombies were GOING to be, which would have helped a bit.
There’s also the issue of server load, one that players on b40 servers will be all too aware of. Imagine 50 players, all in different areas, in different towns, with different areas of the map loaded up. Now your VPS server is suddenly tasked with running the AI, pathfinding and logic for 50x the amount of zombies a single player client has to be, as well as other tasks of updating the world, streaming, altering and saving the map data. This is in a nutshell why our servers operated so poorly compared to many other multiplayer games.
Even Left4Dead, a game that people marvelled at the zombie counts, variety and consistency in multiplayer, had a static map and only 4 players, all of whom were usually in the same area and experiencing the same zombies. Further more one zombie being in slightly the wrong place wouldn’t lead to an insta-death situation because the players were immune to being infected. So yeah, an isometric indie game we may be, but this is a humongous challenge and a big achievement when we get there.
So yeah, lots of challenges to overcome, and approaching this problem we’ve made various steps, with the current work hopefully being the last piece of the puzzle.
STEP 1) Not just client prediction, but client simulation.
So first we thought: Instead of just sending positions the zombies (or players) are in, why not say where they are GOING and let them go there themselves, using the client’s own AI algorithms. Periodically checking in with the server to make sure the server hasn’t ‘changed its mind’, and teleporting the character or zombie if they deviate too much from the server’s ‘truth’?
This is a great first step, as it’s resulted in the complete removal of the juddery, flickery, rubber bandy movement of every zombie and player, marching in lock step with positions a struggling, overworked server is managing to spit down to clients in its busy schedule. It’s why in general zombies and particularly the much less predicable players movement was so janky in B40 MP. Now our characters are butter smooth in their movement at all times, moving very much as they would in SP because they are, effectively, running the same code.
PROBLEM 1) So that’s great, but now we have a new problem. It’s very possible for characters to end up, due to unexpected events that have not yet reached the client, in very different positions to where the client believes them to be. What does this mean? Well, we need to teleport zombies to where they ACTUALLY are. So whenever one of these butterfly effect ‘divergences in the timeline’ happen, we have to detect that has happened and correct the record so the clients agree again on where the character is.
Why do these divergences happen? Well it all comes down to latency again. Imagine two players with 200 ping each. Player A shoots a gun. It takes their gunshot 1/10th of a second to reach the server, where it registers and dice rolls are made to determine if zombies respond to it. A zombie responds to it. Meanwhile, on Player B’s client, that zombie is happily walking toward Player B. It takes a further 1/10th of a second for that zombie responding to the gunshot to reach Player B’s client, where it is told instead to walk toward Player A. Problem is now the position it is walking from is different. Either we teleport the zombies every second to keep them in as close to the right position as possible, or their positions will begin to veer further and further from the server ‘truth’ and we have to do a big teleport to adjust. It’s a bit of a catch 22 situation.
So this is the problem we’ve been trying to overcome since then.
STEP 2) CLIENT (soft) OWNERSHIP
A big thing we wanted to fix was the potential for lag bites, this is instrumental to making such a harsh game where a single bite can end your character work in a multiplayer environment. It should NEVER be the case that a lag spike results in a zombie teleporting next to you and having bitten you twice between two frames of your game.
The solution we came up with for this is for the client to ‘call the shots’ about zombies that are close to them AND chasing them. This means for all intents and purposes, a single player on their own on a multiplayer server, when engaged with zombies, will have a COMPLETELY single player experience with that zombie. There’s no potential for lag bites.
Of course, this does open the door to client side hacks, and this is something we’ll touch on later in regard to our plans, but given PZ is generally a more PVE oriented game than say DayZ or any other game which is a more solidly PVP experience, we feel this is the sensible route to take, and we can deal with protecting from hacks in another way.
So this works, and makes a huge difference. There are two problems it causes or exacerbates, however: Firstly, the moment that a zombie becomes ‘owned’ by a client causes one of these aforementioned teleports, to get it into the correct position the server says it is before the client can start accepting the zombies position on the client as its own ‘source of truth’. The other problem is when multiple players are present, we now exacerbate the problems with clients disagreeing on zombie positions.
This is where we are at currently, and now we’ll go onto STEP 3 which is currently in the works, will likely take a little while as its the biggest and boldest strategy yet, but hopefully once it’s in and balanced we’ll be looking at a much smoother online experience, and the remaining issues to resolve will be much more minor by contrast and we’ll be gearing much more toward sending this out there to you lovely people.
STEP 3) COMPLETE CLIENT OWNERSHIP
Again, this will likely lead to further vulnerabilities to hacks, which we’ll address later. However, the ramifications for this are huge.
First off, it has to be said that on an internet with more than 0 latency, getting rid of any corrections of character positions is impossible. We’re not looking to eliminate teleporting zombies completely, that’s a fools errand. It’s just a case of making it as mild and inobtrusive and non-immersion breaking as possible, and removing any potential for these corrections to lead to a player’s death.
The current plan is for zombies to be completely simulated by clients, and clients alone. With the server acting more like a caretaker who looks over the zombies in a much broader sense. When a character moves into an area of the map, and zombies are spawned, they will simply run on that client. The server will simply dish out where these zombies are, and leave the client to it. This will mean to that client, there will be no teleports, no lags, the zombies are identical to single player zombies, end of story.
The further beauty of this is that the server no longer needs to stress itself to high heaven trying to process the AI for all these zombies, all it needs to do is a much lower effort tracking of zombies to make sure no naughtiness is taking place and zombies aren’t behaving in an odd way that suggests they are being fiddled with on the client via hacks. Suddenly, you have a server with say 64 players, the server is now handling AI for 0 zombies, instead of potentially thousands or even tens of thousands. Who knows what player counts we could entertain as a possibility without having to worry about the exponential rise in zombie processing per player as we have done historically.
Outside server load, this also means that players enjoy a completely single player experience, at least if solo. There are still a few complications when it comes to multiple players witnessing or interacting with the same zombies however.
If player A) is being attacked by a zombie, and player B) then hits that zombie in the back. It’s very possible that player A and player B will disagree where that zombie is. However, now, the ‘truth’ is where player A) says it is.
player A is being attacked, so the zombie is player A’s zombie. player B) hits the zombie in the back (perhaps not being stood directly next to it since player B thinks the zombie is half a tile away). So the zombie turns around and attacks player B). Now in our current version, with the soft ownership, what would happen here? The zombie would teleport to where player A believed the zombie is, meaning it would teleport on player B)’s client, the player who tried to rescue his friend, and potentially teleport to their side and take a bite out of their neck. That’s broken and awful.
However, the new system will always choose the client that is targeted or influenced by the zombie as the ‘source of truth’. So player B’s ‘incorrect’ position for the zombie will now be considered ‘truth’, and player A in this circumstance would be the one to witness any teleporting. This doesn’t matter however, because player A is in no danger from the zombie, the small fade in / fade out position correction has zero chance of negatively impacting them or leading to them getting bitten.
So ideally, there’d also be less of these teleports in the first place. This all comes down to improving the prediction and allowing clients to be closer in their ‘guesses’, but its just not possible to ever get rid of any teleporting without reverting to the much uglier laggy, jerky and rubber bandy movement from yesteryear, our goal is just to minimize it as much as possible, and as described above, organize it so that when it happens it’s always in case where it will never negatively impact a player that it happens to.
There will be also considerations made to lag, so a player with an extremely high ping will never be responsible for zombies that aren’t directly impacting them if a lower ping character is close by.
MP INITIAL RELEASE PUBLIC SERVERS
So the gist of everything above is, once we do get the first IWBUMs out there, we’re going to strongly recommend it purely as a co-op or whitelisted server experience. We’re not going to be able to adequately protect from shady behaviour without delaying the build even further after implementing this stuff, but we strongly feel giving players the ability to run co-op games or closed servers with friends, is more important than holding out and delaying for something which will be cast-iron protected from spoil sports who like to ruin fun on public servers. We hope everyone agrees with this decision, but we will of course do all we can during development of this system to add safeguards in there and expand that after the first release.
The benefits outweight the cost massively.
So that’s all for this, quite different, Thursdoid. Don’t get used to them as they take a lot of programmer time to put together, but we just figured that with the mounting hype and frustration at waiting for MP, a deep and detailed insight into what this ‘teleporting zombie’ stuff we keep talking about is all about and quite how high the bar we’re hoping for the zombie combat experience will be.
Thanks again for all the support we’ve seen from streamers and their communities, as well as of course our own. It’s been a tremendously exciting few weeks and we hope to get MP out there to you all as soon as possible! Love you all.
This week’s mourning cowboy courtesy of grenskei on Steam. A changelist of all IWBUMS patches since the 41 beta was released can be found here. A general list of stuff added to PZ, and vids of features being worked on, is kept here – so you don’t have to plough through endless dev blogs for info. The Centralized Block of Italicised Text would like to direct your attention to the PZ Wiki should you feel like editing or amending something, and the PZ Mailing List that can send you update notifications once builds get released. We also live on Twitter right here! Our Discord is open for chat and hijinks too!