Bitphoria's Release Date Announcement

 I have decided to make my 30th birthday the release date for Bitphoria, which happens to be August 30th 2016. There are several things I'd like to finish, among bug-fixes and networking improvements, before the first release and I thought I'd share that list here. Whatever state Bitphoria is in on that day it will be released.

It is obviously crunch time now (a great motivator for me) so the things on my list that I'd like to have done might be cut short, but I will continue working on it well into the future beyond the release. This will manifest itself in the form of new and updated releases. The initial release itself is meant to serve further development of Bitphoria by allowing other people to finally start messing around, playing games, scripting games, and providing me with bugs and feedback about everything.

I plan on releasing Bitphoria as a sort of public beta, for people to get to know it and see what I've dumped almost two years of my life into. Maybe down the road I will sell it for a few bucks on Steam/Itch.io. Maybe I should crowdfund polishing it, because I clearly have something that's worth something to someone, somewhere, out there, and I think it has a chance at being a crowdfunding success (after all the hard work is done already).

Hardware contributions/donations are always welcome! ;) I'm currently developing Bitphoria on four different machines: two low-end laptops from a few years ago, a low-end pre-built HP desktop from the same time, and a custom-built desktop - all with integrated graphics. I did manage to borrow an nVidia GTX 680 in the custom-built desktop system that allowed me to record some footage of Bitphoria earlier on, but have since been without and could definitely use a decent GPU just for the sake of tuning Bitphoria's graphics to where high-end systems can be fully taken advantage of for hardcore PC gamers, as well as allowing me to record updated videos of what Bitphoria looks and behaves like. Recording video is actually my primary goal insofar as GPU acquisition is concerned, because without it I will have a harder time demonstrating the game. Worst-case scenario hopefully fans out there will take it upon themselves to youtube videos of Bitphoria. If nobody does that then I've failed at making something that sells itself, obviously, and that's the goal of any project of mine!

If the case is that nobody ever cares about Bitphoria, and it just gathers cobwebs and dust in a dark corner of the web (on this blog, and in a few reddit posts), never amounting to anything more than just a "portfolio piece" for me to acquire a dead-end soul-crushing software engineer job, then at that point I will just release the code in its entirety for everyone to tinker with and learn from. At any rate, the code will ultimately be released eventually. The 'when' of that is still up in the air, and probably will be for at least another year. But if I can earn a living for my family via my lifetime hobby then I'm certainly going to try before giving it all away. Over two decades of programming experience has gone into this project, and there are even things that I learned along the way while developing it that I'd like to invest into a new game engine project, but those are things that I'm just going to hang onto until I feel that this project is 'done', through-and-through, which includes promoting it to a respectable degree that hopefully provides it with enough exposure to ascertain as to whether or not it is something that there is interest in.

As far as my pre-release todo-list is concerned I believe that setting up a master server for people to find each-others' games is imperative, otherwise the whole project is pretty pointless if people can only start a game server, run around by themselves, and then quit out and never play it again. Scripting features and functionality seem less consequential and could be added in along the way at a later time after the initial release, but the existing setup is rather competent. Finishing the scripting documentation also remains a top-priority issue due to the fact that I wish for people to be able to start playing around with the engine itself to see what they can come up with within the creative sandbox that it represents. It has its quirks and nuanced requirements insofar as game performance are concerned (graphically, physics-wise, and networking) which is the case with any game engine out there. However, I would like to point out that it takes no game-development know-how or modding experience to make something out of, or with Bitphoria. Anybody should be able to simply open up the script files that comprise the default games included with Bitphoria and start surfing the documentation to figure out how it all comes together.

Here's my list of things left to do for Bitphoria before releasing it that I'm currently focusing on for the next few weeks:


Master Server: Write and launch a master server to run here from my home to allow players to be able to start and find/join eachothers' games. Along with this I'd prefer to invest in a domain name that points to the master server (my house IP) for the in-game server browser to download the server list from. This would actually kill two birds with one stone. Firstly, it would eliminate the learning curve of PHP and MySQL that I'd need to traverse if I were to create an HTTP master server. I *do* have enough experience with both to make it happen but I'm not competent or well-rehearsed enough to simply knock it out within a day or two - and it would require that I implement HTTP request functionality into Bitphoria, be it by hand or integrating CURL/libCURL. Secondly, running a custom-coded master server from home would allow me to easily implement a NAT-punchthrough handshake protocol to allow anybody to start a game without being required to deal with their router/firewall and port-forward. Conversely, anyone else would also be able to join any game regardless of whether or not they themselves are also behind a router/firewall. This functionality requires that the master server notify game servers when someone is trying to join them, and from what IP/port, allowing the game server to go ahead and send a 'trailblazing' packet to the client player for the sake of tunneling through the NAT and have it route packets from that client player's address to the game server. To have a console application running in the background on my desktop, written in C, using winsock or SDL_net, that would be easy to manage would be much simpler and cheaper than any remote/online option I've come across so far. If at some point there's too much traffic for my home connection I'd then move the master server program to a dedicated virtual host and simply point the domain name at that, and everything will just continue working for everyone without any changes for end-users.


Network Buffering: Implement a network buffering system to, for one, allow for the simulation of latency and its fluctuation for testing/tuning purposes. So far I've only been able to generate repulsive/gross/erratic network behavior when my daughter watches Youtube/Netflix on her machine while I have two machines sharing a Bitphoria game over our wireless LAN. Also, in spite of my efforts to fight network update jitter using extrapolation when an update packet arrives later than intended, it would seem that Gaffer's strategy (gafferongames.com) of simply buffering network updates long enough to encompass most network jitter and then subsequently emitting them to the engine internals at the interval they were intended to arrive at would be vastly more effective in its precision and lack of interpolated correction. Buffering network packets as such would also allow me to simulate internet conditions locally without having to track down willing testers (I'm not really a 'people-person', and I have no more computer friends left) and fine-tune everything for what I like to think of as the 'fringe-market' - people who don't have high-end gaming systems (a netbook) or fiber-optic connections. I'm of the mind to release a game that looks prettier the better the setup, but also is completely performant on less ideal setups. Why focus on one area of the market if you can focus on the entire market?

Scripting System Documentation: Finish the scripting system documentation, which would include indications as to where inside of the the sample/default games that are packaged with the initial release that users can find examples of each script command, along with tips/tricks. Also, early on I made it a point to document everything that I script for Bitphoria because I know that people out there should be encouraged to tinker around and pull everything apart.

Artificial Intelligence: Add in simple AI functionality that allows an entity to seek out other entities and designate them as its target, which it can then follow or aim at. This would make it possible for simple zombie-type enemies that just follow players and harass them while they try to carry out other things. I'd like to include some kind of simple navmesh generation that is derived from the distance field of the world but I am thinking I am just going to literally index the distance field and use that for obstacle avoidance while entities try to pursue other ones. If a target is unreachable then it could back out following the distance field outward until the target is visible again, or a new target is found.

World Generation Modes: Add options for the style of the world generation itself. Right now it's a fixed algorithm and starting a game server entails selecting a random seed and then a vertical-scale which is forwarded to clients so they can then generate the same world themselves and play. The world is generated as a 128^3 volume and there's a lot more that could be done with it than just leaving one algorithm in there for people to experience.

Physics Attachment: Entity attachment which allows for an entity to literally attach itself to another entity and inherit its position/motion, with or without an offset that is oriented with the entity being attached to. This could be used for CTF games or power-up modes that want to display some kind of visible effect.


These are more-or-less listed in order of priority. What actually happens by the release date is still completely variable, these are just my intentions and goals, and it's hard to say what exactly will occur as I pursue each bullet point.

Bitphoria Screenshots

 I just felt compelled to post some new screenshots. Not much has changed as I've been working on underlying stuffs and writing the scripting manual, but it sure looks purdy.

Check back soon for more updates. I'm in the midst of designing the master server situation so players can start and find one-anothers' games easily without any port-forwarding nonsense. I'm also hammering out the Bitphoria Scripting Manual as a guide and reference for people who want to make their own games out of Bitphoria and share them with the world via the ingame server-browser. Players do not have to download anything externally to play custom Bitphoria games. It's a thing of beauty.

Game Logic Scripting and Networking

I've been very distracted from working on my project, and this blog, since the holiday season. Various circumstances are resolving themselves, finally, and work will resume. I've also been somewhat stalled out trying to wrap my brain around the topic of this post, and thought it wise to take something of a break from wracking my brain in pursuit of the 'ultimate solution'.

One of the important features of the engine is that it should be easily moddable. My goal is to not only produce a game for people to play, but also a game they can manipulate and customize to further derive enjoyment from. This is also something that I feel affords me maximum engine re-usability insofar as creating and releasing another game is concerned. I have a serious aversion to hard-coding game-specific behavior and logic, because it always gets tangled up in the rest of the engine code, making it a mess to change certain aspects of the engine when trying to build a new game out of it.

The top priority is allowing the people who host game servers online to customize the game in any way they like without players being required to manually download and install anything externally just to play. Server admins should be able to customize everything about the game that people experience when they join their game. Players should be able to see all game servers running on the same engine, and choose between the different games/mods that each server is running. Being that virtually all of the assets and resources used for generating the game experience are scripted procedurally, clients quickly download these procedures and 'rules' upon connecting and the entire game experience they encounter is dictated by the scripted configuration of the game on the server.

Games that are almost entirely hard-coded into the engine usually feature customization of the constant values for things like weapon damage amounts, and other little nuanced values like this, but the behavior of the game itself is otherwise 'stuck' the way that it is. Typically they have some sort of text file where the configuration exists, delineating variables and their values for controlling physics and game behaviors. This is simple enough, and plenty sufficient for smaller projects of a less serious nature.

Most games utilize some form of a scripting language to accomplish the de-coupling of game logic from the game engine itself. There are others which simply incorporate the use of an external compiled binary, e.g. a DLL file. Having a background in reverse engineering and 'hacking' games, I can say that using a DLL is probably the most insecure thing a programmer could do. Operating systems are equipped with all sorts of debugging APIs and features that enable hackers to have a field day with such games.

Another top priority alongside game customization is the quality of multiplayer networking and the resultant online gameplay. It's pretty standard now to just design a server/client model using all the usual tricks that have been around for the past decade to mitigate internet latency and packet loss, to smooth out the appearance of the gameplay that is actually occurring on the host machine that is being simulated remotely. Everything you see on the screen is a virtual lie, and the typical bag of tricks are designed with the intent to please the player with promises that can't always be kept.

It is my opinion that the existing techniques are sub-par and that it is time we begin to explore other options, and come up with new ideas. For my project I am turning conventional networking strategy on its head. In my networking model the client has equal authority as the server and other clients as far as the game state is concerned. The server merely maintains the game rules and authority over who can be connected and participating in the game. It also serves to route the game state between clients as it evolves. No single machine retains the absolute state of the game, and all machines are participating equally in the progress and simulation of the game state as it unfolds.

To make all of this possible, combining a sort of peer-based game state simulation along with client/server networking model, as well as keeping the system for user-made mods in a manageable and user-friendly state, I have opted to use a console-scripted system that is made up of a handful of smaller 'systems' of commands. Everything in the engine is scripted using sets of commands in this fashion.

There are three components to this setup. At the core we have entity 'types', which are a set of parameters that define a specific entity. Properties that don't change about a type of game object are represented as a 'type'. Things like a model, conditional logic, physics behaviors, etc. Properties that are consistent across all instances of an entity type are thus considered aspects of that type.

Secondly, we have entity 'functions'. These are small sets of 'operations' to perform on a given entity. Things like playing sounds, spawning particles, or entities, inflicting damage, etc. These functions are referenced by an entity type's conditional logic definitions. Conditional logic is hard-coded into the engine, there are only a certain set of conditions which the engine detects about an entity and, in turn, executes any logic for those conditions as defined in the entity's type. Conditions such as when an entity touches the world, or another entity, or gets damaged or killed, for example.

Functions can perform a number of operations, but they cannot change anything about the original entity type it is executing upon. However, if something is to change about a specific entity's type, it can simply be changed to a new type with different static properties. If a player is supposed to go from a walking physics to a ragdoll physics, simply change the player entity's type from "player" to "deadplayer", where the physics settings differ accordingly.

This makes the process and/or job of designing entities pretty simple and painless. They can be edited in notepad, and reloaded in a snap. It becomes easy to create variations of the default game.

It also simplifies the networking model. The typical setup most games use for networking the game state do so by 'delta-compressing' entity updates, comparing an older state to the present state to determine what aspects or properties changed and need to be transmitted. This allows developers to define any number of entity changes to occur over the evolution of the game state, and have everything reach from one machine to the other over a network connection.

My implementation boils this same design down around the fact that a lot of times there are many entities which have properties that never change over the course of their existence. These properties can all be lumped together and conveyed in a minimal number of packet bytes by simply indicating which entity should be which type, when that entity becomes that type.

The actual networking system continuously relays 'events' to the other side. The game logic, in the form of entity functions, is responsible for invoking events which have a networked component to them. Events like particles, sounds, etc. all are 'networkable events' - in that they should be seen by other participants in the game. These are queued up to be transmitted in the next outgoing update. An entity's type being set is an example of an event that is serialized and queued up for network transmission.

Not all entity function operations have a networked component. Some things are meant to only occur on the local machine, and even networked operations will stay local if the entity type is defined as being local-only (eg: client-side detail entities). If an entity changes into something completely different, and everything about it changes, this is not a large update. The local machine simply indicates which type the entity is now.

Along with the events queue is a prioritized list of entity positions, velocities, angles, etc.. All the location information about an entity gets tacked onto the update after all the events. Positional updates are 'optional', in that they don't always need to get to the other side the way events are supposed to. Entity positions are prioritized by the entity's proximity to the client's player entity. Entities within a certain distance of the player have their positional information included with every update being sent to them. Once entities become further out, the number of updates they are included in per second begins to lessen down to a bare minimum.

This is a screenshot of the ill-fated Revolude game, circa 2010.

Now one idea I had, back in the Revolude days, was to perform a similar network conveyance of entity properties, and logic, by sending game logic function indices to clients, telling them what functions to execute to bring an entity's state "up to speed". This made sense in my head, but in practice there was an issue between preventing functions from overlapping or overwriting eachother's changes.

The solution was to divide up the game logic into a server-logic and client-logic. Sometimes the two had the same pieces of code but used it in different ways. The server's job was to control the actual state of the game, and direct how clients should be simulating their end, which entities are where, and what functions they are executing.

It never worked out, fully. The Revolude build I still have is wrought with networking bugs. A poorly thought-out event networking system wasn't ensuring all events made it across, in order. Objects can be seen turning topsy-turvy, appearing and disappearing, or never existing (but leaving evidence that they did). It's a nightmare I am happy to never return to.

The networking in BITPHORIA is awesome, though. I am very happy with how the game is coming along.

BITPHORIA, in its current form.

Networking Game State and Events

Distributing the game state among multiple machines can be tricky business, depending on the complexity of your game state itself. Most games treat the game state as an array of entities, sometimes indexed using some sort of a scene graph or other hierarchical organization. At the end of the day, though, it's just a list.

To make designing the game more flexible (what with scripting, etc) usually these entities are represented as merely an entity index, and a chunk of data associated with that entity. This data 'blob' is then structured into meaningful values through the scripting system, which divides it up into entity state variables like origin, velocity, model index, etc.

On the networking side of things the engine discerns what state changes have been made to each entity that are relevant to the machine on the other end of the connection. Typically this consists of the server trying to figure out what information is relevant to each client about all the entities in the game, based on what the server knows the client has received so far, and what the client needs to know for its perspective of the game state.

This entails a complex system of backing up copies of the game state to generate a delta-compressed update unique to each client's situation, based on what the client has acknowledged having received as far as the game state is concerned. Fortunately, things are much simpler for the client, only being required to update the server concerning the player's activities in a much simpler scope.

The system I've been developing the past few days simplifies things, for the most part. It could also potentially simplify peer-to-peer type games, as an emergent property of its design. But for now I'm focusing on utilizing a server for simulating most of the game state, while allowing clients to submit their own simulation states. The goal, for simplicity's sake, is to utilize the same system on both ends for dealing with generating and sharing the game state.

This requires a system that works virtually like two servers talking to one-another. The only difference is that the client has a player manipulating the game state that it's responsible for relaying to the server, which in turn relays them to other clients.

One strategy for simplifying the description of entity types, while simultaneously simplifying serialization of entity states for network transmission, is to utilize entity type templates. Instead of having a scripting language that describes (in some sort of VM-executed code) setting the state of an entity, one variable at a time, why not just have static entities that are essentially copied to real entities based on specific triggers or events occuring?

Then, all that's needed to be transferred over the network when an entity's non-continuous state evolves (eg: entity flags, boolean states, model index, physics type, etc) is an ID value for the specific entity template that the entity should become. This eliminates the need to submit any specific entity properties outside of the continuous state (like origin, velocity, etc). The server could send these templates to clients, so that servers can be running customized games.

Utilizing an event-system, for things like player chat messages, player actions (jump, shoot, kill), entities changing their type/template, etc. allows the game state to be divided into discrete 'frames' that are not based on units of time, but units of actual state change. Each machine, upon generating an event, both executes the event and queues it to be distributed to remote systems.

Clients distribute their locally-generated events to the server (through a Quake3 style re-send until acknowledgement received method) which execute the events, and queue them to be further distributed to all clients. One global queue can be used on the server, in a circular buffer, and each client connected will continuously provide the server with acknowledgement of the last received event.

Right now I am implementing the events system which consists of an event execution mechanism (giant switch/case) and data parsing. Data comes in as a simple void* pointer, which is either forwarded from a network-parsed event, or from wrapper functions which allow engine code to invoke specific events with actual variable parameters by lumping them into data identical to what can be found in a network event.

Links of interest:

Quake3Networking - bookofhook
Game Networking | gafferongames.com

Multiplayer, Netcode, Etcetera

A big giant portion of my game project is multiplayer. Since the days of Quake3 I have found it hard to imagine working on much else than a game which can be played online among friends, enemies, and total strangers. I've also been an avid fan of Counter-Strike since the beta days, and the ultra competitive game play enthuses me to this very day.

Anybody who played QuakeWorld (TF, 2fort4 anyone?) surely remembers the requisite skill of being able to lead your targets in order to actually hit them. This was acceptable to us back then. It was considered a fact of life, and was simply unavoidable. It was also something that could be improved upon.

During what I like to think of as "The Counter-Strike Days" a programmer at Valve Software by the name of Yahn Bernier developed a new approach to networking multiplayer games in the Half-Life engine. It was a two-pronged strategy that consisted of client-side prediction and server-side lag compensation.

Client-side prediction is actually just a technique for disguising the fact that the server is doing all the authoritative simulation and that there is internet-induced latency in communicating the state of the simulation to the player clients which are interacting with it. It goes a long way toward making the game feel more responsive, without actually making the simulated interaction of game objects more responsive. Only superficial aesthetic aspects of the game state can be predicted, and are not necessarily an accurate depiction of the actual game state.

Server-side lag compensation is the closest thing to an actual solution which minimizes the effect network latency has on game play and game simulation responsiveness. More often than not, it works quite well. If you aim directly at an opponent, and fire, the hit will register almost as accurately (but not as immediately) as if the game were being run on the local machine. The server effectively re-winds the game state before performing physics and intersection tests.

On paper it sounds great. In practice, it isn't perfect at coping with latency jitter - or, latency variance (fluctuating ping) among players, which can throw off the compensation wildly. This is a fact that is not well-known. It is the heart of clips being emptied point-blank to no avail. It is also a good reason to avoid playing over WiFi connections, which are prone to random interference and ping spikes.

One more strategy, to smoothing out the game experience, is interpolation. In most modern games, this involves the client storing up multiple updates from the server, so that it can play them as smoothly and accurately as possible - by knowing the starting point and following point of an objects motion. In Counter-Strike Source, for example, this is fixed at 100 milliseconds. So that no matter what the actual network latency is, it is always compounded with an extra 100 millisecond delay for the sake of smoothing out object motion.

This image of Counter-Strike Source with sv_showhitboxes enabled displays the last known position of an entity as received from the server before the 100 millisecond interpolation delay that is used to smooth movement

At the end of the day, it seems like this is a lot of work just to make the game as responsive as possible without actually reducing or eliminating the actual latency between clients and servers. As long as the server is the only 'true' simulation of the game, these techniques or minimizing the effects of latency on game play are as good as it's going to get without upgrading the actual internet itself.

The reason that many games use the above techniques is because they are the best there is right now, and nobody believes that anything better can be done. Thus, nobody really explores different options. They see server-side simulation authority as imperative, because it's the only way to make the game secure from cheaters hacking and hackers cheating. My strategy is to do everything different than what is considered 'right' by many developers.

Firstly, I believe in letting the player's simulation occur client-side. I believe that game play can only be furthered by removing the lag component almost entirely from the player interacting with the simulation. This would be the equivalent of affording some game authority to the client-side prediction being used already.

Of course, the issue of hacking and abuse is the first thought to cross the minds of virtually anybody who understands the difference between client-side and server-side game logic. This is alleviated using a simple array of sanity checks for various 'vulnerable' circumstances. There is always a requisite state, or group of states, which allows a particular following state.

By closely examining the evolving state of a client's side of the game, on the server, it is easy to determine the likelihood that the game has been altered, hacked, etc. Each client will have a dynamic score indicating the probability of game tampering, and a threshold for this value which will invoke consequences (eg: kickban) once reached.

So, lets say that we are simulating a player's influence on the game locally on the client, and we have our hacking detection on the server, and everything is peachy. There's still the issue of latency when the game state is sent to clients. Clients will be interacting with an older state than other players are seeing. If two players are in a firefight, and one tries to take cover, he could end up dropping dead after reaching a safe spot simply because the other player could still shoot him where he was out in the open. This is not fun!

What if we could just predict where other players could be all the time? Or at least guess, so that we're closer to seeing where they actually are, as opposed to where they were? This is called extrapolation. Most games only rely on extrapolation when there is a lag spike, or dropped update packet, and the client's simulation runs out of updates to interpolate between to keep things moving smoothly.

I propose utilizing extrapolation exclusively to substitute for interpolation. When an update is received it should be used to project where the object currently is (based on latency) and where it will be by the estimated time the next update will be received. In the interim the engine can begin interpolating from wherever the object is at the moment (end of previous update extrapolation) to this predicted position.

This will not be nearly as accurate at showing the actual path as existing interpolation/delay methods, but since the server isn't the boss anymore this doesn't even matter! Infact, it will be extraordinarily inaccurate for higher pings.

At a round-trip time of 50 milliseconds (ping), an object will be 25 milliseconds ahead of the position received. If updates are 20hz then we can add another 50 milliseconds (plus or minus whether the update is early/late). So all we need to do is project the received origin out by 75 milliseconds and start interpolating the position to this new spot. A better approach would be to estimate an interpolation vector, scaled so the position will reach the projected position at 75 milliseconds, but continue moving the object with that same vector if the next update doesn't make it in time. When a new update comes, it will cause a smooth correction, and the accuracy no longer is significant as far as manipulating the state of the object locally (shooting and killing it).

Now objects will be more closely where they really are for clients and the server. At least, they won't be far behind like existing methods force them to be. This allows for firefights and interactions to be far more engaging, because it will drastically reduce the take-cover-and-die phenomenon.

The one side-effect of extrapolation is rubber-banding. Let's say we are viewing an object that is stationary, and our client has 200 ping (100ms latency). The object begins moving, and we receive the update 100ms later that is has moved a certain amount so far, and is moving at a certain velocity. Now, we take our stationary local copy of the object and have to accelerate it to the position we predict it will be by the next update, moving it faster than it actually was. Once we receive the second update about its motion, we should be pretty synced up, so long as it keeps moving in a straight line, but there will be a noticeable drop in its movement speed, back to its real speed, when that occurs.

This image depicts a player path (moving upwards) and what it looks like linearly interpolated from one predicted position to the next. If one were to also extrapolate velocity this could be smoothed further.

Inversely, when the object stops moving, we will still be simulating it moving beyond its stopping point, and our simulation will be forced to bounce it back to its resting position. Objects will effectively be racing around to be where they really are, always drifting around. This can be hidden by not allowing abrupt movement changes using low acceleration and friction values, but is not always fun because it lowers the game pace. Further smoothing of positions and velocities can be applied, but ultimately the smoother the result, the less real-time it will be. You will be trading smoothness for delay.

Links of interest:

Netcode Interview - with Yahn Bernier (Gamesurge.com, Aug. 2000)

Lag compensation - Valve Developer Community