I’ll be writing a lot more on this particular issue in the coming weeks as I’ve had more time to review internal reports.
For those of you just joining us let me bring you up to speed.
Our story so far…
Demigod, a high profile, AAA action-strategy-role playing game was released on April 14th. Well, it was supposed to be released on April 14th but actually got released at Gamestop stores early due to a…miscommunication between their corporate HQ and their brick and mortar outlets. This wouldn’t normally have been that big of a deal except this happened to be over Easter weekend and the release servers for the game weren’t yet up. Moreover, it also caused the “warez” version (i.e. there’s no copy protection on the game so the warez version meant someone bravely zipping it up and putting it up on a torrent) resulting in over 100,000 people using it – at once – before we were even back from Easter break. Suffice to say, it wasn’t a pretty picture.
For the first few days, we struggled to migrate people to a different set of servers that only legitimate users had access to. This took about 48 hours. But during this brief window, the game was basically unplayable because you couldn’t even get online – at all. We got whacked with some pretty negative first week reviews not surprisingly.
But our woes weren’t over yet. It became pretty clear that the NAT servers (the servers that negotiate the connection between player A and player 
couldn’t handle the # of users on the game resulting in a horrible online experience. As other people have pointed out, this sort of thing isn’t unique to Demigod (i.e. plenty of other games have had rough online launches) but the big difference is that those other games had a lot more single player content whereas Demigod relies more on its multiplayer experience than most games so it was a much bigger problem.
Like most games, Demigod uses a lot of licensed code. Demigod’s awesome 3D models are powered party by Granny 3D. The videos in the game are powered by Bink. The sound is powered by Fmod. And the network connectivity was powered by Raknet. These are all very good libraries and used by major publishers.
But Demigod’s network requirements are somewhat unusual and demanding. First, Demigod is peer-to-peer and not client server. Everyone connects to everyone. Second, the number of people playing is unusual. Yes, some people do play 4 on 4 games of Supreme Commander or Company of Heroes but typically they’re 1 on 1 or 2 on 2. The more connections, the more complex.
The result was that it was a nightmare to get games going online.
The problems
Demigod’s connectivity problems have basically boiled down to 1 bad design decision and 1 architectural limitation. The bad design decision was made in December of 2008 when it was decided to have the network library hand off sockets to Demigod proper. In most games, the connection between players is handled purely by one source. For instance, in Supreme Commander, GPGNet handled the entire connection.
So in Demigod, on launch day, Alice would host a game. Tom would be connected to Alice by the network library and then that socket would be handed to Demigod. Then, Alice and Tom would open a new socket to listen for more players to join in. As a result, a user might end up using a half dozen ports and sockets which some routers didn’t like and it just made things incredibly complex to connect people and put a lot of strain on the servers to manage all those connections.
Now, the architectural limitation came from the way the network library’s database handled things. We still don’t have a clear idea on why it was so limited but this was the overwhelming problem that only got resolved late last week. Here’s how it works:
Alice hosts a game. In doing so, she sends a message to the NAT server (as well as our servers). Tom wants to join so Tom clicks join and it tells the NAT server to begin connecting them. But, it turned out that a relatively small number of people online at once would quickly result in a huge delay in messages being sent back and forth. For instance, when Tom clicks join it sends a message to the server to tell it to start connecting Tom and Alice. But Alice might not get that message for 30 or 40 seconds. That means, for that entire time, Tom and Alice are “attempting to connect” but haven’t even really started because Alice hasn’t even gotten the message. As more people tried to join the game, that delay could get worse and worse. If someone left the game, it could take that amount of time for the server to realize that player had left (meanwhile it was trying to connect them).
Why did it take 3 weeks to fix?
It took us a solid week to realize that this was the problem because we assumed the issue was compatibility with routers or ISPs. We worked with people to set up their port forwarding, etc. Eventually we started looking at the time stamps but even then, when we saw the 30 to 40 second delays there was an assumption that that difference in time was only because the servers and our machines weren’t time synced. On one of the late nights here, I began insisting on having the machines all synced via one of the atomic clock services and then we became horrified at what we found: A 30 to 40 second delay between the time the server processed a message and actually sending a response. That time delay during peek hours could get over a minute.
Luckily, the developer of Raknet was here last week and was able to help us dig into the bowels of the code and we were able to find ways to mitigate this problem. Much of this is simply because Demigod is peer to peer and so it means there’s a lot of messages being sent back and forth. A lot of our work has been in reducing the # of messages needed to connect people and writing a new network layer that was designed for P2P specifically.
PLUS, we had to write something that network gurus can tell you is pretty damn cool: Proxy sockets. So that we wouldn’t have to hand off sockets, the Impulse team was recruited onto the project and they came up with a way to not have to hand over sockets to Demigod without having to change Demigod.
Now, with that much new code, it wasn’t perfect. Friday’s update resolved things for most people but it also created problems for some people who hadn’t problems before and it caused some grief for people who were trying to play Pantheon games. Plus, you still have people who simply have difficult Internet connectivity that still have to be resolved but now you’re down to like 5% of the base.
Today and later
Right now, we’re staging an update that addresses what was mentioned above. It’ll either get released this evening or tomorrow.
On Wednesday, the plan (not promise) is to release the update that supports Proxies. That update will also support some new command line switches:
/proxyonly (if you know you won’t be able to connect normally you can just have us host your traffic)
/noAI (will tell Pantheon and Skirmish to not allow AI players in even if you have to wait longer) – this may not make it into Wednesday but I’m hoping.
/localproxy (if you have a monster machine with a really really good network connection you can let people who can’t connect route through you which lowers lag – only do this if you have a very fast UP speed).
What else?
There are still some other things on our short list of things we need addressed and developer Gas Powered Games is working on them.
These include:
- When people quit (for whatever reason) it looks like they’re disconnecting (complete with annoying 30 second disconnect dialog)
- We need a concede option.
- We need favor points and favor items properly stored.
- We need to make sure that the stats recorded are more accurate (it’s been flakey).
- After using an ability, some Demigods stop moving until manually given another command.
GPG is working on those 4 items and more but I don’t have an ETA yet but I’m pushing for this week.
What about the future? An FAQ
Q: What is publisher Stardock planning to do for Demigod players?
A: The plan is to send out an email this week to users who purchased Demigod prior to today with a coupon for 50% off of Demigod that they can give to their friends. In addition, next week we will begin sending coupons for other things on Impulse to active Demigod players to help ensure a vibrant multiplayer community. We plan to keep doing that periodically.
Q: What about my pantheon stats?
A: As soon as we feel comfortable with the robustness of the stats, we are going to launch the second Epoch which will see the stats reset (the previous stats will be archived under the 1st Epoch).
As those who visit the Pantheon area know, we are taking this stuff pretty seriously. We are looking at this from the long-term.
Q: When will more Demigods be released?
A: 2 Demigods are in development. I will have a better idea later this week when I meet with GPG on that.
Q: What about modding?
A: We have discussed this issue with GPG and I can’t make promises on this but I can tell you that what we would like to allow is for people to mod AI players (LUA) and then submit them online and let them compete to see how they do. This would encourage good AI modding. We would also like to support mod support in Demigod proper where the Pantheon would support players downloading officially “blessed” mods and be able to play them in game. I don’t have any ETA on this yet.
Q: What about a demo?
A: We’d probably already have a demo out if we hadn’t been messing with this. But yes, there will be a demo. In all likelyhood, it will probably be a multiplayer only demo since we want to reassure people when the demo comes out that connectivity is totally nailed and bullet proof. This is different than our original plan which would have been a single player only demo with 2 demigods and 1 map. So we’re still thinking about how to do this in a way that has the most benefit to us and potential customers.
Q: How are Demigod’s sales?
A: They’re considerably better than Galactic Civilizations II’s but slightly less than Sins of a Solar Empire at the same time. However, there are a lot of variables. Besides the online MP debacle, you have an April release versus a February release, you have a MP-centric game versus a SP-centric game (The most players I’ve ever seen online with Sins is around 500 whereas right now, mid afternoon on a Monday there’s 2065 players playing online). You also have the review difference: Sins has a review average of 88. Demigod’s average is 78 (still pretty good). If you took out the first week reviews the average jumps to 84. Overall though, it looks like Demigod will hit 100,000 units sold before the official European release.
Q: What has Stardock learned from the Demigod release?
A: We’ve learned that you can’t treat networking as just another thing to plug in like you would a sound library or even a 3D engine. It’s a whole different animal. With Elemental (our next game), it’s single-player focused but its MP will be server based (and I mean we literally host the game). After Demigod, I don’t ever want to hear the words “socket” or “port” again. 
Q: What is your honest outlook for Demigod?
A: You know the expected marketing answer. But my view is, the difficult launch definitely hurt the game. There’s no way around it. Besides upsetting a lot of people anxious for a good multiplayer game, you also have the fact that those early negative reviews are going to linger. 1UP promised to re-review the game when this mess is straightened out and so I’ll be talking to them this week about that. But still, Gamespot (6.5) and IGN (7.5) are going to linger. That’s the breaks. One could argue we released a game that wasn’t done (we thought it was done) and that’s what you get.
Now, that said, I do think long-term the game is in good shape. First, we will be putting a lot of effort to build the online community. Scheduled games, strong team and clan support, pro tournaments, matchmaker filtering (let people filter out people they don’t want to play) are all coming sooner rather than later. Those things will help immensely.
A lot of Demigod’s long-term success depends, in my opinion, on whether we’re able to “perfect” Demigod’s online experience before a viable alternative shows up. Right now, Demigod has the luxury of being one of the few modern RTS games that is so well suited to playing online. It also lucked out in that Atari doesn’t release it internationally until later this month which means a huge influx of new players.
