The Forums Are Now Closed!

The content will remain as a historical reference, thank you.

Demigod: Inside the sausage factory

By on April 28, 2009 12:55:31 PM from JoeUser Forums JoeUser Forums

Frogboy

Join Date 03/2001
+1469

To say it has been a difficult 2 weeks would be an understatement.  I’ve been making PC games for about 15 years and never in that time have I encountered a situation quite like what we’ve had with Demigod.

As most of you know, Demigod is an awesome game. If I may be so bold, Demigod is one of the best games that has been released in the last couple of years.  And yet, here we are, 2 weeks after release and the included multiplayer matchmaking is still flakey.

I would put it like this (based on logs and talking to people)

  • 30% of users think the multiplayer matchmaking works flawlessly. You don’t hear much from them (obviously).
  • 60% of users are finding the multiplayer matchmaking to be flakey. It works but it’s a pain in the ass to get into a game sometimes.
  • 10% of users cannot be connected to other users period.

That last number will go up if the proxy servers aren’t put in place prior to the official European release as that last 10% is almost exclusively made up of people using ADSL in Europe.

So how did this happen?

image

Losing Messages

The single biggest problem has to do with lost messages.  In the beta, when we had only a couple thousand people using the system, it worked pretty well.  At first, it was pretty flakey but as we added more servers to handle the connections, the success rate went way up and by the end it was pretty perfect.  To be safe, we added extra servers to handle the expected extra players.

Then, one retailer broke the street date over the Easter break. This in itself wasn’t as big of a problem as one might think as the system could handle the legitimate players. But when the game inevitably got into warez channels, the pirates saturated our other servers (not the NAT server). Those servers handled things like validation and checking for updates. The validation kept pirates from being able to play in the Pantheon, skirmish or custom games but it also ate up a lot of our extra capacity.  The result was that for the first 3 days after the official release of the game, the online experience was totally hosed.  The early reviews reflected that unfortunately.

So for the first week, rather than focusing on the issue that would come to bite legitimate players for this past week, we were scrambling to create a mirror series of servers to isolate the pirates from the legitimate users.  That took us into mid last week.

By mid last week, it became apparent that the something was causing players to not get into the game.  We assumed it was simply bugs in Impulse Reactor since it was new and on Friday released a new version of it.  But it turned out there was nothing wrong with Impulse Reactor, the problem was coming from the NAT facilitator itself – it was sending messages but the game was never receiving them – or at least not before the time out period.

Adding more servers over the past few days has improved things bit by bit but not acceptably.

More Specific

When Bill and Rick decide to play Demigod together. The NAT facilitator connects them together, sends a message to Impulse Reactor which in turn hands over the socket to Demigod’s lobby.  We have verified that the NAT facilitator is sending the message but the message is never being received.  We know it has something to do with the load because when we try it out with a relatively small group of testers (~100) it works flawlessly.  It’s when it is put out there for tens of thousands of people that it falls apart.

Solving it

And that is where we are on Tuesday with regards to the connectivity. Figuring out why the message is sent from the NAT but not received by the game. There’s a lot of thoughts on this.  UDP (unlike TCP) isn’t guaranteed. You can learn more on this here.  Another thought is that the NAT queues up its messages and that by the time it sends it back to the game, it has timed out.    The team tells me they think they’ll have this mystery solved today.  If so, I’ll report here on this.  But this will put us up to 90%.

Proxy Servers

If I had a time machine and knew about the pain we would all be experiencing in the first couple of weeks after release, I would have simply had us set up a series of host servers around the world. When someone started a game, they would simply connect to one of those servers and there’s be no connection issues.  That’s what we have been planning to do with Elemental.  But I was of the same mind that many here have said “Hey, lots of games do this, how hard can it be?”

Well, it turns out, it is very hard.  Most vendors have used GameSpy over the years. Blizzard developed Battle.net over many years. Relic migrated to a system it developed with Quazal to develop. 

The only reason we should be able to develop a Proxy system quickly is because we already have a worldwide network infrastructure because of Impulse so the hardware is already there, the agreements with ISPs in various countries already in place.  Work on this system has been going on in parallel with the NAT issue.  What will happen is rather than getting a NAT failed message, the user will connect to one of our servers and have their traffic routed through that.  I don’t have a good ETA other than (very soon – days not weeks we expect).

On a personal note

We read the forums and we hang out on the chat and we even read comments elsewhere when we’re at home.  First, we appreciate all the support and understanding our community has had.  To those who are experiencing problems, we share your frustrations. Many of us at Stardock and Gas Powered Games have spent an immense amount of time making Demigod a great game.  For those of you who are parents, I suspect you can empathize with what those of us with small children at home who have gone weeks without seeing their dad (or mom). As a game designer, it has been quite maddening to see the launch of a great game marred by IT and network stuff.  And I can tell you, getting this network stuff nailed down is easily as complicated as dealing with all the various video cards and sound cards and other typical PC pain. 

The teams at Stardock and GPG do believe those issues will be solved soon. And we thank you for your continued patience.

 

WEDNESDAY UPDATE

I am currently sitting in the lab with the GPG, Stardock, and Raknet teams interacting.  GPG feels good about having taken care of the in-game disconnect issue. The "big" connectivity issue as described above seems to increasingly appear to be related to having to hand off the socket to the Demigod client. This creates a series of timing issues that gets exponentially worse the more people trying to connect multiplied by the # of users on the NAT server.   

Locked Post 80 Replies +7
Search this post
Subscription Options


Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:07:01 PM from Stardock Forums Stardock Forums

This is a truly excellent post. I wish all developers had such clear explanations of what went wrong, why the system works as it does, and what's happening to fix it.

Network problems are opaque at the best of times, and load-related issues are worse, so I wish GPG+Stardock the best of luck in finding a solution from what are often extremely vague user complaints.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:15:47 PM from Demigod Forums Demigod Forums

And i (like many others, i am sure) thank you for another great wall of text!

 

Things have surely gotten better, and i can just see how they are about to "just work" for many of us if all works as planned... does this mean we can start nagging GPG now?

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:17:22 PM from Demigod Forums Demigod Forums

Another good journal read. Thanks!

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:18:27 PM from Demigod Forums Demigod Forums

I love the pic, kinda gave me a Red vs. Blue flash back there..."Hey Simmons, you ever wonder why we're here?" Thank you for the update!

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:19:35 PM from Demigod Forums Demigod Forums

I feel sorry for you, getting such a bad start, and losing points from reviewers. It's sad to see some people still experience problems with the multiplayer portion (me too, sometimes), but your explanation... I wouldn't say makes up for it, but you're certainly showing everyone that you care.

Keep up the good - and hard - work!

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:22:06 PM from Demigod Forums Demigod Forums

One concern I had; if your ISP hates P2P can you manually opt to connect through the proxy server? So that you can bypass P2P all together. For example, if your ISP will shut down your connection or reset your connection if it detects P2P... you could conceivably connect to other players just fine... but then about 5 or 10 minutes into that game, BOOM. Disconnected. And the NAT wouldn't be able to save you by connecting you to a Proxy because you were apparently P2Ping just fine until your ISP shut you down.

Any thoughts?
-Aleckor

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:22:26 PM from Demigod Forums Demigod Forums

Interesting update.

I'm still not clear on exactly what the NAT facilitator is doing in a case where ports have been manually opened. In that case, why is a facilitator required at all? If the ports are all opened and forwarded to the right place, the other player should be able to connect to me directly.

Maybe it's a silly question, I don't know because I can't see the code. But if I put myself in the DMZ, why do I need to talk to the facilitator at all anymore? Looking in the connection log, it's still trying to do things like NAT punchthrough, which seems unnecessary because in the DMZ I'm wide open anyway. (Verified with port checker that UDP packets can get through on the Demigod/Impulse Reactor ports, and pretty well any other port I care to try.)

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:25:26 PM from Demigod Forums Demigod Forums

I have faith in you and your teams.  I'm happy with my DG experience to date, and am comfortable with what's happened since launch.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:31:34 PM from Demigod Forums Demigod Forums

Thank you for being honest about your game, many developpers will never acknowledge the problems their game have.

 

,

 

Opacitas

 

BTW: your game is awesome

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:33:46 PM from Demigod Forums Demigod Forums

thanks alot.. very informative post

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:52:16 PM from Demigod Forums Demigod Forums

We have your back, Brad. Thanks for the honesty and yes, many of us with kids understand exactly how you feel. Keep up the hard work, it will pay off. You live and you learn. As long as the bills are being paid, it's all good.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 1:53:12 PM from Demigod Forums Demigod Forums

When you have queues of packets building, it is very often better to have a very short queue and throw extra packets away (and resend them) than to keep queueing them up.  You may mess some connections up, but improve the overall throughput and prevent the really bad bottlenecks from happening. Every packet you have in your queue that is effectively invalid due to time-out makes the system deteriorate faster.

Anyway, thanks for working to improve this and for communicating with the fans, and good luck.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 2:04:08 PM from Demigod Forums Demigod Forums

Hello Frogboy, I know that you do not reply in your threads much (or your PMs ). But so far I did not get a useful respons to my problem. I also contacted Stardock support, but they say that this will be solved by the Proxies. However, as you pointed out several times, the Proxies will only help to improve the communication between the NAT facilitator and the Impulse Reactor. In your diagram, my problem is between the Player and the NAT facilitator.

So far it seems that the problem of being unable to connect to the NAT facilitator is not being addressed in any way .

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 2:11:20 PM from Demigod Forums Demigod Forums

What will we see exactly from GPG on the wednesday patch? A fix to favor points hopefully? How much of that diagram is going to be able to be done this week?

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 2:42:31 PM from Demigod Forums Demigod Forums

As long as you fix it and are working on it anything you say is fine by me

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:28:58 PM from Demigod Forums Demigod Forums

Thanks for keeping us informed and treating us like intelligent and mature gamers. Wish other companies did.

 

Jorune

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:33:46 PM from Demigod Forums Demigod Forums

Quoting Aleckor,
One concern I had; if your ISP hates P2P can you manually opt to connect through the proxy server? So that you can bypass P2P all together. For example, if your ISP will shut down your connection or reset your connection if it detects P2P... you could conceivably connect to other players just fine... but then about 5 or 10 minutes into that game, BOOM. Disconnected. And the NAT wouldn't be able to save you by connecting you to a Proxy because you were apparently P2Ping just fine until your ISP shut you down.

Any thoughts?
-Aleckor

I think including an option of using the proxy servers by default isn't a bad idea at all!

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:39:44 PM from Demigod Forums Demigod Forums

Trying to figure out is which is Forces of Light or Forces of Dark Red or Blue.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:43:00 PM from Demigod Forums Demigod Forums

The 30% number that gets flawless connetions into multiplayer game seems kinda high.  Especially from the explaination on the NAT errors you get, but I guess it doesn't matter.

Your explanation seems to say that people would get better connections when there are less people.  I kinda agree, as I seem to play better in the morning USA time when there are less people.  It also explains getting connected with friends is pretty random and just pure luck. 

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:45:25 PM from Stardock Forums Stardock Forums

^ it isnt

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:54:42 PM from Demigod Forums Demigod Forums

I'm among the 10% that can't connect to anyone, but I'm in North America, not the UK.

 

Anyway I just want to say that yes, not being able to play MP sucks, but its your constant feedback to us and keeping us all in the loop that's kept me from seeking a refund. I wish all game devs and publishers would do this.

If MMO's kept us as in the loop as this, I'd probably still be playing them.

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 3:56:03 PM from Demigod Forums Demigod Forums

Quoting Spooky,

Hello Frogboy, I know that you do not reply in your threads much (or your PMs ). But so far I did not get a useful respons to my problem. I also contacted Stardock support, but they say that this will be solved by the Proxies. However, as you pointed out several times, the Proxies will only help to improve the communication between the NAT facilitator and the Impulse Reactor. In your diagram, my problem is between the Player and the NAT facilitator.

So far it seems that the problem of being unable to connect to the NAT facilitator is not being addressed in any way .

Sorry to hear you're having such a rough time with the networking, but I think the reason they said the proxies will fix it is that, if you can't get to the NAT Facilitator for some reason, you'll instead get shunted over to the proxies. See the third paragraph up there under "Proxy Servers".

I have absolutely no idea if that will fix it, but I think that's what they expect.

 

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 4:02:36 PM from Demigod Forums Demigod Forums

thanks for the update Brad. you're a class act. 

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 4:10:43 PM from Demigod Forums Demigod Forums

UDP message delivery is not guaranteed, but it isn't flakey enough to generate the number of problems that I saw last night during my 1.5 hours of trying to get a full set of connections in 4 to 10 player games.

You have said that the NAT Facilitator is sending packets, but Impulse Reactor is not receving them.  Have you verified that they are being sent to the correct IP?  Also, do you know that the port mappings that it is sending out are always correct under load?  If I were debugging the code, I would parse the NAT Facilitator output to see if it is always spewing out the correct port mappings while under load.  To do that, I would write code that would send it fake information and then wait for the replies at a very fast rate, simulating a large scale player base.

This might show you that one of the following is wrong:

1. The IP:Port pairings that the NAT Facilitator is sending out are not always correct.

2. The NAT Facilitator is sometimes sending the mapping messages to the wrong IPs.

3. Under load, the NAT Facilitator is in fact not always sending the messages.

It is true that some people still do not have their ports open, but my friend and I cannot reliably connect to one another and I know that we both have our ports forwarded.  That tells me that either we aren't getting the UDP packet from the NAT Facilitator or that the packet contents are incorrect.

 

Reason for Karma (Optional)
Successfully updated karma reason!
April 28, 2009 4:16:16 PM from Demigod Forums Demigod Forums

Brad, you are,,, GODLIKE 

keep up the good work

Reason for Karma (Optional)
Successfully updated karma reason!
Stardock Forums v1.0.0.0    #108433  walnut3   Server Load Time: 00:00:00.0000031   Page Render Time: