Well I said I was going to start going lighter on these journals since each one seems to give the poorly informed an opportunity to flame me or Stardock for whatever woes they have without realizing what’s really going on.

It just goes against my grain to not keep people up to date with what’s happening.  The only thing I will say is this: I’m just the messenger.  My job is a lot like herding cats.

So that everyone is on the same page of who is doing what:

Stardock

Stardock, who I work for, is the North American retail publisher of Demigod as well as the digital distributor of it via Impulse.  Our job is to take the game, put it in a box (or online) for people to purchase at the store. We also are responsible for the marketing and technical support for the game.

However, in the case of Demigod, we also agreed to provide a website to take posted data from the game and display it (rankings, favor points, etc.).

In addition, Stardock developed Impulse which, amongst other things, includes impulsereactor.dll which has what we call a Common Virtual Platform (CVP).  CVP is designed as an interface to a series of web services.  An application calls a CVP function to send XML to servers as well as receive XML from servers that the game can then display in-game.

Here are some examples:

A game might request a team game. The game developer would call something like CVP.RequestMatch() which would then look at players in the queue and send back an XML with the players and their IP addresses for the game to connect to.

A game may want to record how many kills or how much damage they did. It can use CVP to post this data to our servers.

Now, in the case of Demigod, some special code was added to handle connecting players together because Demigod does not have this. GPG used GPGNet in Supreme Commander but it was not available for Demigod.

Raknet

To take the place of GPGNet, we looked at several options including GameSpy Arcade, Games for Windows Live, and Raknet.  Since Demigod’s original ship date was February, we couldn’t use Games for Windows Live as it was too far along. The team ultimately decided to go with Raknet because it was used successfully with The Political Machine.  But The Political Machine is a client/server game.  Demigod is a peer-to-peer game just like Supreme Commander, Warcraft 3, Company of Heroes, and the Dawn of War series. 

So we worked with the developer of Raknet to use that library for Demigod.  I won’t go into the details here other than to say that obviously everything didn’t go as planned on that.  The basic issue was scalability.

For you web developers, the way I would describe it is imagine setting up a really elegant looking new website that runs great during your testing phase. Then you release it and when thousands of people are on it at once, it crashes and burns.  That’s basically what happened with Demigod. 

I still highly recommend Raknet. If someone makes a website using Ruby on Rails, for instance, they should know what the pluses and minuses of it.  Same here.  Ultimately, Stardock and Gas Powerered Games took the lead on this part of the game and so it was our responsibility.

With that, bear in mind, our anticipated technical involvement with Demigod was essentially that we would provide the website, a database, and finding players to connect to each other.  We treated the connectivity the same as you would any other “engine” you license (i.e. Unreal engine, Fmod, Bink, etc.).

Snowball 

In Russia, the publisher of Demigod is Snowball.  If you bought the game in Russia at the store, Snowball is the equivalent of Stardock there.

Atari

Everywhere else it’s Atari. Whether you bought it at a store in Australia or in the UK or on the continent of Europe, Atari is in charge there.

Gas Powered Games

And of course, the game itself is Chris Taylor’s Gas Powered Games which designed and developed the game. They make the demigods, the maps, the items, etc.  Demigod is their game (i.e. they own it). They are the Stephen King to Stardock’s Del Ray books.

I think most people would agree Demigod is an awesome game. Gas Powered Games is one of the top independent game developers in the industry. You’re talking about total pros here.  If it sounds like I’m a fan, it’s because I am.

My point is that when people have some problem whether it be single player, multiplayer, whatever I will do my best to direct the feedback to the right group.  My default position is that everything is my fault and therefore Stardock’s fault.  But that doesn’t mean we literally coded it. I’m just saying that the buck has to stop somewhere.

However, our ability to exact change is limited by how much of our own resources we can put in to help GPG or Raknet.  Customers of our partners Atari and Snowball benefit from these efforts but I’ll tell ya, I get pretty annoyed when some guy who bought the game in Europe at the store (therefore an Atari customer) is flaming Stardock for whatever problem.  It’s like flaming the fireman for not putting out the fire fast enough.

The Pantheon

The area of pain we’re dealing with this week is the Pantheon. 

Custom game connectivity works very well now and with the updates for this week, they’ll likely reach the point of diminishing returns (i.e. if you still have problems in custom games after this week’s updates, it’s on your end). 

But Pantheon sucks. So does skirmish.  Istari, one of the Stardock developers pulled from Elemental to help on Demigod along with others here have been going through pantheon nonsense all last week and today.

As some of you know (but unfortunately not most people) when things hit the fan with Demigod’s online experience, we brought in developers from Impulse and Elemental to work with GPG and Raknet to get things resolved more quickly. It was a tough call because you don’t necessarily want to have too many chefs in the kitchen but on the other hand, fixing things quickly and well is very expensive and it was either putting the burden on GPG or Raknet and at the end of the day, if I’m making the journals and our name is on the box I feel more comfortable if the people working on it are people responsible to me.  It’s a tough situation all around and everyone is working very hard under difficult circumstances.

Anyway, here’s a typical Pantheon experience:

1. User hits “Fight”
2. They wait awhile while the system tries to match them with people of similar skill.
3. They see 4 people on the connection dialog.
4. One of them disappears.
5. They end up in a 2 v 1 game.

Here’s another typical Pantheon experience:

1. User hits “Fight”
2. They wait awhile..
3. They see 4 people
4. One of them can’t connect back.

Sometimes you get both.

Here’s a report between Jeff and Istari on the subject:

This log finished off with the following:

PlayerName: Istari PlayerID: 11122 SystemAddress: xxx.204.71.133:64840, Connected to: 46966,52623 PendingConnections:

PlayerName: KutsuShita PlayerID: 46966 SystemAddress: xxx.209.39.38:6112, Connected to: 52623,11122 PendingConnections:

PlayerName: F3kay PlayerID: 52623 SystemAddress: xxx.147.232.59:6112, Connected to: 11122 PendingConnections: 46966

KutsuShita and F3kay DID connect. But part of the 2nd-round handshake failed. If you were to lookat F3kay's playerlog, my money is on that you'd see PLAYER_ID_PONG packets in the packet log, but no PLAYER_ID_PONG in the impulsereactor log. = The packets are coming in, but not getting processed.

This is the same problem I've been trying to track down with a couple users by working one-on-one. The problem appears to have to do with ordered packets not all arriving, causing all future messages to not process. There is a SIGNIFICANT chance that this is also the cause of the disconnects-from-faciliator, but I have no proof of that at this point in time.

Additional information about this problem includes that it most open happens to specific players, that other people show connected in their list (e.g. it says they're connected to everyone) but that they sit as "pending" in other people's list.

**My current theory is that their unreliable connection causes permanent packet-loss on some multi-part ordered messages, and once that happens all further messages fail to get processed as RakNet waits for the prior message.

What you see here is one of the problems with debugging a third-party library. We could either send this error to the developer or we can dig into the library’s source code ourselves and try to fix it.  We’ve been increasingly opting for the latter (mainly because for us, time is of the essence and we have a lot of experienced developers in this area who can do this pretty well).

But this gives you an idea of the kinds of things we’re dealing with.  What we find are not “bugs” but scaling issues. So I want to emphasize that we’re not going through other people’s code and saying “Oh, so and so has terrible code!” What we find are simply cases where Dev A didn’t expect Dev B to do a particular thing.

So this is probably more information than you wanted to know but it gives you an idea of the kinds of things we’re trying to help with here.