Demigod: Understanding RTS multiplayer and why it sucks

By Frogboy on May 6, 2009 1:03:23 PM from

This post here gives a good understanding of where we’ve been.

But here I want to discuss the technical issues that have plagued the built in MP match-making.

I’m going to start this by sounding like a weasel but it is also the truth: The Demigod MP match-making is basically the same as nearly every other RTS out there. It’s no better or worse. The difference – the fatal difference – is the demographic of who is trying to play multiplayer is radically different and the number of people playing multiplayer is radically different. In short, Demigod’s built in MP, as it is today and has been since launch, is no better or worse than other RTSs on the market today.

Now, before some of you hit the flame key, let’s talk about why the experience of people using the built in MP match making has sucked so bad.

First person shooters vs. Real-Time Strategy Games

In games like Call of Duty or Counterstrike or Left4Dead, you are largely dealing with people picking a server from a list, typically based on ping time, and joining it. The only connection one needs to make is to that server and that server has typically been set up to be as easy to connect to as possible.

In a real-time strategy game, like Supreme Commander or Company of Heroes the connections are peer-to-peer. Each player has to connect to each other player.

GPGNet (Supreme Commander)

In Supreme Commander, Gas Powered Games developed GPGNet. The way it worked is that a person would create a game. Subsequent people would connect to the host and then enter the lobby. The game would then try to connect each of the players up and if they failed to connect to one another, they would get a yellow ping and the host would have to eject them.

However, the demographic of Supreme Commander is significantly different from the demographic of Demigod (as SupCom players here can probably tell you). Casual players of SupCom played mostly single player and the people who played SupCom online tended to be technically savvy and knew to port forward 6073 (that’s the port it used). And people who didn’t know what they were doing either quickly got ejected from games or learned to set up their routers.

Demigod

Demigod’s setup is similar to GPGNet with a couple critical differences. First, to get into the lobby you currently have to connect to everyone who is in the lobby before getting into the lobby. However, if you enter the lobby at the same time as someone else, you will get a yellow (or red 0) ping to them. If you can’t connect to them, someone will have to leave and try again or need to be booted out.

Super NAT

To improve connectivity we worked with RakNet to develop a kind of Super NAT Facilitator. The idea is that we would make it so that everyone could connect to each other and avoid the whole "people need to open their ports” thing entirely because we knew Demigod would be a multiplayer heavy game. Unfortunately, the system developed is resource intensive on the servers and when thousands of people showed up, the system started to fall apart.

The result is that people could not connect to people easily inside of the lobby because of the increasing lag (not ping lag but lag from the servers being overloaded) and users would just sit there waiting in frustration to get into a game.

During the first week, we basically lost out because of the well known warez users hitting validation and check for updates so we couldn’t even begin to dig into the issue of the NAT servers being overwhelmed until last week.

Super NAT fails

But we have encountered edge cases that have turned out to be catastrophic. Over the past year it seems, an increasing number of ISPs (we believe, we don’t have proof so I’m not naming names here) are trying to shape their bandwidth use and doing things that make peer-to-peer connections more problematic (bit torrent uses P2P).

The net effect is that some people can only literally connect to 1 or 2 people and are then blocked by their ISP. So what happens? Other people try to join a game and they cannot connect to that guy who is in the lobby because that guy can’t accept any more connections. So then the Super NAT goes into play to try to find a way which in turn takes more and more resources on the NAT. You get even a handful of those types of users and it gets to be a real problem.

Blocking Port 6000

Then we find out that some routers block incoming traffic to port 6000. If you very quickly get a “NAT connection failed” that is why. You will need to tell your router to knock it off – port forward UDP traffic from port 6000 to your IP (or just have it unblock that port). I only know of a few routers that do this (Some Dlinks for instance).

So why is this taking so long to fix for those people?

The reason we have to connect to players in the lobby before putting them into the lobby is because Demigod, not us, handle the in game connections. We hand off the connection (the socket) to Demigod once users are connected. So we’ve had to spend time developing something that will let us connect people who are in the lobby even which is some very specialized coding.

We have also had to work on setting up Proxy servers for people who just can’t connect and that’s been no trivial thing either.

The build we put up tomorrow is expected to let people get into the lobby as long as they can connect to the host and then connect people once they are in the lobby first by trying direct connect (that’s if you port forward 6000 to 6200 to your PC’s IP) and then if that doesn’t work to use NAT facilitation and if that doesn’t work Proxy servers. But it’s complex with a lot of moving parts being developed by a team that has been working non-stop for weeks.

Will everyone be happy?

No. Because there are people out there for whatever reason just can’t connect to other people on the Internet. There aren’t many of them but at least we can work with them and see what we can do without those users impacting everyone else.

So that’s where we’re at.

Personally, I think this kind of stuff should be built into the OS. As a game developer, I can’t even begin to express how frustrated I am that we have to do all this. If this were an Xbox game or a PS3 game, we wouldn’t have to do any of this. We’d simply hand the user off to the manufacturer’s already included super mega system and it would take care of it. I could rant for hours on (as you can imagine, I’m extremely tired) how ridiculous that our reputation, game reviews, and our technical competence are being brought down based on how quickly we can implement something that should already exist by the OS maker (maybe Games for Windows Live will eventually evolve into this I hope).

We expect to put up a new BETA of this tonight and if all goes well release it tomorrow afternoon. That’s the plan. It’s not a promise. Given what I’ve learned in the past 3 weeks, when it comes to Windows networking, router manufacturers, and ISPss never assume anything.

+912 Karma | 96 Replies

Quoting ElPopo,

reply 25

Games for Windows Live is just for the matchmaking part. (And hey, that one doesn't add 5 bots in a 3vs3 game...)

Correct, it just adds Trueskill 30+ players to a Trueskill 1-3 game. One sided massacres for everyone!

Quoting Frogboy,

reply 18

Honestly, as technology gets more advanced it is becoming less and less convenient to play games with your friends. I remember playing Goldeneye on splitscreen and Starcraft over the internet without any issues when I was 10-12 years old. But here we are, years later, we all need our own xboxes to play games together and playing games over the internet is a total pain in the ass.

I think you are on to something. When we developed Entrepreneur, an 8 player MP game back in 1997 it let people on OS/2 or Windows play each other. There was no such thing (for us anyway) as NAT negotiation or anything else. It just worked. I'd go on to Stardock.net and it just worked. I could get an 8 person game going no problem.

** NOTE this is general observations NOT a RANT**

no but to be fair, your box no long has a routable address (none rfc 1918), on it anymore, your not hanging directly off the internet, and your not riddled with security problems anymore

Firewalls, NAT, Idp systems all provide functions, that are needed in this day an age; stick a naked PC on the internet and watch what happens to it within SECONDS; hide NAT's in general are a great first line of defence, as they dont rely on any static mappings - with otherthings thrown on top its a good solution to peoples security woes that ensure some safety.

if it was up to me all forms of direct play would get eradicated eventually, every time you map a port to your box, you are placing trust in the application that listens on that port, could GPG tell you guys (both users and stardock), that it wasnt possible to remote code exicute due to some kind of bufferover? Its placing a LOT of trust in a applicaiton that you make internet facing that is essentially a GAME, games arnt security app's and by and large users systems shouldnt be placed ont eh internet with random Apps listening for ip conections.

I would still be interested in some technical docko on how the SNAT facilitator works, that would be superinteresting for someone like me, at least id understand its function and method better

for what its worth, whilst ive been a unsatisfied customer, i am super impressed with stardock as a company during this difficult time; its a companys reaction to difficult situations thats telling.

/Theo

Quoting ElPopo,

reply 25

AFAIK DoW2 is using a p2p system too, Games for Windows Live is just for the matchmaking part. (And hey, that one doesn't add 5 bots in a 3vs3 game...)

Nope, DoW2 uses GFWL for it's entire online system. That's why DoW2's online system works so well - Microsoft have spent a lot of money making sure it does (which also means a lot of certification required - see 1+ months of leadtime on patches).

You also shouldn't confuse their skill-tracking algorithms (aka "that trueskill bullshit") with their connectivity - both work, but the former is based on faulty assumptions (their TrueSkill formula was built for FPSes - they don't apply to team-based RTS games).

About that GPGnet thing - assume it has something to do with GPGnet being Quazal tech, and Quazal having been bought up by THQ quite a while ago.

quoting post

The thing that drives me nuts about this is that this stuff used to be integrated into the OS. It was called "networking". Unfortunately, over the last ten years or so, the Internet has been systematically broken -- or at least, made inaccessible to ordinary users other than as a glorified TV channel. Now no-one has a routable address (my ISP-provided modem doesn't even have a way to disable its internal NAT that I can find), packets that don't go to a "responsible" (i.e., big and rich) party get randomly dropped, connections get shut down if they appear to be using the network in ways not anticipated by the owner, etc. I think that's why Stardock is having so much trouble: they're having to reverse-engineer the sabotage of our network infrastructure, both deliberate and accidental, that's taken place over the last decade.

What ever happened to IPv6 anyway? A lot of this flakiness comes from NAT, which would be totally unnecessary if we had the much larger address space IPv6 was supposed to bring us. (that's assuming the network proprieters allocated addresses in anything like an equitable manner, of course; our current overlords would probably charge you for two connections if you wanted two addresses)

END RANT

These excuses are getting old. Warcraft 2 was peer to peer. That was over 10 years ago. There is just no excuse as to why this game is having so many problems. I regret funding this company.

That was over 10 years ago.

Thats the point; 10 years ago it was easier to do these things than it is today.

Did Company of Heroes had such severe mp problems? Or Dawn of War 2?

No, but, on the other hand, if you tried playing CoH without re-configuring your router and firewalls first, it didn't work. (Don't know about DoW2, never picked that one up.) Which is what Stardock and GPG were trying to get around - make it 'console simple.'

The problem here is that Stardock tried to push the envelope by trying something new, and ran into that old adage that "When you push the right hand upper corner of the envelope, sometimes that's where your postage gets cancelled."

So what's the option? Stardock and GPG could have just stuck to the 'old' system of doing things. And then, a different group of people would have wondered why they stuck with the 'old', 'broken' system.

This is why I stand in awe of coders who do PC/console game development; you create something that's a labour of love, a complicated piece of coding that you've sweated blood over, and you throw it out to the masses where people will second guess every decision you've made. Beyond that, you encounter people who will not only question your decision making process, but your knowledge, your capabilities, and even your lineage.

It's a wonder people write anything other than "Half-Life" clones.

JGH

In theory, IPv6 would negate much of these problems. But it won't. We will still use some sort of NAT for security reasons. Even if our ISP's gave us 5 free routable addresses, I certainly wouldn't want any of my boxes hanging their asses out there unprotected.

Anyway. This shouldn't be such a big deal. Those of us that played games in the "old days" and still play games now in the "new days" (past few years to present), have just continued to learn with the times and have NO difficulty palying games (p2p or otherwise), when the game isn't broken itself. It's no different then going from your vcr+normal TV, to, your dvd then blue ray and HD with surround sound. It's more complicated, but you build knowledge on top of knowledge. Configure shit, and make it work.

Some people just need to read a book ...

And those of us that know, will continue to help our buddys configure their shit too.

I feel confident the Stardock crew will work everything out, and in the end have a nice stable system that works. Yes, they were trying to make things "easier" for people, and deserve credit for that too. I hope they go get some sleep, and come back to this fresh. Too many impatient people around here.

Bottom line: It's just a game. It has some problems. It'll get fixed. Most people will be happy.

Stardock: Take your time, get it right.

Brad,

I really respect you in the industry. Your pioneering work on Sins of a Solar Empire has forged a lot of new ground. In any case, I have to take you to task just a little bit on your net code:

Demigod’s setup is similar to GPGNet with a couple critical differences. First, to get into the lobby you currently have to connect to everyone who is in the lobby before getting into the lobby. However, if you enter the lobby at the same time as someone else, you will get a yellow (or red 0) ping to them. If you can’t connect to them, someone will have to leave and try again or need to be booted out.

If you read that paragraph back, can't you see anything just a little bit wrong with the design? You can't enter the lobby until you have first connected to everyone else who is in the lobby? I can pretty much guarantee you that if you have a server with 1000 random people connected, I won't be able to connect to at least a few of them. Maybe they shut off their computer and the connection is still in an open state. Maybe they are behind a firewall. Maybe there computer is in the middle of nowhere and packets are being brought by carrier pigeon. Anyone with a little bit of networking experience knows that connections on the internet are unreliable at best.

The fact that your netcode relies on everyone on the internet being able to connect to everyone in order to evey see the lobby is just asking for problems.

I hope you can resolve these issues. I'm about to buy Demigod, but before I do, I want to know that your netcode works.

Let me propose a better design:

The lobby is just a list of people stored in a database table on a server. When someone connects, their nickname, info, and IP address get plugged into the table on the server. If someone disconnects, their entry gets purged from the database. When someone needs to login to the lobby, the only thing they need to connect to is the lobby server, who gives them the list (stored in the back-end database). That way you only need to make sure one connection is reliable: the one to the lobby server. You don't care about the other 1000 people that might be in the lobby, unless you want to start a game with them.

For a game like this, the network code really should look a lot more like Napster than what you have. Napster facilitated millions of users connecting to each other. It was easy because all they did was keep a list. I understand you want to help people with NAT that aren't sophisticated enough to forward ports, but really, sometimes you just need to rethink the design if it's this bad. And I wish you had a larger beta test because you probably would have found these issues.

Good luck, and please let me know when this is fixed so I can buy Demigod... I can't wait to play this game!

The security provided by a NAT is comparable to a firewalled router, its not exactly hard to come by and can't be considered a "serious" security measure.

NAT has a laundry list of problems with it and little to no positives (other than allowing IPv4 to continue functioning, which in itself is also a negative).

not to be a downer but I don't like the fact that you are sidestepping inproper testing by pointing at MS for their shortcommings in the OS. The reality is that the functionality doesn't exist in the OS and that it never has. Knowing that, proper testing of MP would have saved you all the lost good will.

Quoting Nubby,

reply 12

I think you missed his point. And also the point of several other posts.

Simply put, he's saying that networking should be done on the OS level, and it annoys him that it isn't (not trying to dodge his responsibility for doing it). On another level... you do realize that they did, in fact, stress test the servers? The problem in the code didn't show up in their synthetic stress tests, probably because they didn't know how to do the stressing 'right'. THe problem with hindsite: it may be 20/20, but it can only be used after the fact.

Keep up the good work. I hope there is a patch coming soon, which lets me play ingame online multiplayer.

I cant play at all with the ingame multiplayer, but it worls perfectly with gameranger.

can´t you code the netcode to act like gameranger?

just an idea...

Quoting Nubby,

reply 12

If this functionality existed in the OS, a lot of singleplayer games probably wouldn't be singleplayer.

My understanding is GFWL will now do this stuff. The tradeoff is your patches get a +1 month delay to them.

You see this in the case of DoW2. Their MP connection may work better. But the actual game bugs themselves take a lot longer to fix (and Relic isn't exactly known for fast, non-buggy, balanced patches to begin with)

http://community.dawnofwar2.com/viewtopic.php?f=45&t=12341

Tell all those people the MP connectivity works better and see what they say

Quoting illumin8,

reply 10

You can't enter the lobby until you have first connected to everyone else who is in the lobby? I can pretty much guarantee you that if you have a server with 1000 random people connected, I won't be able to connect to at least a few of them.

I don't think that's the "lobby", since there isn't a traditional RTS lobby. I think lobby in his explanation refers to the pre-game set-up screen. So that's a lot fewer than 1000 - yet still too many connections that have to be made it seems.

Quoting XSive_Death,

reply 16

http://community.dawnofwar2.com/viewtopic.php?f=45&t=12341

Tell all those people the MP connectivity works better and see what they say

That one is actually a provider problem. The aforementioned traffic shaping.

Did Company of Heroes had such severe mp problems? Or Dawn of War 2?

Company of Heroes - yes. They built a new matchmaking service based on tech by Quazal, and of course, it sucked for the first few months - as do all new matchmaking and connectivity systems. Incidentally, CoH is the reason that SupCom's GPGnet (also Quazal) actually worked so well - Relic Online was basically stresstesting Quazals tech, which GPGnet could then base their system on.

Dawn of War 2 - no. GFWL does a lot of things horribly wrong, but the one thing it does really well is connect stuff.

Quoting DatonKallandor,

reply 17

That one is actually a provider problem. The aforementioned traffic shaping.

And how anyone can tell if their ISP is "traffic shaping"? Besides DG not working...

Well, it would impact mostly P2P. RTS mostly, some download sharing software (Blizzard's Downloader comes to mind - that's one finnicky piece of P2P), etc. .

Many people reporting problems with the same thing, while having the same provider is a big clue.

I'm still confused as to why stardock chose to Jerry rig the connection rather than do what worked in starcraft and wc3. I mean cmon, starcraft is over 10 years old and there is no lag on there even without lan latency when compared to DG. Also, is there any way to get a sim speed reading for players BEFORE the game starts. Playing 5v5 is impossible because if 1 person has a bad computer the game is over instantly and there's no way of knowing.

For this game, a client/server netcode would have been preferred. Yes, i know Blizzard has battle.net, but i don't think Demigod players are that many to need such a complex infrastructure. It's clear that p2p doesn't work for this game, a game with 10 players in it. All other games that are p2p, are RTS, with nowhere near 10 players in a single game. I have a feeling that the netcode in DG would never be fixed...Too bad because it's such a nice game with so much potential.

The Forums Are Now Closed!

The content will remain as a historical reference, thank you.

Demigod: Understanding RTS multiplayer and why it sucks