The Forums Are Now Closed!

The content will remain as a historical reference, thank you.

Flow control?

By on May 13, 2009 1:52:52 PM from JoeUser Forums JoeUser Forums

One of the things we’ve been struggling with, even since the beta, when it comes to match making is the seemingly exponential server requirements for match making.  That’s why when we’d release a beta things would be peachy and then they’d go to the public, it would fall apart.

Well, yesterday they finally found the issue deep deep in one of the licensed network libraries was something called flow control. This limits the number of packets that will get processed based on a number of criteria.  There’s good reason for flow control but because this code is a “black box” to us, until we could meet with the developer in person and go through his code, we had no idea that you’d end up with scenarios where a player might only be processing 3 or 4 packets per second.

Believe it or not, 3 or 4 packets per second isn’t necessarily a bad thing in a client/server game. You connect to the server and they send back the connection info. Maybe that connection info is 10k. So it takes a second or two to get that info. It’s not that big of a deal if it’s 1 vs. 1.  It’s not even that big of a deal if it’s 2 vs. 2.

But naturally, in Demigod, people want to to 3 vs. 3 or 4 vs. 4.  And if you do that, well, very bad things start to happen because it starts to back up. You can tell it’s messed up because your connection window will not seem to make sense and it doesn’t – because it’s incredibly outdated.

If you have to get info every few seconds from say 8 people in a lobby, you’re going to quickly get backed up.  Now, until we fix this, you can resolve a lot of this by keeping your games at 2 vs. 2 or maybe 3 vs. 3 if the people in the lobby have decent machines.  It’s fixed on our end but we have to merge the fixed library into our code which, if you’re a developer, you know how carefully that needs to be done, even with version control.

But this problem I just described has been there since the beginning (i.e. February).  The only reason connectivity has gotten better is because we’ve thrown a ridiculous number of servers (I think there’s like 8 servers now dedicated to just handling requests) plus the work we’ve been doing for the past 2 weeks has been improving things.

BTW, the only reason why the day 0 update worked better for some people than the most recent release is because the day 0 update used NAT and that’s a lot slower (so a lot fewer messages). The faster the system, the more it aggravates this.

I’m going to ask our technology architect to do a write up on this once this is fixed.  I am heading back to doing more testing scenarios.

I plan to also write up an article on how games are made so people can get a better idea of how this sort of thing can happen in a major commercial game in the first place.

+912 Karma | 42 Replies
May 13, 2009 10:09:07 PM from Demigod Forums Demigod Forums

And a light finaly appears at the end of this long dark tunnel.

May 13, 2009 10:44:33 PM from Demigod Forums Demigod Forums

Quoting Makerz,
Also Frogboy will this kill Thrusday's update? As in will the proxys not even matter?

I'm sure that this was not the end-all be-all reason for the issues with networking but I think that it is an important step. The proxy servers should still be implemented for users who are unable to direct connect and are still yet unable to NAT connect. Plus the modified peer to peer networking interface that they are going to implement with 1.0.1 will not change this fact for these people who are unable to connect to anyone. So it is still in their best interest to implement a proxy server system (although the proxy users are not going to be very popular). 

May 14, 2009 1:29:12 AM from Demigod Forums Demigod Forums

Quoting khemintiri,
Im sorry to say ... i dont appreciate this , not that your coming out to tell us , but i really dont feel like playing pantheon aI /drops matches or 2v2 , 3v3 custom games .

 

theres always a new problem coming up , nothing ever gets fixed , so many times have i read your update that says , its going to be fixed in 2-3 days , and now i have more issues then when i bought the game.

 

Im sorry frogboy , but my doubt is spilling up from my gut , problems might have been solved but now i cant play pantheon anymore, without playing with an unfair team every match, and i cant even dream of matches bigger then 3v3 ? , i thought this game was made for 5v5 and the other sized was a nice addition :/ .

 

 6.6 has not been released yet , you still have time , just make that time count man

 

Would you rather the 'normal' approach of spending 3 months to get around to doing that 'just a quick patch, be out soon' patch, or would you rather have them give you what they've got while honestly admitting on the forums whats going on?

Personally, I'd say this is better PR than a patch 3 months down the line, if then.

May 14, 2009 5:13:49 AM from Demigod Forums Demigod Forums

Quoting Ferazel,

Plus the modified peer to peer networking interface that they are going to implement with 1.0.1 will not change this fact for these people who are unable to connect to anyone.

The peer-proxy system has been turned off, according to one of Frogboy's posts today: Late night connection testing (see point 2). It seems the reasoning I gave for it possibly not working in my reply to Demigod: Tuesday 5/12/2009 was on the right track, at least.

 

May 14, 2009 8:10:00 AM from Demigod Forums Demigod Forums

Quoting Ron Lugge,

Quoting khemintiri, reply 12Im sorry to say ... i dont appreciate this , not that your coming out to tell us , but i really dont feel like playing pantheon aI /drops matches or 2v2 , 3v3 custom games .

 

theres always a new problem coming up , nothing ever gets fixed , so many times have i read your update that says , its going to be fixed in 2-3 days , and now i have more issues then when i bought the game.

 

Im sorry frogboy , but my doubt is spilling up from my gut , problems might have been solved but now i cant play pantheon anymore, without playing with an unfair team every match, and i cant even dream of matches bigger then 3v3 ? , i thought this game was made for 5v5 and the other sized was a nice addition :/ .

 

 6.6 has not been released yet , you still have time , just make that time count man

 

Would you rather the 'normal' approach of spending 3 months to get around to doing that 'just a quick patch, be out soon' patch, or would you rather have them give you what they've got while honestly admitting on the forums whats going on?

Personally, I'd say this is better PR than a patch 3 months down the line, if then.

 

no but as i say , theres always a new update every 3 days saying its going to be fixed in 3 days when its not , just saying ... that im starting to see this as more of a pr stunt to gain time then actually making progress .

 

and i would rather had a working game when i bought it tbh , not waiting at all , aoc is more or less fixed by now , but now the player base is gone .. dont want another aoc

May 14, 2009 9:50:52 AM from Demigod Forums Demigod Forums

Hey we appreciate the update Frogboy.  Looks like Raknet is getting replaced, which I think will be a good thing.  Personally if it was my game, I would never trust a 3rd party network library unless I had tested it extensively.

I wanted to recommend to you for testing network games in the future, you really need to get WAN simulators.  You can buy expensive commercial ones, or make your own with a Linux box and a few NICs.  Basically they allow you to turn up the latency and throttle down the bandwidth so you can simulate exactly what it will be like for people connecting to each other over the Internet.

For your QA team, go out and spend a few hundred bucks for parts and build a couple Linux boxes with 6 NICs each in them (3 dual port NICs).  Use 1 NIC to connect out to the Internet (or your test matchmaking server), and hang cheap Linksys/Dlink/Netgear routers off of the other 5 ports, then plug them into your QA gaming computers.  Then you can simulate real-world 5v5 games (1 team of 5 players on each of the two Linux boxes) and dial up the latency and dial down the bandwidth to your hearts content.  Your QA team can simulate port forwarding or no port forwarding on their Linksys/Dlink/Netgear routers, and since your machines connect out to the Internet (or your testing network) to the matchmaking server it simulates what a real world scenario is like.

Another benefit of this setup is that you can capture all of the network traffic through tcpdump on the Linux box to a file, and play back packet traces through Ethereal on your Windows desktop, so if a user gets a connection issue, all you have to do is capture his packets during the issue, and you can have a packet-by-packet timestamped replayable trace you can email the developer so he can see exactly what data the client and server are sending to each other.

PM me if you're interested in more info.  You have the luxury of a very easy to simulate testing environment.  Of course the matchmaking server has more than 10 players connected at a time, but for individual games, you know there are no more than 10 end-points, which can be entirely packet captured and traced for easier troubleshooting.  Client-Server games with thousands of players on a single server can be much more difficult to troubleshoot.

May 14, 2009 10:16:52 AM from Demigod Forums Demigod Forums

Client-Server games with thousands of players on a single server can be much more difficult to troubleshoot.

Sadly, Demigod is a P2P game. So Stardock servers don't see exchanged packets during the game.

In the case of servers used as proxy for some people, that is another thing.

May 14, 2009 10:34:38 AM from Demigod Forums Demigod Forums

Or use the all mighty internet. I think there will be posts about raknet and p2p somewhere

 

But its to late *g

 

The 22. may is the last chance to get thinngs working. If there still problems. We will see demigod go down the line ;(

 

Btw: I read somewhere that most "big" servers on mmos and so on use erlang or langfeld? Or something like this.

May 14, 2009 11:27:04 AM from Demigod Forums Demigod Forums

no but as i say , theres always a new update every 3 days saying its going to be fixed in 3 days when its not , just saying ... that im starting to see this as more of a pr stunt to gain time then actually making progress .

I believe someone has alreayd mentioned the equation...

(True Time Taken For Project) = (Estimated Time Taken for Project) * 3

They used a slightly different format, but that one should be easy enough to understand.  SD honestly believed they'd have it fixed quickly.  The fact that they didn't had more to do with the complexity of the problem then them being incompetent.  Don't forget, the big 3-monther companies probably do the same thing, they just don't tell everyone about the setbacks and take 3 months to get anything out.

May 14, 2009 11:39:24 AM from Demigod Forums Demigod Forums

Quoting Ron Lugge,

no but as i say , theres always a new update every 3 days saying its going to be fixed in 3 days when its not , just saying ... that im starting to see this as more of a pr stunt to gain time then actually making progress .

I believe someone has alreayd mentioned the equation...

(True Time Taken For Project) = (Estimated Time Taken for Project) * 3

They used a slightly different format, but that one should be easy enough to understand.  SD honestly believed they'd have it fixed quickly.  The fact that they didn't had more to do with the complexity of the problem then them being incompetent.  Don't forget, the big 3-monther companies probably do the same thing, they just don't tell everyone about the setbacks and take 3 months to get anything out.

 

 

this is my feeling about this i feel like theyre just buying time , my post was to stardock and frogboy , not to the community.

if you dont feel the same way thats up to you , im just feeling aoc all over again

May 14, 2009 12:03:29 PM from Demigod Forums Demigod Forums

Quoting khemintiri,

this is my feeling about this i feel like theyre just buying time , my post was to stardock and frogboy , not to the community.

if you dont feel the same way thats up to you , im just feeling aoc all over again

Yes, and even though you don't use punctuation (I've lasted this long but the grammar nazi in me can remain silent no longer!), making it unpleasant to try and read your posts,  you're entitled to your opinion.

That said, here's mine: AoC all over again? Demigod fans get daily updates from the CEO of the publisher. I don't know of any other game, especially when said game wasn't performing as advertised, where customers have got service like that. At best you get cryptic snippets of information from community managers.

May 14, 2009 12:45:36 PM from JoeUser Forums JoeUser Forums

I'm glad you come forward with this kind of info.

3 or 4 packets a second is miserable. That's a really weird setting, even for a client-server game.

To put that in perspective, most modern-era PC shooters use a network frame rate (for lack of a better term) of 20Hz. That is, your PC sends data about its state to the server 20 times a second, and receives data on the state of the world 20 times a second. If your actual graphics framerate is higher (let's say 60fps), then the location and animation state and so on of everything is interpolated. In fact, the whole prediction/interpolation stuff for figuring out the proper world state with unreliable network data has been a huge part of the evolution of FPS games and client/server game code over the last decade or so.

Of course, there's some variance to this, depending on the game. Some games are variable - the simulation and network rate of Team Fortress 2 and Counter-Strike Source, for example, is variable. Some servers actually run the networking at 100Hz, which of course puts a much bigger CPU strain and bandwidth strain on the server - but if it can keep up, you get very responsive gameplay.

You said 3-4 "packets" a second and honestly, I have no idea what's in a demigod "packet." In most client-shooter PC game, one of the "snapshots" of gameplay is typically what each client can potentially interact with or see. A "potential visible set" of gameplay. They do all kinds of tricks like updating the status of distant players less often, various compression techniques to keep data rates low, etc.

I'm curious what you're going to set this 3-4 packet per second variable at. I've noticed that quite a few games - even 2 on 2 games - have a responsiveness problem. They stutter a bit, your commands respond slowly, etc. It seems like a much, MUCH higher number would be required.

May 14, 2009 12:58:34 PM from Demigod Forums Demigod Forums

Quoting Extremist,



Quoting khemintiri,
reply 10

this is my feeling about this i feel like theyre just buying time , my post was to stardock and frogboy , not to the community.


if you dont feel the same way thats up to you , im just feeling aoc all over again



Yes, and even though you don't use punctuation (I've lasted this long but the grammar nazi in me can remain silent no longer!), making it unpleasant to try and read your posts,  you're entitled to your opinion.

That said, here's mine: AoC all over again? Demigod fans get daily updates from the CEO of the publisher. I don't know of any other game, especially when said game wasn't performing as advertised, where customers have got service like that. At best you get cryptic snippets of information from community managers.

 

yes and aoc had all the time in the world to fix their issues aswell , issues they said would get fixed soon .

And i dont care about your stupid grammar crap as you can probably read what im saying anyways.

 

or maybe your just a dyslektic in that case i apologise.

and im tired of having to defend myself for my opinions , as i said i dont F care about your opinions just let me express what i feel towards stardock without you grammar nazis and fanboys get up my case.

 

and people complain about dota community , you cant say anything thats not "I LOVE FROGBOY" without getting shot down on these forums

 

 

May 14, 2009 2:18:22 PM from Demigod Forums Demigod Forums

Quoting khemintiri,

and im tired of having to defend myself for my opinions , as i said i dont F care about your opinions just let me express what i feel towards stardock without you grammar nazis and fanboys get up my case.

Actually that post I responded to is the first post of yours I bothered to read because it was short enough for the inconsiderate grammar to not make my eyes bleed.

And since you don't "F care" about my opinion I certainly couldn't care about yours. But since you get to express yours doesn't that mean I get to express mine?

Respond if you like, but if you don't at least try to make your post easy to read (capitals in the correct place, full stops, and decent spacing) you can be assured I won't bother again. Not that you care much I assume.

May 14, 2009 3:04:59 PM from Demigod Forums Demigod Forums

Quoting Extremist,

Quoting khemintiri, reply 13
and im tired of having to defend myself for my opinions , as i said i dont F care about your opinions just let me express what i feel towards stardock without you grammar nazis and fanboys get up my case.


Actually that post I responded to is the first post of yours I bothered to read because it was short enough for the inconsiderate grammar to not make my eyes bleed.

And since you don't "F care" about my opinion I certainly couldn't care about yours. But since you get to express yours doesn't that mean I get to express mine?

Respond if you like, but if you don't at least try to make your post easy to read (capitals in the correct place, full stops, and decent spacing) you can be assured I won't bother again. Not that you care much I assume.

 

May 14, 2009 3:40:53 PM from Demigod Forums Demigod Forums

Quoting Peace Phoenix,

Client-Server games with thousands of players on a single server can be much more difficult to troubleshoot.
Sadly, Demigod is a P2P game. So Stardock servers don't see exchanged packets during the game.

That's why I'm proposing this setup to Frogboy as a way to allow his QA team to see all of the traffic going to/from each client during testing sessions.  I realize that Demigod is P2P, but troubleshooting connection issues is much easier when you have a network sniffer and you can see the traffic flow between each client.

May 14, 2009 4:20:10 PM from Demigod Forums Demigod Forums

well hopefully this fixes issues with 3v3 and up connectivity to others , best of luck SD.

Stardock Forums v1.0.0.0    #108435  walnut2   Server Load Time: 00:00:00.0000641   Page Render Time:

Stardock Magazine | Register | Online Privacy Policy | Terms of Use

Copyright © 2012 Stardock Entertainment and Gas Powered Games. Demigod is a trademark of Gas Powered Games. All rights reserved. All other trademarks and copyrights are the properties of their respective owners. Windows, the Windows Vista Start button and Xbox 360 are trademarks of the Microsoft group of companies, and 'Games for Windows' and the Windows Vista Start button logo are used under license from Microsoft. © 2012 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc.