GoldenEye: Source Forums

  • November 27, 2024, 10:31:15 pm
  • Welcome, Guest
Advanced search  

News:

Pages: [1]   Go Down

Author Topic: [BUG] Server hangs periodically  (Read 6611 times)

0 Members and 1 Guest are viewing this topic.

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
[BUG] Server hangs periodically
« on: December 15, 2010, 04:21:02 pm »

So probably everyone has noticed this, that every once in a while (several times a day) a server will hang, or crash, or something. I've yet to see a MDMP file on either of our servers, probably because they restart automatically if they're unreachable for 15 minutes straight. It's getting to be a problem because last night, both hung at the same time, and maxed the CPUs, dropping our CSS pub from 500fps to 20fps. Is there any indication of what causes this or if it can be prevented?

It seems to happen on a variety of maps, and based on what I see in HLSW when they go down it is not tied to any obvious event (last log is not even always a kill). I see some gaps in one graph even from when players were not playing, meaning to me that probably it has hung sometimes either while empty or shortly after the first connect from empty.

We are running 1.0.1.0 on Windows/Xen using http://allnightersunite.com/gesource/GoldenEye_Source_v4_1_Server.zip
« Last Edit: December 15, 2010, 07:19:58 pm by killermonkey »
Logged
5 2 3 7 1 4 6 0

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #1 on: December 15, 2010, 07:18:47 pm »

If no mdmp is produced that almost surely points to a system level crash or a crash in valves hlds itself. A crash in the loaded binaries will produce a mdmp 95% of the time. Suggest you try reinstall of hlds.
Logged

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #2 on: December 15, 2010, 08:46:52 pm »

I don't think it's a crash per se, as a crash/exit would normally cause a quick restart, there would be a MDMP file, and the server maxes out the core its on after this happens (until the daemon manager forces it to restart). I'm going to install a server on my PC to see what happens but tentatively I don't think it's a fluke given that I see most of the other servers (through HLSW) go offline periodically, usually for a half hour or more at a time. I don't know if I've seen a Linux server go down or not.
Logged
5 2 3 7 1 4 6 0

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #3 on: December 16, 2010, 02:49:14 am »

I am having a hard time believing its GES thats causing the hangs though.... considering nothing drastic has changed on the server from v4.1 to Beta 4 besides some extra python callbacks and cleanup of code. Unless your logs or a mdmp can point to a single point of failure or a common thread I can only consider it to be in Valve's underlying dedicated server code base (which is notoriously buggy.

I wish you could run a profile on the server to show where each frame is spending it's time... there might be a way to do that, but I am not sure what command it would be.

Try adding the -debug switch to your server profile, I believe this forces a dump every time the server is killed unexpectedly.
« Last Edit: December 16, 2010, 02:56:31 am by killermonkey »
Logged

Wake[of]theBunT

  • Former [PR/SFX/Webmaster]
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 3,371
  • Reputation Power: 108
  • Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!Wake[of]theBunT is awe-inspiring!
  • Offline Offline
    • Goldeneye Source
Re: [BUG] Server hangs periodically
« Reply #4 on: December 16, 2010, 02:53:41 am »

The guy who does our officials also is stating there is some random things occurring he can't explain.

I'm betting it's source dedicated server for mods being lame VERY rarely. Let's just hope its not like before in beta 3 and they started breaking things all the time!...
Logged

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #5 on: December 16, 2010, 03:27:06 am »

Ok, so here is my plan. We need to narrow down when the hang/cpu load is occurring. This will help me identify the source of the problem. Also, if there is no general time/event that this is occurring on, we can safely say its HLDS acting foolish.

I need the following data:

- When in the match life cycle does it hang: Map load, First Round, Round Change, Match End?
- What gameplay is active?
- Was teamplay active?
- How many players were in the server?
- Do you have any mods/plugins running?
Logged

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #6 on: December 16, 2010, 08:35:29 am »

Samples of what I can see in log files, scanning by eye for consecutive non-mapchange log files. Unfortunately log files get truncated by any kind of crash or forced termination, so the last lines are pretty much random, and whatever was in the buffer is lost. I'll scan up in my HLSW next time it happens to see what can be seen as far as timing. In the past though I didn't notice anything unusual, just normal kills, connects, disconnects.

Deathmatch, FFA, round timer set to 420, round end time set to 12 (so 7:12 per round). Time limit is 28 minutes.

Source MetaMod 1.8.4, SourceMod 1.3.6, no other plugins, only using stock SM scripts. SourceMod logs errors for ge_bunker_classic only, something to do with entity outputs (almost certainly unrelated to this).

Started ge_facility at 01:01:53, log ends 01:14:52 (12:59+).
Started ge_facility_backzone at 03:25:19, log ends 03:40:03 (14:44+).
Started ge_silo at 20:58:03, log ends at 21:04:44 (6:41+).
Started ge_facility at 18:44:32, log ends at 19:03:46 (19:14+).
Started ge_caves at 21:08:22, log ends at 21:29:33 (21:11+).
Started ge_complex at 18:42:35, log ends at 18:48:51 (6:16+).
Started ge_archives at 15:58:17, log ends at 16:06:33 (8:16+).
Started ge_archives at 21:10:31, log ends at 21:17:56 (7:25+).
Started ge_casino at 02:03:14, log ends at 02:17:51 (14:37+).

If you figure there might be anywhere from zero to three minutes of data in the buffer when it goes down, it could be right after a round start. The only time I remember being on when this happened, it happened within ten seconds of a round starting. Then again, if you allow a fudge factor that's over 40% of your unit of measure, you can make almost anything fit a pattern.

I see several times Luchador being dropped for no Steam logon among the last few lines. Probably unrelated.

Already using -debug, no dump files.
-game gesource -ip 216.52.143.68 +maxplayers 16 -debug -console -nocrashdialog -norestart -tickrate 66

Edit:
Attached a log file appended with what appears in my HLSW. 14:09 from start to last logged event. Unfortunately I don't see things like round end and round start getting logged so there's no way to tell if it got past the end of the round or went down with seconds to spare.

Edit2:
Another log file with truncated part appended. 20:39 from start to last logged event, so unless there could be 45 seconds without a kill happening (unlikely since the server was fullish) anything directly related to round changing looks very unlikely. Also, there was almost five minutes of data lost in the buffer, meaning all the fancy subtraction I did above is most likely meaningless.

Edit3:
Another log file. 17:18 from start to last logged event. Roughly halfway through the third round.
« Last Edit: December 16, 2010, 07:45:37 pm by mookie »
Logged
5 2 3 7 1 4 6 0

Mark [lodle]

  • Retired Lead Developer
  • 007
  • *
  • Posts: 1,411
  • Reputation Power: 1
  • Mark [lodle] has no influence.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #7 on: December 16, 2010, 10:15:44 am »

It could be the garbage collector in python kicking in. Esp for game mods that create a lot of objects.


If you can,

run srcds using gdb and then when it hangs hit ctrl c which will pause it. At that point type: generate-core-file and send us the file so we can have a look at what its doing.

Edit:

If your using windows you can create a crash dump via task explorer (right click on srcds.exe and select create dump file)


Edit:

A unix tool you can use is http://htop.sourceforge.net/ to monitor srcds
« Last Edit: December 16, 2010, 01:12:07 pm by Mark [lodle] »
Logged

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #8 on: December 16, 2010, 09:27:31 pm »

OK I got one right now, do I do a minidump or a full dump?
Logged
5 2 3 7 1 4 6 0

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #9 on: December 16, 2010, 09:31:51 pm »

Full dump if you can as that'll include variable values
Logged

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #10 on: December 16, 2010, 09:48:44 pm »

Logged
5 2 3 7 1 4 6 0

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #11 on: December 16, 2010, 11:09:07 pm »

If this was a crash, it occurred in engine.dll while doing a bullet trace through triggers. The crashed originated from a pistol shot and it ultimately happened in the engine during a ray trace. It looks like the trace had no END (at least it looks undefined), so that might be a cause of the crash, I am still searching through the code.

Any subsequent crashes, please use mini dump just so I can verify location of crash.

It was also 700 seconds into the match, exactly.
« Last Edit: December 16, 2010, 11:14:26 pm by killermonkey »
Logged

mookie

  • Idiot
  • 00 Agent
  • ***
  • Posts: 252
  • Reputation Power: 0
  • mookie is looked down upon.
  • Offline Offline
Re: [BUG] Server hangs periodically
« Reply #12 on: December 17, 2010, 02:08:22 am »

Logged
5 2 3 7 1 4 6 0

killermonkey

  • GES Programmer
  • Retired Lead Developer
  • GE:S Fanatic
  • *
  • Posts: 5,473
  • Reputation Power: 346
  • killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!killermonkey is awe-inspiring!
  • Offline Offline
    • DroidMonkey Apps
Re: [BUG] Server hangs periodically
« Reply #13 on: December 17, 2010, 02:12:44 am »

Great news, this latest crash is in the same exact place.

Bad news, I am not sure of the root cause of the crash...
Logged
Pages: [1]   Go Up