Don't wanna sound whiny, but it's getting really annoying - it already got me killed one.. I've had a ping running on the side, so there's nothing wrong with my connection.

Any chance of getting whatever is causing it resolved soon?

edit: by "right now" i mean for the last 2 days or so, but it seems to be getting worse.

i added timing code to my mud client, so that i get roundtrip times
for every command i send to the mud. i think this is an easy way to test
if the MUD is underperforming in terms of user experience.

i also had ping running in parallel:

(ping, i'm playing from US)
576 packets transmitted, 576 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 39.107/43.370/85.268/4.596 ms

so network seems fine.

(roundtrip per command measurements in milliseconds)

(211 429 96 100 304 446 360 397 393 655 733 370 486 725 1946 264 263 132 491 797 415 651 32 342 32 118 571 652 972 422 287 1175 531 128 745 487 741 614 1733 235 119 201 100 202 176 202 260 616 477 48 643 527 127 586 47 318 511 462 529 665 988 846 965 71 164 504 47 605 344 115 664 471 420 109 225 196 770 241 859 893 61 796 213 218 82 204 352 341 220 125 144 134 145 379 280 201 126 256 197 263 106 194 235 285 179 193 136 1403 1231 3162 358 1170 1487 3113 157 217 242 302 198 173 154 1131 304 591 233 355)

min: 32, max: 3162, avg: 466

(small values like 32 should be disregarded as they are almost
always caused by the mud sending back a message that is not
a reply to the command i sent and started timing. for instance
i send a command to the mud, start my timer, new tick, mud sends
back updated status bar and i stop the timer. i could rework this to only time
commands that i can easily match to mud replies but the simple solution
i implemented here is good enough i reckon. the spikes are not affected by this.)

i get roundtrip times higher than they should be given the pulse times mentioned in this thread.
the big issue is the > 1s spikes which in this case happened when i was in a fight.
things would be better pk-wise if the lag was consistent and smooth but
going from 200ms to more than a second seemingly at random
can throw ones sense of timing off.

CF has been like this for me for more than a month now, and PK
is very frustrating as a result. Two years ago i used to play from the UK and
averaged 400ms pings but the experience was a lot better
as i experienced no spikes.

isildur mentioned this could be caused by the hypervisor if running on a vps, but i don't think so
since the ping times are fairly stable and there are no lag spikes there. this means that the OS kernel
(contains the code that replies to echo requests) is properly scheduled by the hypervisor.
An example of hypervisor throttling can be seen in amazon ec2 micro
instances:

64 bytes from 107.20.233.210: icmp_seq=48 ttl=46 time=44.176 ms
64 bytes from 107.20.233.210: icmp_seq=49 ttl=46 time=42.826 ms
64 bytes from 107.20.233.210: icmp_seq=50 ttl=46 time=46.124 ms
(0% cpu load so far, stable ping replies)

i run a process that consumes 100% cpu, the hypervisor starts giving
my virtualized instance less resources (via scheduling) and this is reflected in the ping reply times:

Request timeout for icmp_seq 51
64 bytes from 107.20.233.210: icmp_seq=51 ttl=46 time=1275.408 ms
64 bytes from 107.20.233.210: icmp_seq=52 ttl=46 time=577.873 ms
64 bytes from 107.20.233.210: icmp_seq=53 ttl=46 time=994.976 ms
64 bytes from 107.20.233.210: icmp_seq=54 ttl=46 time=825.837 ms
64 bytes from 107.20.233.210: icmp_seq=55 ttl=46 time=1325.752 ms

what could be the case here are non-uniform IO wait times or process scheduling
(another process stealing cpu, the DB if running on the same machine etc)

that the first step for me would be to verify that there are
spikes happening at the mud end. you can do that by timing
how much the mud spends to process every user request.
gettimeofday should be more than enough for that.

Zulg's indicated it's not the CF code per se that's the bottleneck. You've demonstrated it's not your connection either. It's something else running on the box. If the game runs in a virtual OS instance and the greedy process isn't running inside that instance then it may be impossible for the staff to figure out what's causing the contention.

Though, if that's the case then I'd chew out someone at the hosting company.

Abernyte commented on it. I've been getting it (though not REALLY badly), and now you have it.

Maybe out internet is straining under the quantity of riot-organising messages.

I am getting it too, that, and it's consistent with other characters/players complaints and actions in game and ooc.

I don't think it's the game; I get short bursts of lag while I'm editing files on the box.

As a heads up, a Denial of Service attack has symptoms rather similar to what CF is experiencing right now, and with as easy as it is for script kiddies to get a hold of software that would make doing such a snap for them it isn't completely unlikely that a certain someone who has recently shown his face again could manage running that.

a generic DOS against the server itself would be visible
from your end in ping roundtrip times for example.

there could be a DOS against the mud server process by someone
connecting to the mud and trying to overload it but i guess
this is easily detectable and it couldn't be happening constantly
for more than a month now.

Oh, if it has been going on more than a month it certainly isn't a denial of service, I thought it had just begun happening.

Toss that out the window then.

>Zulg's indicated it's not the CF code per se that's the
>bottleneck.

Last time I thought that, it did turn out to be the CF code (way to many damn SQL queries during update), and I fixed it. Tonight I added in some more performance tracking metrics around SQL to see if something else might be causing a similar problem, and will reboot to get that code in.

All the functions called during update are not showing anything out of the norm (there are little spikes outside of the goal, but they are typically less than 0.05% of the total number of calls to those functions, and max out around 0.6 seconds to complete).

So long, and thanks for all the fish!

how are statistics?
how are transaction logs?

hopefully you don't have them set to grow automatically by 10%....which is the default if its MSSQL

they would never show up as bad performance in queries but transaction log autogrowth lags out the cores especially if they are bigger than available RAM....

If you're still measuring process time and not wall-clock time then checking the duration of SQL queries (in the calling process) may not be useful. My guess is that they'll all appear to be near-instantaneous given an ODBC call doesn't use much CPU in the calling process.

Is the database server on the same box? If the game can tolerate a slight increase in query latency then you might try hosting the database server on a different machine, just to remove the possibility of contention between it and the MUD process. Though, obviously, two boxes costs more than one.

Might also look at running the MUD process at a higher CPU and I/O priority. "nice" and "ionice" respectively if it's hosted on Linux.

Hmm, 0.6 seconds is actually quite long. Considering that a CF pulse is what, 0.25 seconds - which is also what your brain synchronizes to when playing the game so you expect feedback every 0.25 seconds.

That being said, I haven't noticed any correlation with ticks or similar, and the lag spikes have felt longer than 0.6 seconds, but if you add the internet latency on that it would be about 0.8, which feels about right. Aka an eternity when involved in a PK situation.

I work for a Firewall company and I see this sort of problem with the Apache frontend and Postgres SQL database we use for reporting.

The problem is probably related to disk I/O on the box and could probably be solved with Solid State storage.

I can't see bandwidth being a problem but when you have 30-40 people on I am sure that's quite a significant amount of reads/writes occurring.

Just out of curiosity - What are the current specs of the server that CF resides on?

Is the server ours or on externally hosted hardware?

Id be up for paying for some SSD's just for giggles.

>The problem is probably related to disk I/O on the box and
>could probably be solved with Solid State storage.

Or host the database server on a different box. Or configure it such that its data lives on a different physical drive than all the data the MUD server accesses.