Next Up: Client Performance Issues?

[Hurin]

Greetings All,

Once the datacenter/server-side issues are smoothed out, I hope that Turbine is finally able to begin addressing the client performance issues that have now become so apparent.

These issues are not new. For years, those with higher-powered gaming rigs have noted that parts of the game demonstrated inexplicably low rendering performance (note: Client rendering performance has nothing to do with server performance, or internet connectivity). From Galtrev onwards, we'd note frames-per-second (fps) well below other portions of the game, and well below what our hardware should be capable of rendering given the apparent complexity of the scenes/regions/areas being rendered.

This could be overlooked, however, because it was only in small swaths of the game. And because the dips in rendering performance weren't as egregious.

That is no longer the case, however. The performance of the client has gotten so bad that the game is no longer enjoyable for many. And improvements made seem to be "nibbling at the edges" rather than tackling the root cause.

In a nutshell, the LotRO client is not asking enough of our video hardware. There is a bottleneck in the client where it, in layman's terms, fails to deliver the game's data to the client computer's video hardware (gpu) fast enough to keep the computer's video hardware busy (regardless and independent of the computer's ability to actually deliver that data from the computer's hard drive or SSD to the client). This can be best illustrated in the following screenshot. . .



Please note in the screenshot that this is a Titan X, essentially the fastest video card available on the market (980ti being equivalent), powering a nine year-old game. And it can't even reach 50fps in this screenshot. Now note that the GPU (video card) is only having 29% of its processing power utilized.

Turbine. . . if you would, please, two questions:

1. Is it reasonable to expect anything substantial to be done about this?
2. Will Turbine make this a priority in the coming year after the datacenter fallout has settled?

I hope that something more can be done than simply "nibbling at the edges." Perhaps it's too much to ask at this late stage in the game's development for a wholesale optimization pass on the core of the game's engine. . . but that would be the dream scenario.

I have to say, between all the years of accumulated burn-out, disappointment with endgame content over the last several years, concerns about the business model, and these client performance issues, it is these client performance issues that are truly making it very hard for me to get excited about logging in to the game.

Turbine, you've provided me and so many others with nine years of adventure and joy in Middle-earth. Sometimes we don't appreciate that enough. So please accept this post/thread in the spirit in which it is offered. I realize resources are limited and not everything is possible. But I hope you'll do whatever you can to make substantial client performance improvements possible. In my humble, non-expert opinion, the further success and longevity of the game depends upon it.

Thanks!

--Hurin

P.S. No, I couldn't really afford the Titan X and knew better than to buy one (figured the 980ti was on the way). But my kids weren't going to enjoy college all that much anyways. . .

[Hurin]

One addendum. . .

Now, I'm sure some people might be saying: "Isn't 50fps plenty?"

If it were a "smooth" 50fps. . . perhaps.

But the performance dips and darts all over the place. In prior years, the fact that it would do so at an overall/average higher fps masked the issue. But now that we're dipping into the 40s, 30s, and even (quite often) 20s and teens, it's apparent, jarring, and immersion-breaking.

[guffyjohnson]

One addendum. . .

Now, I'm sure some people might be saying: "Isn't 50fps plenty?"

If it were a "smooth" 50fps. . . perhaps.

But the performance dips and darts all over the place. In prior years, the fact that it would do so at an overall/average higher fps masked the issue. But now that we're dipping into the 40s, 30s, and even (quite often) 20s and teens, it's apparent, jarring, and immersion-breaking.

I agree with this topic - this game has always ran poorly for me - it's playable but it's always had hitches/hiccups or whatever. With mounted combat this problem is a lot worse

[Amennator]

This is the problem I have. I have driven myself nuts over the years trying to figure out why Lotro ran so poorly on my machine. It didn't help that for years we were being told that it was our ISP or computer that was the problem. I knew it wasn't but I hate when anything technical doesn't function properly. A fix to this issue would motivate me to log in again. Doing one of the new epic battles and getting 15-25 fps is really frustrating.

[kaput79n]

Guys, I suspect that the game is now CPU limited...this means that a better graphics card is not going to help...you'll need a step up in CPU department instead...

[LagunaD2]

I'd love to see video performance improved.

But I'm gonna vote that they get the client to stop crashing to desktop or hanging several times per play session first. This problem also predates the server transfer disaster, and has gotten worse and worse over time. Gondor/Minas Tirith are frequently, but not always, involved.

It's entirely possible, of course, that the performance and stability issues are related.

[Hurin]

Guys, I suspect that the game is now CPU limited...this means that a better graphics card is not going to help...you'll need a step up in CPU department instead...

Nope. That this is not a case of CPU-limiting is easily demonstrated by disabling my CPU's overclock or even underclocking it. If the client is CPU-bound, doing so should result in a noticeable/substantial decrease in performance. But that is not the case.

--H

[LagunaD2]

Guys, I suspect that the game is now CPU limited...this means that a better graphics card is not going to help...you'll need a step up in CPU department instead...

This is trivially easy to check using Task Manager, and completely untrue.

On my system (i7, quad-core, 3.5 GHz), the game uses about 15% of the machine's CPU capacity when running (horribly) on fully maxed-out video settings.

The issue is almost certainly that the 10+ year-old code is not written to use multi-core machines efficiently. It is completely unaware of the 85% of the CPU resources that are sitting idle. That is a problem we are stuck with, though, because re-engineering 10-year old serial code into parallel is something not even Blizzard could pull off, and something nobody in their right mind would even consider trying.

[Daguest]

I'd love to see video performance improved.

But I'm gonna vote that they get the client to stop crashing to desktop or hanging several times per play session first. This problem also predates the server transfer disaster, and has gotten worse and worse over time. Gondor/Minas Tirith are frequently, but not always, involved.

It's entirely possible, of course, that the performance and stability issues are related.

No. Even though the game is poorly optimized for multicore, I'm never reaching 100% in a single CPU core, and I still have terrible fps drop in MT. GPU is less than 70% to (I don't have a titan ).

[Hurin]

This is trivially easy to check using Task Manager, and completely untrue.

On my system (i7, quad-core, 3.5 GHz), the game uses about 15% of the machine's CPU capacity when running (horribly) on fully maxed-out video settings.

The issue is almost certainly that the 10+ year-old code is not written to use multi-core machines efficiently. It is completely unaware of the 85% of the CPU resources that are sitting idle. That is a problem we are stuck with, though, because re-engineering 10-year old serial code into parallel is something not even Blizzard could pull off, and something nobody in their right mind would even consider trying.

Yet if the game is not cpu-bound (and we agree that it isn't), then it being single-threaded doesn't really matter.

The issue is that the game client is inefficient. The resources are there (whether CPU, or GPU), it just isn't using them. Even on a single-core (CPU) system.

I imagine a painter with one hundred hands. . . and he uses all one hundred to draw and then re-draw the sky, and then only goes on to finish painting the entire scene with one or two hands. If he'd reapportion the scene to all of his resources more equitably, he could draw a lot more paintings.

[Ogmios]

This is trivially easy to check using Task Manager, and completely untrue.

On my system (i7, quad-core, 3.5 GHz), the game uses about 15% of the machine's CPU capacity when running (horribly) on fully maxed-out video settings.

On one of the systems I support for family, they are playing on a very old Core Duo 8500. Max CPU usage is less than 35% - ever.

On another, the CPU is a 2600K . . . max CPU hovers around 17%.

And on the newest machine, running a still old Intel 4770K, the max CPU is around 12%. There's never been a strong correlation between lag spikes and CPU usage. If anything, every time lag hits, or a glitch happens, the CPU usage drops a bit.

My video cards also show low usage - from an old NV 660 to a NV 980ti.

All of these machines are running with excellent quality Intel or Sandisk or Samsung SSD's for both the OS and LOTRO . . . so disk I/O is not a problem either.

Edit: Finally, our internet connection(s) are almost always excellent - and I mean really excellent. Ping from our homes to our nearest cap-end is typically under 5ms. Ping to the new datacenter is also typically less than 70ms (and this has been to the new datacenter.) Speeds on our connections are very high, around 150Mbps down and 25Mbps up. I should note that - oddly, latency still shows as excellent for us even when the game laggs, rubberbands or crashes since the datacenter move. Which it's done a lot lately . . . sigh.

[Hurin]

Run some performance monitor traces and watch the CPUs. I suspect there's a (lack of) concurrency issue here, and that's something a performance monitor trace can call out, assuming you don't use the TOTAL bucket, and instead capture all the things, and filter it down accordingly.

That would also make for some interesting graphs, as it shows what the system sees. While you can do the same thing with the Windows Performance Toolkit, it's overkill for us as none of us have source nor debugging symbols from Turbine, so we can't pinpoint where the performance issues, and report the issues to a bug tracker once found. Same is true for other profiling tools (like those in the newer versions of Visual Studio).

That said, I would be interested to know if there was any performance (or stability) differences with a x64 Windows client compared to the x86 version that we've been using for forever. I know from experience with modded Kerbal that if you overload an x86 app and push it beyond the x86 memory limit, the app will crash (community workarounds included forcing all textures to a more memory efficient format, and shifting rendering to either OpenGL or DX11).

Settle down, professor.

[Snowlock]

Always heard it was the game engine and it could never be fixed to optimize sote rigs. Would be nice to be wrong

[Hurin]

Eh, I've used that tool enough that (for an inbox tool of all things) it's actually been a useful attention getter =P

That would also provide insight into the design, as there's certain performance patterns that would make a lot more sense with more data.

Is there any chance that you yourself would be willing to carry out the analysis you suggest and post the results here in a way that would be useful to Turbine? Or, if that's not possible, at least casually interesting to the layman?

--H

[Thorgrum]

Is there any chance that you yourself would be willing to carry out the analysis you suggest and post the results here in a way that would be useful to Turbine? Or, if that's not possible, at least casually interesting to the layman?

--H

I concur, I think more time expended trying to identify issues is a great use of the player bases time.

Who knows, you might be able to fix the product you are paying for.

[The_Jagannath]

With the way this community is, not a snowball's chance on Mt. Doom.

If you want to do it, here's how you configure Performance Monitor to do a trace:

blah blah blah... technical stuff you wouldn't understand blah blah blah... i'm important... everybody look at me... blah blah blah...

The rest gather information on where I suspect the problem lies, so it's a matter of right clicking the data collector, starting it up, then running LotRO, noting the time when issues occurr (prefarable at the second resolution, as a minute is an eternity in computers), then making bad stuff happen.

From there, it's just pattern analysis combined with knowing what is expected to be low/high, and looking for outliers.

So in other words you have no idea? = |

[Hurin]

I actually misread and didn't realize we were just talking about Windows Performance Monitor. My mistake.

Nevermind.

[BeachGolf]

From what I've observed in the progression of town updates in particular, Turbine is putting too many 3-D objects in the scenes. The early towns had a few fences, maybe a barrel. Now, towns have so many objects (obstacles, really) that it's now very difficult just to navigate your way down the streets. And that has nothing to do with the fact the roads aren't straight to begin with. There's a fine line between making the game realistic and going too far that it affects most of the players. Now that Minus Tirith is out, we all realize this has gone too far. Not only is the entire town full of objects that are apparently the root cause of the extremely low frame rates, but go in a building and it's just as bad (if not worse) in there! I would rather see far fewer objects and more playability in the game, although I do like the randomness that exist in some of the quests and hope to see more of it. Ideally, Turbine would make another pass through many of the towns to remove unnecessary objects and clean up the towns so we can travel at a relatively smooth frame-rate. (And I'm not totally disagreeing with the original post because obviously that's an issue too.) Thanks.

[Ogmios]

Is there any chance that you yourself would be willing to carry out the analysis you suggest and post the results here in a way that would be useful to Turbine? Or, if that's not possible, at least casually interesting to the layman?

--H

Without debug symbols from Turbine, the information might not be as useful as it should be . . . assuming they even have them.

[Lord.Funk]

Please let's turn this thread into just one more where people pee their turf in order to show who's got the biggest (knows it all) and bash everyone else. There sure aren't enough such threads already of various topics that all degrade and end up pretty much the same.

Keep it up !! Noone would want to surprise/dissepont anyone else now would they ???

[Specter666]

You all might want to read the following links. They 'll explain the problem and also how you can check it out on your own system.
Unfortunaly they are in german, I hope this ist no problem.

http://www.computerbase.de/forum/sho...247&p=15947165
http://www.computerbase.de/forum/sho...d.php?t=818979

[Hurin]

Ok genius, how are you going to assess performance of a black box on the system?

What's with the hostility?

I merely said I misread and thought you were describing the use of something other than Windows Performance Monitor.

As others have pointed out, there wouldn't be much point to doing any of this. Turbine is well-aware of the problem, it's apparent to them just as it is to us, and Turbine likely already has all the information we'd be able to provide.

This is a problem that affects everyone, across the entire playerbase, as you know. As such, there's very little need for us to "document" it beyond the screenshot in the initial thread. Turbine can reproduce and document the issue at will.

So, I gotta ask, what's with the hostility? You've been one of the louder voices decrying the client issues. Can you remain on good terms even with those who agree with you? Or does your credential-wagging and jargon-tastic posts require you to eventually turn on everyone who doesn't stroke your ego sufficiently?

--H

[Hurin]

They 'll explain the problem and also how you can check it out on your own system.

Anyone can "check it out" on their system just by playing the game.

There's no magic system setup or configuration that is suddenly going to make the Minas Tirith area render any faster. The limiting factor is the game client itself.

[Unthariel]

There was a lot threads posted by players frustrated by performance in newer areas. And revamped ones, like Bree. This town used to run at 120+FPS on high resolutions and now its barely playable, because performance is looking like this:

Its not only about average framerate, but about its consistency.

Just FPS drops in games, even heavy ones, are manageable by graphics settings, but in LOTRO they are jumping all around the place like author of this thread said, and from my experience as author of graphics settings guide(that isn't too accurate in terms of performance measurements, mostly just showing what every setting does), all of those performance problems are happening more or less regardless of your graphics settings. I am getting FPS drops and stuttering in Harwick even on Low settings, on 980 Ti. And it runs as bad for every resolution. Also, there were some people saying that LOTRO is not optimized for DirectX 10/11 and this is causing issues, but I checked it few times also and switching back do DX9 gives no improvement in places like Bree, Harwick, Forlaw and all of those that run so terribly. Lowering graphics settings will give you higher average framerate on paper, but stutters and FPS drops will be almost the same. Its just not worth it to lower visual quality of the game if it gives so little improvement. So in conclusion, those performance problems are completely unmanageable, you just cannot help it.

Here is how Harwick runs, just terrible:

Every city or even small village in Rohan is running terrible, just stutter festival. I cannot recall even a single game that runs so bad except LOTRO.

The problem is, like author of this thread said, that game client is not utilizing player's hardware properly. I never reached high GPU usage even on much cheaper card than 980 Ti, on R9 270X, and I was getting the same kind of "bottleneck" and performance problems while my card wasn't even used in half.

But the main reason for all of those problems are even happening is irresponsible development, without proper testing and understanding of current client limitations, because I cannot imagine any serious developer knowingly releasing content that cannot be rendered at comfortable framerate at any existing hardware at time of releasing and in future, because problem is not that this content is too demanding for hardware, it just crossed client limitations and requires more power than current client can draw from player's hardware. There was a lot of sings of this even before Rohan release, in revamped Bree, in Stangard, generally Great River is showing some significant signs of performance problems, in Tharakh Bazan in Moria, in Galtrev, some places in Gap of Rohan, they should have learned from those performance problems and release Rohan with respect for client limitations, but they didn't and they released highly unoptimized and barely playable content. And later, they again didn't learn anything from past mistakes and they extended this unoptimized content with all of its problems into Gondor, and now they will do the same with Mordor and any future content.

They could just be responsible, respect client limitations if they were unable to manage those limitations and just release optimized content, like the one from Eriador. There would be no performance problems, no need to talk about them and no need for client changes and updates that are desperately needed now, because any content from Rohan and further, with some places before Rohan like Bree, Stangard, Galtrev, are just not enjoyable, barely playable and are delivering more frustration than pleasure from playing.

One thing that I cannot understand i why they even crossed those limitations? Are places like for example Enedwaith, Evendim, Mirkwood, Moria or Lothlorien looking bad so they had to destroy performance in Rohan and Gondor to make them look better? I don't think so. They are looking good enough, for a lot of people this is the best part of the game. I bet everyone would prefer playable and optimized content kept in old graphical style known from Eriador regions than stuttering festival that Rohan and Gondor delivers now.

And of course, those problems are not addressed and no one ever got any answer from Turbine in any of those threads concerning performance problems. And they are releasing new content and extending all of those problems into futher regions, making it worse and worse with each update like they are completely unaware of those problems. What do we have to do to make them aware? Send a letter to their office or what? Because no one is reading forums apparently.

Seriously we should create some petition or something like this because if it keeps going like this there will be no Mordor, there will be only slideshow from Mordor, because we will get performance of 1 frame per minute there.

I cannot make such petition for sure, because I am not good at PR as you can see, I just tell what if think if I even decide to talk, and even though its always or almost always covered by indisputable facts, its in most cases seen as personal attack, even if it doesn't contain a single name or surname.

[Thorgrum]

Please let's turn this thread into just one more where people pee their turf in order to show who's got the biggest (knows it all) and bash everyone else.

I'm always up for that. Ready? Or are you a drive by white knight?