Thread: Official: January 30,2011 Service Issues

At 10:20 PM EST, my.lotro.com and lorebook.lotro.com both seem to be working again, too.

Originally Posted by 37Mythias42

Same; Null Error as for logging into the game itself, and the other sites themselves are still not working; only the forums. (And even some of the forums links are busted, taking me to the 'scheduled maint' screen. Obviously this isn't a complaint, just trying to help isolate issues.

I agree with this as well, it was weird with me i hope this is unrelated, but as soon as i logged into the game, i was on about 3 mins i killed a few creatures and then i froze up i thought it was me, so i checked it again it still didn't work, i restarted my computer, and when i got back mysteriously all my drivers on my computer were mysteriously gone, i had to back up my pictures and personal stuff on my computer and had to format my computer by doing a system recovery, painstaking as it is i'm still trying to get everything back on my computer but found out as of 10:30 Pm est i still cant get the website to redownload and reinstall the game, but has this happened to anyone else or is it just me? i thought i may have been hacked at first because i had no problems till after the crash, then bam no drivers on my computer.

I was rolled back a little - no more than ten minutes.

May we please know what the problem was? I appreciate the updates on Twitter and Facebook saying you were working on it, but I'd still like to know what "it" was.

I was checking DNS records and thought I saw "turbineinc.com" expired on 27 Jan 2011, but when I checked later to verify it said 2012, so now I doubt that it was right, and I have no idea what turbineinc.com is used for anyway, so I don't know whether it was relevant even if I wasn't looking at it crosseyed. The other Turbine-registered sites looked like they had good long expiration dates.

But I'd prefer to hear something other than speculation. Thanks!

I just wanted to say Ive played since Beta and this is a rare, rare, rare case. I thank you all for a fine game and a fine record of service. Im sure you did all you could the best you could to get back up quickly and I thank you all! You are appreciated - now go home get some rest.

Congratulations Turbine on resolving the technical issues of the day.

I must mention however that I am disappointed in the lack of readily available information on the forums detailing what went wrong. I tried a few times to get on the game during the afternoon and then much later found that it is up and running again this evening. The scroll-box display on the log on screen does not even mention that there was an issue at all. I'm not expecting a play-by-play detailed explanation, but I would like some admission that there was in fact a problem and a brief explanation as to why game was down for hours on end on a Sunday afternoon. In to "good old days" of the old forum format, one could far more easily seek out and find answers to these types of questions: "What happened?" and "What went wrong?". (Once the forum was back up that is!) Now we must waste more time hunting deep within the forums to try to find a reasonable amount of information and answers to intelligent questions.

Twitter was indeed of some help! Now Turbine, complete your service to the customer by adding such event information to the front page (the very front page) of the "Forums" and/or the top of the "News" tabs of the Community(beta) sight. In addition and better yet, put some mention of the event in the scroll-box on the log on screen with a link to a more detailed explanation. This small task would provide your customer the required answers to their question as to why their game was not there for them come Sunday afternoon game time.

The current presentation (or lack thereof), of such information may lead one to conclude that Turbine is trying much harder to downplay the downtime event than to provide the customer the service of information and answers.

Just wanted to say thanks to all the people who had to work late in order to fix the issues.

Originally Posted by heruelladan

Congratulations Turbine on resolving the technical issues of the day.

I must mention however that I am disappointed in the lack of readily available information on the forums detailing what went wrong. but I would like some admission that there was in fact a problem

Like...um...this thread?
It would have been hard to find anything else on the forums today, what with the fact that they and the forums for all 3 Turbine games were sort of down, too. :-P

and a brief explanation as to why game was down for hours on end on a Sunday afternoon. In to "good old days" of the old forum format, one could far more easily seek out and find answers to these types of questions: "What happened?" and "What went wrong?".

To my knowledge Turbine has *never* given an explanation for downtime more detailed then "a server issue" or "a network issue." :-p

I wouldn't say Twitter was especially helpful, other than acknowledging that everything was down, that it was getting worked on, and (eventually) that there was a 2-hour estimate for completion. There was nothing about what the problem really was. The rest of it was crowd control: games including posting what server you play on and when you first logged onto LotRO. On Facebook it was much the same - generic "It's down and we're working on it" and an encouragement to post things relating to bacon.

As to what really went wrong, it was all just guesses from users about connectivity, DNS resolution, certificate expiration, and the inevitable trite hamster jokes.

Same going on here. "It's over, nothing to see here, move along."

Originally Posted by Tuco

As to what really went wrong, it was all just guesses from users about connectivity, DNS resolution, certificate expiration, and the inevitable trite hamster jokes.

What would the player do with that information anyway? It's not like it's actionable information.

Originally Posted by Roberto_the_First

What would the player do with that information anyway? It's not like it's actionable information.

How about fostering an atmosphere of communication and trust between the company and its customers? How about owning their mistakes (if they are theirs)? When I worked day-to-day computer support, I *always* tried to give the people I was responsible to and for the best explanation I could about downtime and other problems, and why a permanent fix was either in the works or impossible. Maybe Turbine and/or Warner Brothers (and their apologists) should consider doing the same...

Originally Posted by Roberto_the_First

What would the player do with that information anyway? It's not like it's actionable information.

It might help a knowledgeable customer form an informed opinion about the company they're doing business with. For example, if a pattern of repeated outages emerges with a related cause, the customer may wish to reconsider doing business with the company. Conversely, a knowledgeable customer might gain additional appreciation into the company's actions and become more loyal to the company.

That trade-off has to be balanced against the concerns of less knowledgeable customers, who might only be confused or intimidated by further details.

I'd much rather see an occasional note that says, "We're working on it" or "ETA: two hours" than notices about what went wrong, what they've diagnosed, what is fine, what isn't. In other words, work on the problem, don't hold my hand. If I don't see a notice that the game is back up, I assume it's they're still working on it. Not cackling over piles of money while we twitch at our keyboards. ;p

Thanks, Sapience... and all the others who plowed through what I can only imagine was a nightmare. You're well appreciated.

Originally Posted by Fredelas

It might help a knowledgeable customer form an informed opinion about the company they're doing business with. For example, if a pattern of repeated outages emerges with a related cause, the customer may wish to reconsider doing business with the company. Conversely, a knowledgeable customer might gain additional appreciation into the company's actions and become more loyal to the company.

For those of us who have been around a while, the fact that outages like this have been extremely rare is enough to indicate that Turbine has qualified technical staff who generally take good care of their servers. But I agree that more concrete information about outages would help their credibility, especially among newer players who haven't seen as much of their track record.

Initially, while the systems are down, an ETA for getting them restored is the most important information to disseminate, but then once they're restored, it's good business practice to provide a clearer explanation to customers about what happened.

Originally Posted by Arbalister

Pretty sure it would have to have been network related - a fibre cut, massive DDoS, or something similar.

Publishing a blank routing table to your data center?

....not that my company's ever done something like that before, or anything...

Originally Posted by Ailedra

Publishing a blank routing table to your data center?

....not that my company's ever done something like that before, or anything...

*Cough* Um...yeah...something like that would do it...but...*ahem* never happens does it? :-P

(We had something like that happen a week or so ago. :-p Rebooted a core router and it decided to load an *old* image...)

I will say, congrats to turbine for getting everything back up, even though it took you guys hours, for a data center repair, I am impressed.

However, on Nimrodel I am now experiencing MASSIVE lag, when I restart the client it goes away fora few minutes but returns very soon. Any answers?

Originally Posted by Elloric

How about fostering an atmosphere of communication and trust between the company and its customers? How about owning their mistakes (if they are theirs)? When I worked day-to-day computer support, I *always* tried to give the people I was responsible to and for the best explanation I could about downtime and other problems, and why a permanent fix was either in the works or impossible. Maybe Turbine and/or Warner Brothers (and their apologists) should consider doing the same...

I'm pretty sure this will be going on -- in house, where it belongs. I would bet there are going to be a lot of reports and explanations all the way up to the top of the ladder about what happened today, what the fix was, what should be done in the future, and how to prevent it.

As for apologies, Turbine is usually pretty good about that. But I don't really expect them to post late on a Sunday night after a nightmare of a day all about it. I think they deserve a good night's sleep and a nice danish in the morning before deciding what form their apology might take.

This must have been a living nightmare for everyone involved at Turbine. Of course, the goal of their systems and net operations teams is to have the services up and running and have contingency plans for rapidly restoring service in the case of system and network failures. So this outage, given its duration, must have been close to a "worst-case scenario" as outages go (though, the truly catastrophic scenario, I would think, would involve actual data loss and the need to roll back a substantial amount of time). There are probably a lot of people at Turbine who are exhausted right now and they're probably still sweating things and dreading the meetings that will come from this. But hopefully this outage was the result of a truly bizarre confluence of very unlikely events that overwhelmed the best laid plans of a well-intentioned, conscientious team . . . and not the result of any egregious flaw in design, planning, or implementation.

Yes, hindsight is 20/20 and some suit somewhere will probably --with the benefit of that hindsight-- ask why this outage was even possible and wasn't avoided (or recovered from more quickly) in the first place. Hopefully it can just be (honestly!) blamed on a vendor (heh! j/k). But if not, let's hope the "suit" in question is bright enough to grasp all the nuances at play and is a bit more realistic about the basic reality that "stuff happens" despite everyone's best efforts.

I'm not sure why so many folks here and elsewhere across the community are demanding irrelevant technical details about the outage as though they are entitled to such information or could do anything with it other than satisfy their own curiosity. Our curiosity about things doesn't entitle us to information. Nor does our status as a "customer." Turbine got word out about the outage as best they could given that their web services were down. I think the vast majority of people are satisfied just knowing that folks were aware of the issue and dilgently working on the solution(s). If I were at Turbine, I certainly wouldn't be in any hurry to provide clearly combative, antagonistic folks who are a bit "perspective-challenged" with information they don't actually need. . . just so they can continue to give us a hard time over what must already have been a nightmarish event. As customers, we are entitled to know that there is a known problem and that a resolution is underway. And that's pretty much it. They're not required to give out any more information. Especially since such information probably wouldn't be properly understood anyways and/or would be willfully misrepresented/oversimplified by folks with chips on their shoulders and/or bones to pick already. This ever-adversarial attitude towards a game company that has provided years of low-cost entertainment has always struck me as incredibly bizarre.

Anyways. . . I think the vast majority of folks who understand the complexities at play in a service like an MMORPG have nothing but sympathy and feel only appreciation for those who spend their nights and weekends away from family and friends so that they can do battle with ever-encroaching entropy on our behalf. Thank you!

H

Late Edit: I do not mean to imply that everyone asking for more information is doing so out of malice and/or for nefarious purposes. Clearly, some people are just curious and politely asking for more information (though I still don't think it's incumbent upon Turbine to provide it). But, then there are those demanding additional information. . . as though it is owed to them personally. . . and they seem pretty prepared to parse whatever information they receive as uncharitably and combatively as possible.

Originally Posted by Fredelas

That trade-off has to be balanced against the concerns of less knowledgeable customers, who might only be confused or intimidated by further details.

Sorry to anyone offended by this, but that "trade-off" argues very strongly toward a policy of sticking to results, and avoiding any details. Not just for Turbine, but for most anyone who isn't selling primarily to alpha geeks. It's certainly what the technology company I work for does, and what every other tech company I've worked for has done for the past 27 years.

Khafar

pass my thanks along to the people that came in on a sunday on short notice to deal with this.

Originally Posted by Hurin

This must have been a living nightmare for everyone involved at Turbine. (Hurin said a bunch of other stuff, too)

This.

For those of you who've never been part of one of these little parties, it goes a little something like this:

"What the ****?"

"Uhm, dude, everything's down"

"That's not funny"

"No, seriously. Call everybody."

"Oh, that. That server bounces all the time"

"No, right now, get off the forums, call EVERYBODY"

~20 minutes later, with God and country on the phone~

*frantically searching through every change made to the platform in the last couple of days*

"No, it doesn't seem to be related to change. No, that doesn't make any sense. Ok, yes, it COULD have had something to do with it, but it still doesn't make any sense. Ok, how long to back it out? That's funny. Really? Ok, I'll let everybody know"

~20 minutes later~

"How's progress?"

~10 minutes later~

"Yes sir, I understand customers are out of the water. I'm really sorry about that, but yelling at me won't make this process come up any faster."

~5 minutes later~

"Ok, we're still seeing it, and the forums are blowing up. That wasn't it. Maybe restart this process, see if that clears it? Yes sir, I know we should have had this back up half an hour ago. We tried to tell you this change wasn't related, but you insisted we back it out, just in case."

~20 minutes later, after applications have restarted, all connectors have bound, and people try to validate~

"****. Ok, what about..."

"Sir, that vendor's busy getting people into the office, too.....it's Sunday for them too, you see. Yes, I'll call them again, but they said half an hour 5 minutes ago"

~2 hours later~

"I REALLY need a smoke, and I have to pee"

"You can wait....this is big"

~2 hours later~

"You know, I was off an hour ago. Where are we on this?"

"I don't' want to hear it...today was my day off. We were going to take the kids to a movie. My wife is going to kill me"

~2 hours later~

"I still really need a smoke. Just sayin'"


Everybody complaining about lack of communication, or that the issue shouldn't even happened in the first place....shut up. Seriously. Systems break. Things happen. When you're trying to put out the fire, giving forum junkies an accurate update isn't anywhere near as simple as you think it should be. I don't know if you realize how much money the people involved in something like today make, but I'm here to tell you, it isn't as much as you think, and even if it were, it still isn't enough to cover the running gunfight nightmare something like this ends up being.

Folks at Turbine, I'm sorry your Sunday sucked.

Another word of thanks to the folks who dug in and fixed this. We appreciate it.

Originally Posted by Ailedra

Publishing a blank routing table to your data center?

....not that my company's ever done something like that before, or anything...

Originally Posted by Arbalister

*Cough* Um...yeah...something like that would do it...but...*ahem* never happens does it? :-P

(We had something like that happen a week or so ago. :-p Rebooted a core router and it decided to load an *old* image...)

At least it wasn't my (former) boss dropping the public IP address of the router he was logged into (via that IP)... After he panicked for five minutes, he called me and I fixed it in two. (This is why I always keep a backup copy of the serial IP address list of routers for myself.)

EDIT: Wow, way to spazz out on the original reason for reading this, Underhill... Thanks to all the Turbine folks who sweated and strained to fix this as quickly as they could! I didn't get to play today, but that's alright... it puts me that much closer to finishing Mass Effect 2.

Thought this was a mylotro fix, but it still isn't working, it has not updated for ages.

Heck of a thing to have to happen on a Sunday of all days - glad it's back, curious as heck as to what it was, realize will likely never know.

Originally Posted by Niwashi

For those of us who have been around a while, the fact that outages like this have been extremely rare is enough to indicate that Turbine has qualified technical staff who generally take good care of their servers. But I agree that more concrete information about outages would help their credibility, especially among newer players who haven't seen as much of their track record.

Initially, while the systems are down, an ETA for getting them restored is the most important information to disseminate, but then once they're restored, it's good business practice to provide a clearer explanation to customers about what happened.

While I don't disagree with you the biggest [problem when giving an ETA is that people start ranting if it goes over.

I've had large computer installations go down and been working through the night to fix the problem. Often for quite a while we're not sure what exactly the problem is, when asked to give a rough ETA, we simply pick a wild figure out of the air as we do not know. It might take 30 mins, it might take 10 hours, we know more when we work out exactly what's caused the fault.

While LOTRO forums has more intelligent and mature members than a lot of other MMO's you only have to read some of the responses on their facebook thread with regards to this issue to see how much more abuse they would have got if they had given a ETA and gone over.

If they did give a reason, you would then get reams of people self diagnosing what should have been in place, saying xyz should be sacked, saying such systems should/shouldn't be in place etc, for every one reasoned response from someone that knows what they are talking about, you would get 10 from people that don't, the thread would end up in a complete mess, probably ending up locked and while I am interested to know what went wrong, to be honest it's better if they just say the minimal possible.

I would have more of a problem is this was a regular occurrence, but I've been playing for 4 months on the US servers and this is the first problem I've encountered.

Thank you very much guys for your hard work to get this resolved.

I am 3rd level 3rd shift Blackberry and Lotus Notes support for a very large international health insurance company. I work alone due to the hours I support, and much like the techs that came into their office yesterday, if an outage like this came during my shift, I can promise you that others would have to come into the office to help resolve the problem as soon as possible, so to the guy saying people didn't sacrifice anything, you're an idiot.

For those requesting further information about what specifically went on. First off it isn't part of our subscription, that is if you even pay for one, to get up to the minute status updates of detailed issues. The fact that they resolved the issue and kept the angry mob of users at bay via twitter and facebook is commendable. They used whatever means they had necessary to alert the player base of the problems and possible resolution times. More importantly if they actually gave specific beyond "server problems" or "network issues" I wouldn't doubt 95% of the player base became more confused and didn't understand the technical jargon, and that is even taking into account that most of the players of these games in question are more technical, nerdier and older than your average computer user. That being said I would understand what they told us, but honestly I don't really care, the servers are back up and running now, knowing what is already fixed isn't going to solve anything, it wont cure cancer or create world peace, the answer is already solved for the day.

So again Turbine, thank you for your outstanding work, if you worked on your day off to resolve this I hope you get thanked and get some great overtime.

The cat was chasing the mouse and ran into the power connection knocking it loose. The mouse was fired and the cat was reprimanded for the problem created.

I just gotta say that, compared to another MMORPG game I used to pay for, the general attitude of the end users here is definitely refreshing. At the other game, if the system went down for even an hour there would be forum posts and tweets and blogs about what ****** service their paying for, and how they deserve a refund for all the time lost from their daily grinding of some end boss for phat l00t. Over here, when a good chunk of the game/service is free, most of the posts I've been seeing are notes of thanks and condolences for the Turbine/LOTRO staff that had to work on a Sunday to get this problem fixed.

So kudos to y'all for being a lot more understanding to the people who make this game work, and big thanks to Sapience and everyone at Turbine who came in, worked on the problem (did they ever find out who let the (mouse loose?), fixed it and tried to salvage the end of their week-end.

Thanks
(I'm going back to my corner of the forums and lurking again now)

I m glad everything is back up and running.

Originally Posted by Nitemayne

there would be forum posts and tweets and blogs about what ****** service their paying for, and how they deserve a refund for all the time lost from their daily grinding of some end boss for phat l00t.

A long time ago, in another galaxy, I was a GM for a game like this. Used to get those calls *all* the time...to the point where I started telling people that if they want their 25 cent refund, I'd pay it personally. All they had to do was mail me a stamped, self-addressed envelope. :-P

Never got one...

Originally Posted by Edlenn

Another word of thanks to the folks who dug in and fixed this. We appreciate it.

Hear, hear!

Originally Posted by Ojustaboo

If they did give a reason, you would then get reams of people self diagnosing what should have been in place, saying xyz should be sacked, saying such systems should/shouldn't be in place etc, for every one reasoned response from someone that knows what they are talking about, you would get 10 from people that don't, the thread would end up in a complete mess, probably ending up locked and while I am interested to know what went wrong, to be honest it's better if they just say the minimal possible.

Beyond that, in a time of denial of service attacks, do we really want them to identify some single point of failure that was responsible for an outage, or say a particular server in their infrastructure has some weakness? It would be reckless to give us too many technical details of what went wrong. It should be enough for us that they are working hard to avoid outages and try to bring services back up as soon as possible when something does go wrong. Sometimes there are necessary tradeoffs because the redundancy needed to guarantee uptime is simply too expensive or otherwise impractical. As others have said overall the track record is pretty good.

Originally Posted by Ailedra

This.

For those of you who've never been part of one of these little parties, it goes a little something like this:

"What the ****?"

"Uhm, dude, everything's down"

"That's not funny"

"No, seriously. Call everybody."

"Oh, that. That server bounces all the time"

"No, right now, get off the forums, call EVERYBODY"

~20 minutes later, with God and country on the phone~

*frantically searching through every change made to the platform in the last couple of days*

"No, it doesn't seem to be related to change. No, that doesn't make any sense. Ok, yes, it COULD have had something to do with it, but it still doesn't make any sense. Ok, how long to back it out? That's funny. Really? Ok, I'll let everybody know"

~20 minutes later~

"How's progress?"

~10 minutes later~

"Yes sir, I understand customers are out of the water. I'm really sorry about that, but yelling at me won't make this process come up any faster."

~5 minutes later~

"Ok, we're still seeing it, and the forums are blowing up. That wasn't it. Maybe restart this process, see if that clears it? Yes sir, I know we should have had this back up half an hour ago. We tried to tell you this change wasn't related, but you insisted we back it out, just in case."

~20 minutes later, after applications have restarted, all connectors have bound, and people try to validate~

"****. Ok, what about..."

"Sir, that vendor's busy getting people into the office, too.....it's Sunday for them too, you see. Yes, I'll call them again, but they said half an hour 5 minutes ago"

~2 hours later~

"I REALLY need a smoke, and I have to pee"

"You can wait....this is big"

~2 hours later~

"You know, I was off an hour ago. Where are we on this?"

"I don't' want to hear it...today was my day off. We were going to take the kids to a movie. My wife is going to kill me"

~2 hours later~

"I still really need a smoke. Just sayin'"


Everybody complaining about lack of communication, or that the issue shouldn't even happened in the first place....shut up. Seriously. Systems break. Things happen. When you're trying to put out the fire, giving forum junkies an accurate update isn't anywhere near as simple as you think it should be. I don't know if you realize how much money the people involved in something like today make, but I'm here to tell you, it isn't as much as you think, and even if it were, it still isn't enough to cover the running gunfight nightmare something like this ends up being.

Folks at Turbine, I'm sorry your Sunday sucked.

Ah, the good old days. I remember having to deal with situations exactly like this. And for the record peeing, while frowned upon, was at least allowed, though smoking seemed to be mandatory even inside the server room.



Avolow.

Big thanks to all who worked on fixing this, yesterday!

Thanks to all the people at Turbine who worked hard, gave up their Sunday or part or it, and solved the problems. Your diligence is to be commended. (Not that you really had a choice with all those suits yelling at you.)

Someone in my kin mentioned that yesterday was probably not a great day to work for Turbine. I bet that is an understatement. I hope no one lost their job over this. Stuff happens.

Originally Posted by secondstar

It should be enough for us that they are working hard to avoid outages and try to bring services back up as soon as possible when something does go wrong.

That might be true for you, but not for everyone, and certainly not me. As the CTO of a large SaaS organization, if one of my datacentres induces downtime for a paying customer, it is their right to know the details and impacts of such downtime, and it is my responsibility to be transparent in such matters. As a paying customer here, with literally thousands of hours invested in the world and my characters, hard work and good wishes are NOT enough. The DC guys working their asses off to get systems up is something I both respect and commend. The decision to communicate details to customers though is a customer service and executive decision, and I don't necessarily believe that their interests and mine are aligned, so I expect transparency in the face of disaster both in respect to immediate response as well as timeline for future actions. I, and am sure many others, need to know that Turbine is being accountable to its customers, that the safety of our largest investment, our characters, is top of mind when Turbine is responding, and transparency of causes and actions provides this. I pay for and expect results, not effort. I've also been faced with what I expect is an entirely similar situation, where edge network equipment at the DC has failed and even failed on redundancy, it's nasty, stressfull work and requires focus to resolve the issue, beat on suppliers and reduce downtime to a minimum. But after the fireworks are over, accountability NEEDS to take place at several levels, and certainly one of those is to the customer.

Originally Posted by Bill_the_Cat

That might be true for you, but not for everyone, and certainly not me. As the CTO of a large SaaS organization, if one of my datacentres induces downtime for a paying customer, it is their right to know the details and impacts of such downtime, and it is my responsibility to be transparent in such matters. As a paying customer here, with literally thousands of hours invested in the world and my characters, hard work and good wishes are NOT enough.

As a CTO, I would hope you understand the VAST difference between transparent root cause and preventative communication business-to-business, and business-to-consumer.

No, as a game subscriber, Turbine does not owe you that level of detail. You can't do anything with it, and you can't make a consumer's buying choice based on it.

First off, thx to any and all at turbine who spent their Sunday fixing this problem.

Second, as much as i too would like a semi-detailed explanation of what caused the problem we are not likely to ever know. Turbine is not known for having good customer service and as far as i know has never explained an unplanned or unexpected downtime.

Yes I agree Turbine you DO owe us an explnation as to what happened. I am a lifetime subscriber. I desearve to know what happened. Why? Because people that pay like me pay your saleries.

Yes I am thankful for all who sacraficed thier Sunday to fix it but now is the time to own up. What Happened Turbine??

The no answer we have been getting from you so far makes me think you have something to hide from the customers.

Nothing angers me more than companys that keep things from the customers.