Thread: Official: January 30,2011 Service Issues

Update: 10:49PM Eastern - my.lotro.com and the Lorebook are now available. All services are once again available.

Update: 10:07PM Eastern - myaccount.turbine.com is now available.

Update: 9:17 Eastern - myaccount.turbine.com, my.lotro.com, and the Lorebook are currently unavailable. We are also continuing to investigate the out of sync forum posting issues.

Update: 8:44PM Eastern - All worlds are available

Update 8:43PM - We are aware that time stamps on forum posts are out of sync.

Update: 8:20PM - We're still validating the game servers. We'll update this thread when they are up.

Following today’s service interruption all services are being brought back online. The forums are available along with the game worlds.We appreciate your patience as we worked to resolve the issues.

We’ll continue to bring systems back online and monitor them throughout the evening. Please be aware as we bring our systems back up and enter the primetime gaming hours that you may occasionally experience login issues or other issues. We appreciate your patience and understanding as we continue to fully restore our services.

Thanks for the post!

Woot! And thanks to everyone who sacrificed their Sunday afternoon to get things running again.

Thanks for all the bacon related work you guys did today to feed our jones. Scented Candles for Turbine

Can you tell us what happened, Sapience? I don't think I've ever seen all three games and all the sites associated with them go kaput like that. I know I've been teasing you all afternoon about your laundry being responsible for this, but I really am curious.

It is 8:15pm EST and I can log in, but the game still can't connect. I'm getting the error that says "The following weberror occurred while trying to access..." the Lotro server.


Edit for those arriving late: Some decided to post here because the OP originally said game servers were up. That has since been edited to say only that web services were up.


Rip

Originally Posted by GV-Tanith

Can you tell us what happened, Sapience? I don't think I've ever seen all three games and all the sites associated with them go kaput like that. I know I've been teasing you all afternoon about your laundry being responsible for this, but I really am curious.

Pretty sure it would have to have been network related - a fibre cut, massive DDoS, or something similar.

Please note that sapience did NOT say that the game was up, but that web services were up.

FWIW, when I ping gls-auth.lotro.com, I get numerous timeouts.

e.g.

>ping gls-auth.lotro.com
Pinging gls.lotro.com [74.201.102.12] with 32 bytes of data:
Request timed out.
Reply from 74.201.102.12: bytes=32 time=102ms TTL=238
Request timed out.
Reply from 74.201.102.12: bytes=32 time=103ms TTL=238
Ping statistics for 74.201.102.12:
Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),
Approximate round trip times in milli-seconds:
Minimum = 102ms, Maximum = 103ms, Average = 102ms


(I'm west coast US, so successful ping times are appropriate.)

Well I obviously can't say what happened for sure, but it wasn't a 'fiber cut' or anything else that wide scale. I have servers at the same DC Turbine does and I didn't experience any downtime.

I'd say the issue was isolated to some equipment for Turbine's servers. I don't think it was a DDOS because of how things are coming back up but that is just a guess...

The servers are currently unavailable. They should be back online soon.

Originally Posted by Boraxxe

Please note that sapience did NOT say that the game was up, but that web services were up.

Actually, he did. He just edited it to remove that.

Edit: and the time-warp thing is really bad. It's almost impossible to follow the threads with the responses bouncing before and after the posts.

Originally Posted by Arbalister

Pretty sure it would have to have been network related - a fibre cut, massive DDoS, or something similar.

I expect so. I don't know whether it would help at this stage, but I will relate this.

About twenty minutes before the Great Darkness fell, I was on my baby burglar on Landroval. I noticed a large (20-30 seconds) lag spike, and asked in glff if anyone else had experienced that. No one had, so I shrugged it off. And a short time later, BLAM!

Originally Posted by DunasConnor

Actually, he did. He just edited it to remove that.

Edit: and the time-warp thing is really bad. It's almost impossible to follow the threads with the responses bouncing before and after the posts.

Correct. Originally we were going to open everything, but as you can see we're still seeing some issues with the forums (check the time stamps) and back end services. We're holding off on the servers until we stabilize a few more things.

Originally Posted by probitas

What exactly happened? Is there a chance we lost our accounts? Was there a roll back? We know there was a problem, you don't shut down all your sites and the games for nothing less, but what could we experience as a result?

Servers got rolled back to 2008...sry

Originally Posted by GV-Tanith

Can you tell us what happened, Sapience? I don't think I've ever seen all three games and all the sites associated with them go kaput like that. I know I've been teasing you all afternoon about your laundry being responsible for this, but I really am curious.

I know what happened!!! Turbine just moved to a new facility and somebody <looks at Patience> brought their cat to the new offices!

http://www.youtube.com/watch?v=PJW3Jpqjx5s

I have been attempting to trace IP's since the issues began. For a good few hours all of Turbine IP's were unrecognised. Later they were traceable but showed moderate pl on the last 4 hops. All appear not bad now other than between 10 and 60 pl at 74.201.102.154 (second to last hop)

It's 8:31 EST and Landroval is up.


Rip

Originally Posted by Arbalister

Pretty sure it would have to have been network related - a fibre cut, massive DDoS, or something similar.

If it was just a connectivity issue, they'd be running the moment they got it back, but they aren't, so it's not that simple. Whatever it is, it is massive, and all encompassing.

By the way, this post came AFTER my post asking what happened, just 2 down...

Successfully logged into Landroval at 2033 Eastern. Thanks for all the work.

I just tried logging in repeatedly leaving the "last server" checkbox unchecked and was able to get in eventually.

Vilya is up, logged in at 8:35 EST

First time able to access forums here in South Florida; on log in, still getting nulls. I'll keep my fingers crossed.

Notice the posting counters have reset... Hmmm... rollback??
I also notice my sig is gone.

Well, in now and some things are off on the character I was playing during the crash:

1. Some things I sold are back in my inventory, though I did not lose the money I got from selling them (vendor trash)

2. Some things I got during questing/skirmishes are gone.

3. All my Skirmish Dailies are available. I already did half of them before the crash, and did not lose the xp I received, nor the deed counts earned during those Skirmishes.

On 1. and 2. only some of the things, it was random in both cases, not just things I got or sold during the last hour before the crash.

Might need to run an integrity check on the character DB.

Edit: Not this account, my other one, if it matters.

Originally Posted by ericpae1

Vilya is up, logged in at 8:35 EST

I still can't get in on Vilya.

8:42pm EST - and I'm back to endless waiting on the "getting data centers" message.

8:46pm EST - finally got to character screen on Vilya!!!!!!!!!!!!

8:50pm EST - but the game is not responding....

8:53pm EST - Connection failed. Retry....

8:59pm EST - Finally In.

When I was logged out, I was finishing up the advanced warg slayer deed, and I was in the broad ways. I'm now in 21st Hall. I started at 219/240, killed up to 228/240. Now I'm back to 219.

Check your quests, deeds, and other stuff...looks like you might be reset back to where you were at the start of the day.

I am in on Gladden.

DDO all worlds have been up and running for about an hour now. I or some of us still cannot get in the game. DDO must be there BABY for Turbine. They get the fixes first.

All worlds are now available.

Originally Posted by Winterfell

Servers got rolled back to 2008...sry

Oh man!

can't get pasted "Examine Programs"

9:01 EST

Yeah finally got back in (at about 9pm Eastern time).

It would still be nice to know what happened, though. What caused the interruption?

FWIW, I'm getting the Sitedown page when I try to go to My LOTRO.

Originally Posted by probitas

What exactly happened? Is there a chance we lost our accounts? Was there a roll back? We know there was a problem, you don't shut down all your sites and the games for nothing less, but what could we experience as a result?

I don't understand. What makes you think anything happened to our accounts? They had Datacenters issues. It took a long time to diagnose and fix the issue. Now (almost) everything is up. End of story.

Same; Null Error as for logging into the game itself, and the other sites themselves are still not working; only the forums. (And even some of the forums links are busted, taking me to the 'scheduled maint' screen. Obviously this isn't a complaint, just trying to help isolate issues.

Logged into Crickhollow with no issues.

What exactly happened? Is there a chance we lost our accounts? Was there a roll back? We know there was a problem, you don't shut down all your sites and the games for nothing less, but what could we experience as a result?

My husband was crafting when he got kicked off earlier and now after logging back in he's found that the items he crafted as well as the materials he used to craft them are gone.

Originally Posted by Fionnuala

My husband was crafting when he got kicked off earlier and now after logging back in he's found that the items he crafted as well as the materials he used to craft them are gone.

Maybe if he files a ticket, a GM will be able to look back at logs and help him out as part of their new lost item reimbursement policy.

I'm IN on Firefoot...19:34:22 Central time zone.

I just want to take a moment to give props to Sapience, Tolero, Frelorn, and others who have been keeping us all up-to-date on developments today. Also to the techies who came running in to put things right again. Whatever happened, you all did very well. Go have some bacon.