Delta airlines failure today!

[Woodhauler]

My wife was caught in the middle of that this morning! Had to fly to Knoxville this morning for work, plane was to leave at 6:40 am, never got off ground! They couldn't get her on another flight so she drove 150 miles to Portland maine and just got on a plane 10 minutes ago! Wonder how delta will handle this with all the flights world wide grounded???  Bangor maine airport wasn't on the same computer deal as rest of country so she was never contacted about problems.

[tree-farmer]

The employees that had to deal with hundreds of angry passengers had a Monday to remember. It they did not drink before, they probably started today... :D

[coxy]

what happened  ??? ???

[Ianab]

Their main data center went down, apparently when the backup generator caught fire, and took out the mains supply as well.

You don't keep 800 planes organised without the computers, and even once they get the data center running again, most of the planes and crews are now in the wrong place, so the disruptions continue as they reshuffle things. 

[DanG]

This is just blatant corporate incompetence!  Every telephone office in the country runs its computers on batteries.  Commercial power only keeps the batteries charged, so they don't even blip when the power goes out.  Besides that, every one of those pilots knew when and where they were supposed to go next, and the passengers already had tickets. There is no reason to rely so heavily on computers that you don't allow people to think!  Truth be known, I'll bet their "massive" computer system has less power than the average laptop anyway! DanG thing is probably 30 years old. :

[pine]

This is just blatant corporate incompetence!  Every telephone office in the country runs its computers on batteries.  Commercial power only keeps the batteries charged, so they don't even blip when the power goes out.  Besides that, every one of those pilots knew when and where they were supposed to go next, and the passengers already had tickets. There is no reason to rely so heavily on computers that you don't allow people to think!  Truth be known, I'll bet their "massive" computer system has less power than the average laptop anyway! DanG thing is probably 30 years old. :

Agree on the blatant incompetence. 
It happened to my company a couple of years back and to Southwest Airlines several months ago. 
It cost money, a fair amount of it, to properly backup your systems, not just power but the computers as well ( I am not talking about data backup but actual parallel running computers) but to the best of my knowledge none of the airlines have those system installed. 
Inexcusable IMO.

As to tickets that folks have purchased, there are industry reports that over half of all tickets are totally electronic no paper evidence at all.  Spirit airlines actually charges a huge premium for a non-electronic ticket.  So there is literally no ticket if there is no computer.  We have become slaves to computers.  Look at yourself and your computer and your cell phone.

Pilots may know where they are supposed to go but without the computer systems they are not authorized by the national airspace structure and the FAA approved "Operations specifications"  (ops specs) to go.  Flight plans have to be filed a certain way, dispatch has to perform certain functions to include tracking of all flights, weight and balance has to be accomplished and weight and balance on a 747 is not like you may have done on a general aviation aircraft if you happen to be a pilot.

Even before 9/11 things had to match in computers but post 9/11 if stuff does not match you don't fly, ticket or no ticket, without the working computer it can not be verified.

Yes people could be boarded but unverified as to whether they have a ticket or not, pilots could fly from point A to point B but not legally and they could never get an ATC clearance to takeoff.

Face it we are slaves to the computer and technology.  What we have to demand is for mission critical elements in all industries, and yes the airlines have become mission critical to the economy of the US and the world, to have scaleable functioning backup systems.

[DanG]

Pine, the phone companies actually do have parallel running computers on everything, even the little roadside boxes you see everywhere. They still manage to make a profit. Don't tell me the weight & balance calculations are done on the ticket computer!!    W&B is not nearly as critical on an airliner as it is in a helicopter, where we had to move the battery to a different location according to the load we were carrying!  If it was, they would be assigning seats based on passenger weight.  :D :D

We do agree overall that their system is poorly planned and poorly run. That is one of the reasons I haven't flown with them in over 20 years, and will not in the future. That goes for all airlines.

[Ianab]

This is just blatant corporate incompetence!  Every telephone office in the country runs its computers on batteries.  Commercial power only keeps the batteries charged, so they don't even blip when the power goes out.  Besides that, every one of those pilots knew when and where they were supposed to go next, and the passengers already had tickets. There is no reason to rely so heavily on computers that you don't allow people to think!  Truth be known, I'll bet their "massive" computer system has less power than the average laptop anyway! DanG thing is probably 30 years old. :

Unfortunately they did have that backup power, with UPS and Generator backup. That's what caught fire. 

Then about 600 servers in the server room suddenly go quiet.,  While I bet a lot of the systems are probably getting a bit out of date, there is going to be more going on than you would track with a PC. All the tickets, all the luggage, all the meals etc. Then the background stuff like staff rosters, payroll, plane maintenance, and the real time reporting to various govt agencies.

The issue was having a single point of failure (One data center) Of course having complete redundant data centers costs a LOT, and that's where the problem comes in.

[pine]

In the Delta case I don't know if only the ticket computer went out or if the entire Delta computer system went down.' Does not really matter
You would be surprised how inter-connected the computer systems are and how the loss of one will take out or effect another.  The point is and my point specifically is that there needs to be a more robust system that is not so easily effected by a single point failure.

As to the phone company and their backups.  All I can say is 20-30 years ago I rarely to never lost phone service. 
Now a simple power outage takes out the phone service.  The fiber optic systems fail when the power goes down. There is supposed to be battery backup but there is not.  Now I lose my landline everytime the power goes out, I never used to.  10 years ago we were without power for 10 days due to a major storm, phone worked the whole time no problems.  Now if the power goes out, the phone goes out as well. No we are not talking VOIP, we are not talking cell, we are talking standard hard line "Ma Bell" type phone service.  It is no longer a reliable phone service or emergency phone capability.  My lousy cell phone as bad as it is has a higher reliability than my hard-line phone.

Don't tell me that the phone system is reliable as it is a joke. If  the phone company has parallel running computers with multiple power backups and are as pathetic in service as they are then I have to revise my call for the airlines to do the same as it does not work; so why even try it.

[DanG]

Well I retired from the phone company 12 years ago. Guess I shouldn't have, eh?  :D :D My point was, I know what can be done to avoid interruptions and Delta didn't care enough to do it.  If the phone systems are no longer redundant, then they are just as much at fault as Delta.

[Brucer]

To add to DanG's comment, Google and Facebook have multiple redundant systems. They can (and do) lose servers at any time but you would never notice. It isn't just a matter of having backups -- they are designed to keep on working seamlessly if a server suddenly drops out.

They also keep server farms in multiple locations supplied by different utilities. The technology exists but there has to be a will to use it.

[Ianab]

Even in a redundant system things can still go wrong. The software / hardware that handles the switchover can fail.  :

On a modern phone system a single controller can handle a huge number of lines. If may not go down very often, but if it does you might loose a million or so lines. Even the interlinks between the redundant systems are vulnerable. Big bits of NZs phone systems can go down, sometimes from hardware failure, sometimes human error, sometime a natural event.

One time a rat ate a fibre cable under a bridge in the middle of nowhere, at the same time a digger cut through the 2nd cable some place else. Even then they had planned for that, and there was a third cable. But they hadn't planned quite well enough, and it couldn't handle the traffic with the other 2 out, and the system went down.  Do'h

But Delta hadn't even got as far as a redundant data center, so they have a single point of failure.

From what I've read it was the main power into the data center, so they lost everything.