BA IT Breakdown

[davemercedes]

Fri Jun 2 - If you're game for a laugh:
http://uk.reuters.com/article/uk-britain-airports-ba-idUKKBN18T0KT
BA's IT outage cause by contractor...
- This says a contractor inadvertently switched off the power supply...
- Err - surely they have a UPS (Uninterruptable Power Supply) ?

 

[Frosty149]

Nah, the cleaner plugs her vacuum into the socket that powers the UPS every night...

 

[davemercedes]

Many a true word spoken in jest!

 

[Chrishazle]

Wonder if the contractor's insurance covers £150 million of Gross Negligence? Somehow doubt it!

 

[davemercedes]

Highly unlikely... we had insurances to about £6M as I recall for on site work. But our contract stipulated that the client insured against "mishaps". Our contract also put the responsibility for losses arising (lost/delayed revenue etc) on to the client so if we screwed up the billing operation (not that we ever did!) it was their insurance that had to cover it.

How on earth this problem occurred without an instant high level alarm is the first question (it seems the first they knew was when user screens failed), the second is how on earth it wasn't put right within an hour. Twenty years ago a company like this would have had a parallel-running device with its own power source and battery backup that would switch over instantly if the parent died so there's not much new about belt and braces! Mind you, backup devices have to be regularly tested to make sure their power is okay (been there etc...).
- It will be interesting to see if news about the cost of "lost data" emerges later on....

Ho ho ho - remember the slogan "We'll take good care of you..."

 

[Frosty149]

Highly unlikely... we had insurances to about £6M as I recall for on site work. But our contract stipulated that the client insured against "mishaps". Our contract also put the responsibility for losses arising (lost/delayed revenue etc) on to the client so if we screwed up the billing operation (not that we ever did!) it was their insurance that had to cover it.

How on earth this problem occurred without an instant high level alarm is the first question (it seems the first they knew was when user screens failed), the second is how on earth it wasn't put right within an hour. Twenty years ago a company like this would have had a parallel-running device with its own power source and battery backup that would switch over instantly if the parent died so there's not much new about belt and braces! Mind you, backup devices have to be regularly tested to make sure their power is okay (been there etc...).
- It will be interesting to see if news about the cost of "lost data" emerges later on....

Ho ho ho - remember the slogan "We'll take good care of you..."

I understand that the backup system is only a few miles down the road and suffered from the same power outage!
Some IT bod was asleep and I dare say looking for new employers about now....

 

[davemercedes]

The IT Manager/Director's position is probably doubtful...

 

[Frontstep]

Hopefully the same people will not be covering the movement of London Airport Air Traffic control away from the control towers.

I must be old fashioned but I like the idea of people looking through adjacent windows with the addition of suitable tech not 80 miles away.
Three cable connections to be made and looking at this an Ebay power supply on a double socket with the brew room kettle.

BA no doubt had some wiz in the accounts department find some amazingly cheap IT company out in India.

 

[davemercedes]

I certainly agree on Traffic Control being to look out the window and see where you are...

On the Campaign4Change website it says BA is “reviewing” what went wrong... but has no regulatory duty to publish the findings.
- It also mentions the UPS (Uninterruptible Power Supply) - which was also apparently turned off (my first question).
- That uses mains electric, diesel and batteries
- Apparently when turned back on in an uncontrolled fashion the UPS caused “catastrophic physical damage” to BA’s servers.
- Hmm What kind of UPS is that then?
- I'd love to read the in-house emergency procedures!

No doubt they will all be ducking and diving until they find a "minnow" to blame.

 

[Frontstep]

It seems like an oxymoron "the uninterruptible power supply was switched off"

The emergency procedures are buried in book 3/9 14U in the boot of my car which the wifes got.

Errh aren't that lot in India sorting that now ?

Or "how the f**k do I know we always just switch stuff off then on again".

 

[davemercedes]

According to an article on Auntie today, the BA chief says their IT fiasco will cost "in the order of £80M".
http://www.bbc.co.uk/news/business-40285288
- There's no mention of whether that guesstimate includes future lost sales revenue/profits.
- Part of me would like to see them suffer like that because of the disgraceful way they seemed to just abandon passengers.

Today (Thursday) BA were one of of several airlines affected by a baggage system problem at Heathrow

Thank heavens I only fly rarely nowadays!

 

[Frontstep]

So just how much did they save outsourcing their IT ?
That's another £80 million to go off the savings so far.
Hopefully the bonuses are still being paid

 

[davemercedes]

So just how much did they save outsourcing their IT ?
That's another £80 million to go off the savings so far.
Hopefully the bonuses are still being paid

In fairness they're all at it - they all outsource whatever they can, the theory being that they can take care of the business while other "expert" companies run operational things to their instructions... but in this article from 2014, one paragraph begins with these wise words:
>> But while outsourcing saves money, it can come with big risks.

https://www.ft.com/content/4a6ead0e-c54f-11e3-89a9-00144feabdc0