LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > General
User Name
Password
General This forum is for non-technical general discussion which can include both Linux and non-Linux topics. Have fun!

Notices


Reply
  Search this Thread
Old 05-28-2017, 02:15 AM   #1
hazel
Senior Member
 
Registered: Mar 2016
Location: Harrow, UK
Distribution: Debian, Crux, LFS, AntiX, NuTyX
Posts: 1,445
Blog Entries: 1

Rep: Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614
Another IT disaster! You don't even need malware, just incompetence.


The whole of British Airways closed down yesterday when their IT systems crashed. It will take days to clear up the mess.

This time there was no sign of malware. It was just a systems crash. Employees of the company say that the IT department has been starved of funds and important maintenance work has been contracted out.
 
Old 05-28-2017, 04:02 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,959

Rep: Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211
What chance they decided to use the long weekend for some work in the datacentre ?.
I've worked for a vendor that had customers (BIG banks) that decided to do major upgrades over the "quiet" time between Christmas and New Year.
Sometimes (not always) the customers didn't even notice ....
 
Old 05-28-2017, 05:07 AM   #3
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 2,641
Blog Entries: 3

Rep: Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160
Quote:
Originally Posted by hazel View Post
It was just a systems crash.
A systems crash or were they trying to run production services on top of M$?

I'm pretty jaded about these kind of things and when I see that they have avoided mentioning specifics it usually turns out that M$ products were the software culprits. We'll have to wait and see if more information is ever released. Remember the LSE.

Though if it is the case they have deployed M$ products instead of production-ready systems, then the real culprits are the managers that brought in the M$ staff and told them to deploy M$ products in a production environment. So we might not ever find out what really happened, especially if it was M$ related.

An anecdote: I recall one bank that went down hard for weeks. It was actually months before they could really resume normal operations. During the first weeks many could not transfer money into or out of their accounts. That applied to businesses as well as individual private customers. On occasion, it was possible to access other accounts and transfer money in or out, but not your own. It turned out to be 100% an M$ problem, mainly Sharepoint. However, the media loyally reported on 'computer glich' and 'IT problems' instead. When pressed, they repeated the talking point "IBM is responsible for the network" though the problem was unrelated to the network.
 
Old 05-28-2017, 11:59 AM   #4
Soadyheid
Senior Member
 
Registered: Aug 2010
Location: Near Edinburgh, Scotland
Distribution: Cinnamon Mint 17.3 and 18.2 at present.
Posts: 1,295

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
Quote:
What chance they decided to use the long weekend for some work in the datacentre ?.
Er... unlike banks which go quiet on holiday (Bank Holiday?) weekends, Airlines volume of business increases!

The press was quoting some sort of power failure? Huh? I really can't believe that in this day and age a company as global as British Airways doesn't have at least a second DR Datacentre, if not more, fed by multiple power sources and backed up by diesel generators!

I'd reckon they're still investigating and the media are just putting out statements which they think Joe Public will accept as the cause.

Play Bonny!

 
Old 05-28-2017, 07:33 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,959

Rep: Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211
Neither do banks - hence my use of double quotes.
 
Old 05-29-2017, 07:56 AM   #6
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 2,641
Blog Entries: 3

Rep: Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160Reputation: 1160
It looks like there are several different kinds of stupid involved, above and beyond simply outsourcing core, mission-critical functions.

Circumstantial evidence is starting to pile up. Here are some relevant items from last year:

1a. BA faces IT staff protest over offshoring
1b. Fears for up to 800 UK jobs as British Airways hires Indian firm Tata Consultancy Services to provide IT services

2. TCS Recognized as a Global Leader in Microsoft Enterprise Applications Implementation by IDC MarketScape

In all likelihood, M$ has flown out several teams of their highest paid PR flunkies to help draft some more misleading announcements to be published soon, maybe even to hide the BA board of directors involvement. I'd guess no major decisions, like those leading up to the disaster, could be made without the board members' knowledge and approval.
 
Old 05-29-2017, 08:49 AM   #7
Jjanel
Member
 
Registered: Jun 2016
Distribution: any&all, in VBox; Ol'UnixCLI; NO GUI resources
Posts: 997
Blog Entries: 12

Rep: Reputation: 359Reputation: 359Reputation: 359Reputation: 359
RHEL? https://www.redhat.com/en/about/pres...cloud-platform
Quote:
The Group’s work with Red Hat started in 2003, with British Airways selecting Red Hat Enterprise Linux as its Linux-based platform to support core operational functions, including flight and passenger management. International Airlines Group was formed in 2011 following the merger of British Airways and Iberia and has since acquired Aer Lingus and Vueling. The core commitment to an open source technology backbone remained, with Red Hat Virtualization now running on more than 600 physical servers and supporting more than 7,500 virtual machines.
August 9, 2007 an electrician accidentally blew out the power to datacenter where cisco.com had webservers, with no dynamic web content failover implemented. https://blogs.cisco.com/news/final_u...iscocom_outage
 
Old 05-29-2017, 01:37 PM   #8
DavidMcCann
Senior Member
 
Registered: Jul 2006
Location: London
Distribution: CentOS, Salix
Posts: 4,678

Rep: Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494Reputation: 1494
The latest from Reuters:
Quote:
BA Chief Executive Alex Cruz said the root of the problem had been a power surge It was so strong it also rendered the back-up systems ineffective.
This sort of thing is hardly uncommon: Lufthansa, AirFrance, and Delta have had similar, but smaller, crises in the last year.

The staff protests were about the out-sourcing of a data-centre, which is obviously unconnected. Knowing the GMB (General, Municipal, and Boiler-makers!) union, I can sympathise with BA not wanting them handing their customers! How's this for a conspiracy theory: GMB member sabotage?
 
Old 05-29-2017, 06:16 PM   #9
Soadyheid
Senior Member
 
Registered: Aug 2010
Location: Near Edinburgh, Scotland
Distribution: Cinnamon Mint 17.3 and 18.2 at present.
Posts: 1,295

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
Just a few overall thoughts...

One of the problems being generated by globalisation is the acquisition and merger of large companies; airlines, banks,etc, into even larger ones. This involves attempts to rationalise their associated IT infrastructures (including staff, don't we know it!) to lower costs and increase profits. Invariably the rationalisation will involve some legacy equipment which defies upgrade attempts as it supports some invaluable application which was written in a language no longer taught and was only understood by programmers now retired or long dead.

The equipment itself, which has chugged along merrily for decades, has been superseded by newer and faster kit becomes harder, and more costly, to maintain. (Just finding spare parts for the irreplaceable 1960's ZXQ3000-1 surface-mounted-yogurt-maker is infinitely time consuming and pretty much pointless, even using EBay!)

Trying to "bolt" together the differing networks and make the resulting conglomeration of ill matched hardware and software function as the proverbial "Well oiled Machine" is a gigantic task which may now be beyond the actual human resources available. There seems to be a distinct lack of programmers, analysts and other IT staff in the world to go round!

Just to make things worse, we in the UK, have managed to add to this problem, by voting to leave the EU which is going to add to the resource depletion as untangling all the legal, trade, health, security, financial treaties, etc, to enable us to "Brexit" is going to be a nightmare. And the Politicians say it'll take only two years! Two years my a*se! It'll be nearer ten, and that's probably an underestimation.

Perhaps we should be more worried than we appear about the future, technology failures and our dependencies?

This problem was foreseen back in 1909 by E. M. Forster, The Machine Stops.

I'm now going to lie down quietly in a darkened room...

Play Bonny!

 
Old 05-30-2017, 02:35 AM   #10
hazel
Senior Member
 
Registered: Mar 2016
Location: Harrow, UK
Distribution: Debian, Crux, LFS, AntiX, NuTyX
Posts: 1,445
Blog Entries: 1

Original Poster
Rep: Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614
Quote:
Originally Posted by Soadyheid View Post
Invariably the rationalisation will involve some legacy equipment which defies upgrade attempts as it supports some invaluable application which was written in a language no longer taught and was only understood by programmers now retired or long dead.
I love it! There is a place somewhere in the US (I've forgotten the name) where they emulate all kinds of obsolete hardware and software. Apparently there's an awful lot of data stored on tape in obsolete formats which can no longer be read on modern machines.
 
Old 05-30-2017, 08:00 AM   #11
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 8,629
Blog Entries: 4

Rep: Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999Reputation: 2999
I consider it most likely that this was a small-scale experimental attack on this business, at a most-critical time in its business year, and that it was a successful one.

Like many other corporations, BA "outsourced" its most-essential operation – its IT operation – to a "third-world third-party," with no more motivation than: "it would allow thousands of 'expensive British people' to be replaced by 'cheap third-world people,' hence 'more profits.'"

So, working very quietly from their out-of-the-country 'cheap' data centers, these 'cheap' out-of-the-country workers brought British Airways to its knees.

They could have – and still could – bring airliners crashing to the ground ... the full-scale equivalent of a terrorist hijacking ... from thousands of miles away, just by tampering with digital records. And, no one from Britain would know, because they'd all been laid off "to save money."

No one is watching the chicken-house. This attack was completely successful.

By steadfastly ignoring the human factor of IT ... by pretending somehow that we can trust machines and thus hire "anybody, anywhere (as long as they're 'cheap')" to run them, and that we can locate them "anywhere in our 'happy little cloud' (as long as labor and electrical power are cheap there)" ... we have set ourselves up worldwide for "devastating 'Acts of War.'"

We naively assume that every person anywhere is totally trustworthy (as long as s/he is 'cheap'), and then we profess to be 'mystified' when events like these occur. "What fools these Mortals be!" Do we not know that there are dark-hearted people out there who possess powers of imagination?
 
Old 05-30-2017, 12:24 PM   #12
Soadyheid
Senior Member
 
Registered: Aug 2010
Location: Near Edinburgh, Scotland
Distribution: Cinnamon Mint 17.3 and 18.2 at present.
Posts: 1,295

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
Hmmm... Interesting theory, probably the same off-shored IT people who brought down the financial institutions in 2007 - 2008.

As airline disruptions go, I prefer this method, no IT involved!

Play Bonny!

 
Old 06-01-2017, 03:03 AM   #13
hazel
Senior Member
 
Registered: Mar 2016
Location: Harrow, UK
Distribution: Debian, Crux, LFS, AntiX, NuTyX
Posts: 1,445
Blog Entries: 1

Original Poster
Rep: Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614
The latest version on the BBC News this morning:

1) There was a power failure.
2) The backup power system didn't work so they had to reboot the servers.
3) The people who did the reboot screwed up. This could have been because they had outsourced their expertise to Tata as per Turbocapitalist.
 
Old 06-01-2017, 07:06 AM   #14
Soadyheid
Senior Member
 
Registered: Aug 2010
Location: Near Edinburgh, Scotland
Distribution: Cinnamon Mint 17.3 and 18.2 at present.
Posts: 1,295

Rep: Reputation: 301Reputation: 301Reputation: 301Reputation: 301
Quote:
1) There was a power failure.
In both Data Centres? Any global company worth their salt must have at least two Data Centres. Generally quite a distance apart (In case a BA plane crashes into one of them?)

Quote:
2) The backup power system didn't work so they had to reboot the servers.
I wonder how often they fail over to DR systems to check they work? Not often enough probably. Sometimes critical maintenance procedures like this just get risk assessed and somebody takes the gamble. Not worth it!

I've been in a Data Centre when the power failed, head stuck in a server rack, pitch black, scary as hell, then I noticed all the little red lights and the screaming audio warnings from the UPS' Yup, it was way back when all the servers were clustered, no VMs, each box had its own UPS. Luckily the power came back before the ten minutes UPS autonomy expired. Changed days!

Play Bonny!

 
Old 06-01-2017, 07:37 AM   #15
hazel
Senior Member
 
Registered: Mar 2016
Location: Harrow, UK
Distribution: Debian, Crux, LFS, AntiX, NuTyX
Posts: 1,445
Blog Entries: 1

Original Poster
Rep: Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614Reputation: 614
It's an old story. People religiously do backups and store them away, then one day they get a data crash and need to restore from backup. And whaddyaknow! The backups are corrupt, probably have been for a long time. Because no one bothered to test them.

I bet they don't often test backup generators either.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Using Android 4.3? Don't let malware snatch your private login keys LXer Syndicated Linux News 0 06-30-2014 07:41 PM
Considerations with regard to incompetence Dogs General 6 04-28-2010 11:57 AM
May have contracted malware. Yes, malware. Firefox on Ubuntu Fiesty. Seeking a fix drachenchen Linux - Security 1 06-12-2008 06:10 AM
Installation incompetence mst3kman Linux - Software 2 01-03-2006 04:14 PM
VMware GSX, PAM, and my general incompetence with servers Electrode Linux - Software 5 03-28-2005 05:38 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > General

All times are GMT -5. The time now is 02:38 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration