NHS IT System Failure
IT Failure Affects 80 NHS Trusts
Pathetic!
Computer services in 80 NHS hospital trusts are down after an equipment failure on Sunday.…
Failed Sunday morning, to be fixed by Monday evening? Boy, that’s real quick…
This incident was caused by storage area network equipment failure and has affected several other organisations," he said.
"Technical issues following power system interruptions mean that data held on computers in the central data centre for the region cannot be accessed.
"The nature of the incident meant that service could not immediately be provided by the back-up systems.
Well that’s good to hear. Their SAN goes down, and there’s no disaster recovery. The "experts" running this £20 billion project haven’t heard of database replication, even though switching it on in Oracle, SQL Server, Informix or DB2 is as difficult as switching on a DVD player.
He said trusts could register patient movements using a paper system, but that it was more "time-consuming and cumbersome".
Paper system? I’d laugh if it wasn’t my money being spent. £20 billion and your contingency plan for server breakage is a paper system? So, no possible way that the customer information could have been collected on a local system, and held until the servers came back online, at which time it would get transferred? It’s not that hard.
Experts from CSC Alliance, the company which manages the systems, and its sub-contractor Hitachi have been "working round the clock to restore access to data", he added.
Experts? Pah! If you’d had a disaster recovery site, then the job to "restore access to data" is little more than a redirect from live to backup data. When I’ve had to redirect traffic, it’s a five minute job changing a config file.
80 NHS Trusts were knocked out by this failure. So, they couldn’t have put the data for each trust on a separate database server, so that if one fell, the rest could carry on? I wouldn’t be at all surprised if the whole thing is based on a vertical architecture.
This doesn’t bode well. Lack of disaster recovery, a system that takes most of a day to bring up, no consideration for network breaks, and the data being subject to a single point of failure.
Update: According to e-health insider, some of it is still down, and will be until Thursday. This is bad. Apparantly, there was a disaster recovery solution, but "The ‘hot’ standby disaster recovery data centre that was then meant to automatically take over, ensuring an uninterrupted service, failed to work.".
I’d like to know the full details of the "failed to work". Hardware? It’s possible that the backup SAN could also fail at the same time. Possible, but I’ll give it a "computer says no". I’ve never heard of it ever happening in such a short period. If it were something to do with network routing, then why would it take 3 days to fix? Networking isn’t a speciality of mine, but I can’t see why swapping IP addresses on the router/domain controller thingies would take that long.
I’ve got my own ideas on what’s happened, but I’d be interested to know opinions.
RSS
"I’d laugh if it wasn’t my money being spent."
I am not laughing either. Who is analyzing the risks here?
Every single large-scale computer project that is Government specified/funded/administered/maintained has turned out to be an enormous, inefficient and ineffective waste of public money. There is no argument for a nationwide NHS patient record system - that is unless things get so bad that patients in Lands End have to get treated in John o’ Groats!
The sole upside of the observation that Government IT projects are virtually pre-ordained to fail is that the national ID card system will almost certainly never be implemented.
Tim, you’re certainly right about the risible level of redundancy and backup in this system. Any modern DBMS includes replication and clustering (even MySQL) and the fact that their idea of ‘high availabilty’ means a day’s downtime is grounds for someone in management to get the axe (I and my IT team get yelled at by the powers that be if our servers go offline for more than thirty seconds). Of course, this is the State, where failure seems to have no consequences.
"Fixed by Monday evening". According to reports at http://www.e-health-insider.com/news/item.cfm?ID=2044, it’s still down NOW on Tuesday evening! As you say Tim, this is unbelievable.
I disagree with "a Taxpayer from Sussex" though. I do believe that we need a nationwide system. Why? Well, if I have a life threatening illness, actually any illness requiring intervention by a surgeon, I would happily travel the length and breadth of the country to find the best qualified surgeon, with the best measured results for my specific condition. I do NOT want to be constrained by what the chaps at the local golf club… ahem, sorry, general hospital can do.
Thanks for all the comments, people.
I’ve added an update based on ClickRich’s comment.
I think my greatest concern is in the concentrated nature of the architecture. That one hardware failure knocks out 80 trusts. My own approach to design is about trying to create as much horizontal scaling as is sensible. It doesn’t stop those servers talking to each other, but it means that when one fails, it doesn’t take the rest out.
NHS Medical Data Base - Government plans to put medical records on a national electronic data base. My personal view is that psychiatric files should inform the appropriate agencies if a person was genuinely potentially dangerous (to themselves or others) or if the person was in danger from others. Documentation should also inform the appropriate agencies as to the person’s level of vulnerability and need of assistance. I have very great concerns about the proposed new data base. I have seen huge amounts of paperwork concerning myself, most of it is inaccurate, misleading and open to interpretation. .
Complaints - One of the main problems service users have when complaining about mental health authorities is that the authorities decide the rules of their complaints procedure. In my experience there is also a tendency to penalize the complainer and be-little the complaint as being part of a psychiatric dysfunction. This site will campaign for greater accountability from psychiatric services. Their exists no independent authoriety, to protect the rights of mental health service users. No board of vistors, no independent complaints commision etc. This is clearly wrong and un-democratic, it is very easy to dis-couredge a mentally ill person from complaining.
Duplicate comment removed
[…] like in July 2006 when a piece of computer equipment got knocked out and 80 NHS trusts had to operate without any […]