Ulster Bank counts the cost of catastrophic IT meltdown

Given the febrile climate of financial markets, the technical problems could not have come at a worse time for a group that has already had a chequered recent history, writes SIMON CARSWELL,Finance Correspondent

LONG LINES OF post-its and large charts of tasks to do paper the walls of the offices of the fourth floor of Ulster Bank’s head office on George’s Quay in Dublin.

The rooms named after Irish islands – Lambay, Aran and Tory – are usually used for sit-downs with top clients and meetings between managers and staff. For the past two weeks, the fourth floor has served as the base for a “war cabinet” as senior management at Ulster Bank, owned by the state-controlled British lender Royal Bank of Scotland, struggle to deal with one of the worst technical failures experienced in world banking.

“I have been with various parts of this group for 38 years and it is unprecedented in world terms – that is why the whole world is watching it and will learn from it,” said Chris Sullivan, the head of UK corporate banking at RBS who is responsible for Ulster Bank within the larger British banking group.

Just how a technical glitch triggered by a software upgrade to a computer in an office in Edinburgh can collapse the processing of a daily “batch” of 20 million transactions at one of the biggest banking groups in the world has left many scratching their heads.

It has caused upheaval in the lives of customers in Britain and Ireland, but most spectacularly for well in excess of 100,000 customers at Ulster Bank who have struggled to access their own cash and have been unable to make everyday payments since June 19th.

After getting the timing of when the problem would be fixed wrong twice since the crisis began, Ulster Bank said on Wednesday that it was confident that next week would be the last for the computer problems and that the accounts of customers at Ireland’s third largest bank would start return to some kind of normal service by Monday, July 16th.

Given the febrile climate of financial markets and customer concerns about the security of deposits amid a four-year banking crisis, the technical problems could not have come at a worse time for a group that has already had a chequered recent history.

“We did have contingency plans in place. It is fair to say that there are gaps in how that unfolded,” Ulster Bank chief executive Jim Brown told The Irish Times. “The key is getting through these issues and how we can minimise going further and how we can prevent this happening again.”

The board of RBS is finalising the terms of reference for an independent investigation into how the software upgrade at a processing centre in Fettes Row in Edinburgh culminated in knocking out the group’s processing systems for RBS and its banking subsidiaries, NatWest and Ulster Bank.

The main question to be asked by what bank staff have called a “massive drains-up investigation” is how such an IT failure could have occurred in modern banking.

“That is an understandable question and a question that we have got to ask as well,” said Sullivan.

“It is not as if we didn’t have back-up contingency for our IT systems. It is there. It is not as if it wasn’t audited, externally and internally. It is not as if we didn’t have disaster recovery plans. It is quite clear that they failed in this scenario.”

The bank denies the failures occurred because of cost or corner-cutting through restructuring or redundancies, or as a result a failure to invest money in new computer systems.

“You just wouldn’t do that,” said Sullivan. “This is a key process for the whole bank. We have seen the impact of it. We saw the impact of it across all three brands. You just would never risk that happening.”

The investigation will also look at making recommendations to ensure this doesn’t happen again.

“People outsource all kinds of activities in different directions. We need to understand very clearly what the organisational structure is and whether that had something to do with it, as well as the processes in play, and we need to take appropriate actions to put it in the right play,” said Sullivan.

The bank has been heavily criticised not just by tens of thousands of affected customers but by the Central Bank for the response.

This has not just been an IT problem for the bank but a public relations disaster, albeit driven by complications over trying to fix the computer problem, given the repeated inaccurate timings the bank gave on when the problem would finally be resolved.

Bernard Sheridan, the Central Bank’s director of consumer protection, told the Oireachtas finance committee this week that contingency planning had “self-evidently been appalling” and that the bank’s approach to communicating with customers had been “exasperating”.

It is too soon to say how much the IT meltdown will cost RBS and Ulster Bank, both in terms of cash and the associated cost of the damage to the bank’s reputation. It may difficult to say for some time given how the 176-year Irish franchise has been tarnished as customers threaten to move accounts to other banks once they can access their funds.

Ulster Bank has already received almost €12 billion in capital from RBS to cover the Irish bank’s disastrous adventures in property lending. The bank is undergoing a severe restructuring programme that involves 950 job cuts and aims to save €80 million in overheads.

The payroll bill arising from the computer failures will, in itself, be significant. More than 100 senior executives and managers have worked around the clock, while 2,000 branch staff have worked additional hours dealing with customers and facilitating payments.

Fortuitously for the bank’s staff, Ulster Bank pays its wages in the middle of the month so they were paid last month prior to the computer crash and the problem should be fixed in time for the next monthly payroll being processed.

This week the bank – for the first time – raised the prospect of customers inconvenienced by the meltdown being compensated beyond their out-of-pocket expenses being covered.

This coincided with the Central Bank telling that the Oireachtas committee that it would be insisting on “a comprehensive restitution plan” and that small and medium-sized businesses as well as personal customers who have been “seriously inconvenienced” would have to be compensated.

Brown said a compensation package was being worked on by Ulster Bank and that details of what would be offered would be agreed before this weekend. The offer would revolve around fees and charges for customers and probably beyond that as well, he said.

The bank has also said that compensation may be extended to customers of other banks affected by transactions not going through.

The Central Bank has not yet gone as far to discuss whether the bank would face regulatory penalties, which run to €5 million against individual institutions.

RBS and Ulster Bank has also not yet said whether any bankers would lose their jobs.

Brown bowed to political pressure last night and agreed to waive his annual bonus for 2012. RBS chief executive Stephen Hester had already said he will not take his bonus for this year due to the IT glitch. “Stephen, as the man at the top of the organisation, made the gesture straight away,” said Sullivan.

“He felt that it was the appropriate gesture. The rest of us have to focus on the right

thing for the customers. When we have done the investigation and seen all of the issues, then appropriate actions are going to get taken.”

SYSTEMS FAILURE HOW THE BANK’S COMPUTER SYSTEM BROKE DOWN

IN THE RUN-UP to Tuesday, June 19th, staff the Royal Bank of Scotland’s processing centre in Edinburgh noticed that a piece of software known as the scheduler was running more slowly than usual.

The software, called CA-7 after CA Technologies, the US company that built it, sets the sequencing for a daily batch of 20 million transactions that takes place across RBS and its subsidiaries, NatWest and Ulster Bank, after branches close every day.

Ulster Bank accounts for about a fifth of these daily transactions.

The scheduler puts about 250,000 processes in place that have to take place over any particular night and puts them in order so they can run effectively over a period of six to 12 hours.

If those processes are not in the right sequence, then the daily batch of transactions cannot be processed overnight as planned and the direct debits, standing orders and scheduled payments to and from customer accounts cannot be completed.

On June 19th, an outside service provider ran what is known as a “patch” on the scheduler to ensure that the software was running at the correct speed so that this particular problem was resolved.

However, the processing centre subsequently tried to run the scheduled batch of transactions for June 20th in parallel with the batch of June 19th and the system collapsed, corrupting the data.

This appears to have created a problem back within the scheduler, whose memory can say where it is in the 250,000 processes to be completed. The problem was exacerbated to a catastrophic level when that memory was emptied.

This meant that IT workers had to trace back to find out which processes had been completed.

“This took days and so you then had a back up of days and days of transactions and you have to put them through in order,” said RBS senior executive Chris Sullivan.

HARDEST HIT WHY WERE ULSTER BANK TRANSACTIONS THE WORST AFFECTED AT RBS?

THE WAY DAILY transactions are processed within the RBS group mirrors how the group was built up in the first place.

RBS bought NatWest in 2000 and integrated its system into RBS that same year. Ulster Bank was integrated into

the group systems on top of NatWest in 2006.

This meant that the IT problem had to be fixed for RBS and NatWest first before the Ulster Bank transactions could be processed. This happened within days of the computer glitch on June 19th.

The nature of the Ulster Bank computer systems posed more complicated problems, which caused delays in the repair work.

The Irish bank is in effect made up of two systems – Ulster Bank and First Active. The Ulster Bank system also has to deal in two currencies – sterling in Northern Ireland and euro in the Republic.

A major technical difference between the NatWest/RBS and Ulster Bank systems is that the UK system is “date-specific”, said RBS executive Chris Sullivan.

This means that if it is running scheduled transactions for June 19th, any other day’s transactions will automatically be excluded until the June 19th batch is completed.

“The Ulster Bank system is agnostic of date,” said Sullivan. “It will not automatically kick out the wrong date,

so they have to be manually extracted. The minute you get into manual processing, it gets extremely difficult.”

This meant that because Ulster Bank’s daily batches of transactions got mixed up, they had to be retraced manually and unwound before being restarted in order. As a result, the back-log of daily batches took far longer to process than normal to process – 24 hours instead of six to 12 hours.

This left RBS open to charges of treating Ulster Bank customers like second-class citizens within the wider banking group and led to Ulster Bank twice giving incorrect statements about when the problems would be rectified.

“All the statements the Ulster Bank guys have put out have been put out in good faith based on the information given to them by the IT guys,” said Sullivan.