Cashless Society: Not so Fast Wells Fargo!

Well this week, we were treated to the Great Banking Outage of 2007 sponsored by Wells Fargo and most of the customers were treated to a day of 100% illiquidity. While it has been under reported, this has to rate as one of the top major corporate e-meltdowns of the year, perhaps the last few years. The worlds 5th largest financial institution just decided to go offline for a day. Now, in the old days, this wouldn’t have been much of a big deal. After all, back in the “good ole days” you used credit cards mostly for large purchases and business dinners and went to the bank to cash a check when you needed money. However in today’s near cashless society, being without access to your bank is somewhat more serous. No purchases of any kind are really possible, unless you are like me and still carry that ugly green stuff around with you. No gas for your car, no lunch or dinner, no services of any kind for you, check card boy! And god forbid you were trying to do something more serous like closing on a house. Apparently the outage was firm wide and affected all their divisions including their home mortgage division. According to the posts on the internet, numerous people were not able to finalize their home purchases, because Wells Fargo refused to fund their loans during the outage. Their lovely IT chieftain person, Avid Modjtabai claimed that they were up and running within an hour and half of the initial outage Sunday, with limited outages after that. I call BS on him, being a customer both commercial and person of the bank. The way I found about this little “snafu” was that I went to the bank to do a wire transfer and was informed that the entire bank was down, city wide (turns out nationwide) and they couldn’t process any transactions, even accept deposits. They were only able to cash checks for small amounts of cash if you could prove you had an account there. According to the guy I talked to, they were basically operating in the dark, blind with no access to information. According to the company spokesperson, the problem occurred during some “routine” maintenance. Now, I’m not trying to start a conspiracy theory here or saying they got hacked or anything but my IT “Spidey” senses are tingling over this one. Something is not right here and we are not getting the full story. Either they guilty of vastly gross incompetence of the highest order in terms of their IT design or something else happened that they don’t want to talk about. Whether thats a hack, release of information, or something else, I don’t know. But something just doesn’t smell right there. When you are bank of even near that magnitude, you have redundant systems upon redundant systems that have redundant systems. I know banking IT systems and I can tell you, that banks of even regional stature usually have continuous computing sites, multiple computer centers, on-line real time backups and so on. The way they make it sound, we are supposed to believe that Well Fargo is running their entire world wide banking business off a single non-redundant system vulnerable to routine maintenance failures?? What are they running on, an old rusty main frame in the CIO’s basement?? No, I call him out on BS. If there system was this vulnerable to a single point of failure, they never would have passed their IT audit or federal examination which every bank has to have at least once per year. No, something else is happening here and it just hasn’t come out yet. Now, maybe these guys did design the world’s worst banking system from a disaster recovery standpoint. After all, they did have a similar outage a few years ago, but it only took out half the country, not the whole thing. I mean a first year IT student can tell you you can’t have your whole world wide systems relying on a single system, a single software instance. If you are doing a software upgrade, you keep your backup systems on the old version and ready to switch over. Yes, of course small software bugs can cause huge system outages. We’ve seen it before in Google, Yahoo, Microsoft’s website. But that didn’t affect a huge chunk of the American banking public. That’s why you have multiple computer centers, that’s why you have roll-back capability, that’s why you spare parts. This is IT 101. So if it turns out that I’m wrong and it is just a case of “Opps!”, then not only should heads roll at Wells, starting with Chief Information Doffus Avid but also at the OCC who supposedly examined and approved their IT Systems. I mean I wouldn’t pass local community bank with an operation like that. Who was the Federal Bozo who signed off on this Rube Goldberg IT system? Another gem of wisdom from our favorite IT guy about his customers. “We want to assure them that all data about their accounts has remained safe and secure throughout this disruption.” Hey pal, read up on the definition of IT security definition in your college textbook. IT Security 101 is defines information security as keeping maintaining confidentiality, integrity and availability. If I can’t get to my funds and your employees cant access information on them, then they aren’t much good to me, as “secure” as they might be. So this bozo needs to be brought to task, his boss needs to be brought to task and there needs to be some serious questions asked of the examining agency. Just a final note on this “Disaster”, mostly of a PR nature for Wells Fargo. Why didn’t they just go ahead and approve all those ATM and debit card requests coming through during the outage? What would have been their maximum exposure, especially since most of these are going to be for a small dollar value from legitimate card holders? The good PR value would have almost eclipsed the negative fallout of this screw-up. Compare that to the likely loss of millions in bounced check fees reimbursed, interest lost, etc. What’s a days worth of lost business worth for a concern like Wells Fargo? Well using simple math, 1/365th of their annual 2006 revenue of 46 billion is 131 Million Dollars. That will pay for a few redundant systems, wont it?. And that’s not considering cost of other external charges, lost customers and lawsuits that are likely to be filed in the aftermath to compensate the “victims”. Lawyers, you gotta love em. At least when they go after incompetent goliaths like Wells.So either these guys get the bozo IT design award of the decade, or something else is afoot. Either way, its stinks! More on the implications of our non-perfect cashless society next time