We’ve all heard this one, but I wanted to share a few ‘horror stories’ with you as a prequel to a blog I will post shortly on the incredible challenges that our new world of digital content puts to us in terms of quality control of content. Today we are swimming in an ocean of content: movies, books, music, financial information, e-mail, etc. We collectively put an alarming level of trust in all this digital information – we mostly just assume it’s correct. But what if it is not?
The following ‘disasters’ are all true. Completely. They were originally compiled by Andrew Brandt of the IDG News Service in October 2008 as a commentary on the importance of good QA (Quality Assurance) teams in the IT industry.
<begin article>
Stupid QA tricks: Colossal testing oversights
What do you get when you add the human propensity to screw stuff up to the building of large-scale IT systems? What the military calls the force-multiplier effect — and the need for a cadre of top-notch QA engineers.
After all, if left unchecked, one person’s slip of the mouse can quickly turn into weeks of lost work, months of missing e-mails, or, in the worst cases, whole companies going bankrupt. And with IT infused in every aspect of business, doesn’t it pay to take quality assurance seriously?
Let’s face it. Everybody makes mistakes. Users, managers, admins – no one is immune to the colossally stupid IT miscue now and again. But when a fat-fingered script or a poor security practice goes unnoticed all the way through development and into production, the unsung heroes of IT, the QA engineers, take a very embarrassing center stage. It may seem cliché, but your IT development chain is only as strong as its weakest link. You better hope that weakest link is your QA team, as these five colossal testing oversights attest.
Code “typo” hides high risk of credit derivative
Testing oversight: Bug in financial risk assessment code
Consequence: Institutional investors are led to believe high-risk credit derivatives are highly desirable AAA-rated investments.
Here’s the kind of story we’re not hearing much about these days despite our present economic turmoil.
According to a report published in May 2008 in the Financial Times, Moody’s inadvertently overrated about $4 billion worth of debt instruments known as CPDOs (constant proportion debt obligations), due to a bug in its software. The company, which rates a wide variety of government bonds and obligation debts, underplayed the level of risk to investors as a result of the bug, a glitch that may have contributed to substantial investment losses among today’s reeling financial institutions. CPDOs were sold to large institutional investors beginning in 2006, during the height of the financial bubble, with promises of high returns — nearly 10 times those of prime European mortgage-backed bonds — at very little risk.
Internal Moody’s documents reviewed by reporters from the Financial Times, however, indicated that senior staff at Moody’s were aware in February 2007 that a glitch in some computer models rated CPDOs as much as 3.5 levels higher in the Moody’s metric than they should have been. As a result, Moody’s advertised CPDOs as significantly less risky than they actually were until the ratings were corrected in early 2008.
Institutional investors typically rely on ratings from at least two companies before they put significant money into a new financial product. Standard & Poor’s had previously rated CPDOs with its highest AAA rating, and stood by its evaluation. Moody’s AAA rating provided the critical second rating that spurred investors to begin purchasing CPDOs. But other bond-ratings firms didn’t rate CPDO transactions as highly; John Schiavetta, head of global structured credit at Derivative Fitch in New York, was quoted in the Financial Times in April 2007, saying, “We think the first generation of CPDO transactions are overrated.”
Among the U.S.-based financial institutions that put together CPDO portfolios, trying to cash in on what, in late 2006, seemed to be a gold rush in investments, were Lehman Brothers, Merrill Lynch, and J.P. Morgan. When first reported this past May, the Financial Times story described the bug in Moody’s rating system as “nothing more than a mathematical typo — a small glitch in a line of computer code.” But this glitch may have contributed in some measure to the disastrous financial situation all around us.
It’s kind of hard to come up with a snarky one-liner for a foul-up like that.
Testing tip: When testing something as critical as this, run commonsense trials: Throw variations of data at the formula, and make sure you get the expected result each time. You also have to audit your code periodically with an outside firm, to ensure that a vested insider hasn’t “accidentally” inserted a mathematical error that nets the insider millions. There’s no indication that such an inside job happened in this case, but such a scenario isn’t so far-fetched that it’s beyond the realm of possibility.
Sorry, Mr. Smith, you have cancer. Oh, you’re not Mr. Smith?
Testing oversight: Mismatched contact information in insurer’s customer database
Consequence: Blue Cross/Blue Shield sends 202,000 printed letters containing patient information and Social Security numbers to the wrong patients.
Of course, it sounded like a good idea at the time: Georgia’s largest health insurance company, with 3.1 million members, designed a system that would send patients information about how each visit was covered by their insurance. The EOB (explanation of benefits) letters would provide sensitive patient information, including payment and coverage details, as well as the name of the doctor or medical facility visited and the patient’s insurance ID number.
Most insurance companies send out EOBs after people receive medical treatment or visit a doctor, but the Georgia Blue Cross/Blue Shield system so muddled up its medical data management functionality that its members were sent other members’ sensitive patient information. According to The Atlanta Journal-Constitution, registered nurse Rhonda Bloschock, who is covered by Blue Cross/Blue Shield, received an envelope containing EOB letters for nine different people. Georgia State Insurance Commissioner John Oxendine described the gaffe to WALB news as “the worst breach of healthcare privacy I’ve seen in my 14 years in office.”
As for the roughly 6 percent of Georgia Blue Cross/Blue Shield customers who were affected, I’m sure they will be heartened by the statement provided by spokeswoman Cindy Sanders, who described the event as an isolated incident that “will not impact future EOB mailings.” It’s a mantra Georgia Blue Cross/Blue Shield customers can keep repeating to themselves for years as they constantly check their credit reports for signs of identity theft.
Testing tip: Merging databases is always tricky business, so it’s important to run a number of tests using a large sample set to ensure fields don’t get muddled together. The data set you use for testing should be large enough to stress the system as a normal database would, and the test data should be formatted in such a way to make it painfully obvious if anything is out of place. Never use the production database as your test set.
Where free shipping really, really isn’t free
Testing oversight: Widespread glitches in Web site upgrade
Consequence: Clothier J. Crew suffers huge financial losses and widespread customer dissatisfaction in wake of “upgrade” that blocks and fouls up customer orders for a month.
On June 28, 2008, engineers took down the Web site for clothes retailer J. Crew for 24 hours to perform an upgrade. In terms of the results of this effort, one might argue that the site did not in fact come back online for several weeks, even though it was still serving pages.
The company’s 10-Q filing summarized the problems: “During the second quarter of fiscal 2008 we implemented certain direct channel systems upgrades which impacted our ability to capture, process, ship and service customer orders.” That’s because the upgrade essentially prevented site visitors from doing anything other than look at photos of fashionable clothes.
Among the problems reported by customers was this whopper: A man who ordered some polo shirts received, instead, three child-size shirts and a bill for $44.97 for the shirts, plus $9,208.50 for shipping. And before you ask, no, they weren’t hand-delivered by a princess in an enchanted coach.
As a result, the company temporarily shut down e-mail marketing campaigns designed to drive business to the Web site. It also had to offer discounts, refunds, and other concessions to customers who couldn’t correct orders conducted online or who received partial or wrong orders.
But the biggest story is how the Web site upgrade affected the company’s bottom line. In a conference call with investors in August, CFO James Scully said, “The direct system upgrades did impact our second-quarter results more than we had anticipated and will also impact our third-quarter and fiscal-year results,” according to a transcript of the call.
Ouch.
Testing tip: When your company’s bottom line depends on the availability of your Web site, there’s no excuse for not running a thorough internal trial to probe the functionality of the entire site before you throw that update live to the world. Bring everyone on the Web team into the office, buy a bunch of pizzas, and tell them to click absolutely everything. And keep full backups of your old site’s front and back end, just in case you do somehow push a broken site update live and need to revert to save your company from unmitigated disaster.
Department of Corrections database inadvertently steps into the “user-generated” generation
Testing oversight: Trusted anonymous access to government database
Consequence: Database queries in URLs permit anyone with passing knowledge of SQL to pull down full personal information of anyone affiliated with the Oklahoma Department of Corrections, including prisoners, guards, and officers.
Anyone who’s ever been an employee of the Oklahoma prison system or an unwilling guest of the state now has an additional issue to worry about: identity theft. Thanks to a poorly programmed Web page designed to provide access to the Sexual and Violent Offender Registry, Web visitors were able to gain complete access to the entire Department of Corrections database.
Among the data stored in the database were names, addresses, Social Security numbers, medical histories, and e-mail addresses. But the problem was far worse that that: Anyone who knew how to craft SQL queries could have actually added information to the database.
Got an annoying neighbor who mows his lawn too early on a Sunday? How about a roommate who plays his music too loud, late into the night? Annoying ex-boyfriend or ex-girlfriend? Why not add them to the Sexual and Violent Offender Registry and watch them get rejected from jobs and be dragged off to the pokey after a routine traffic stop?
To add insult to injury, when Alex Papadimoulis, editor of dailywtf.com, alerted Oklahoma corrections officials about the security problem, they fixed it immediately — by making the SQL query case-sensitive.
So instead of adding “social_security_number” to the query string that retrieves that bit of information, it only worked if you used “Social_security_number.” Genius, huh? Nobody would ever have thought of that.
The database-on-a-Web-site issue is only a slice of the problems Oklahoma’s Department of Corrections faces when it comes to IT. An audit of the department published at the end of 2007 explains that the OMS (Offender Management System) is on the brink of collapse. “The current software is so out of date that it cannot reside on newer computer equipment and is maintained on an antiquated hardware platform that is becoming increasingly difficult to repair. A recent malfunction of this server took OMS down for over a full day while replacement parts were located. If this hardware ultimately fails, the agency will lose its most vital technology resource in the day-to-day management of the offender population.”
Testing tip: When you’re building an interface to a database that contains the sensitive data of hundreds or thousands of people, there’s no excuse for taking the least-expensive-coder route. Coding security into a Web application takes a programmer with practical experience. In this case, that didn’t happen. The money you spend on a secure site architecture at the beginning may save you from major embarrassment later, after some kid breaks your security model in five minutes. Remember, “security through obscurity” provides no security at all.
Busted big-time — by the bank
Testing oversight: Contact fields transposed during financial database migration
Consequence: Financial services firm sends detailed “secret” savings and charge card records made for mistresses to customers’ wives.
It’s hard to get away with an affair when the bank won’t play along. That’s what some high-roller clients of an unnamed financial services firm learned when the firm sent statements containing full details of account holders’ assets to their home addresses.
Although that might not sound like a recipe for disaster, this particular firm — which requires a $10 million minimum deposit to open an account — is in the business of providing, shall we say, a financial masquerade for those who wish to sock away cash they don’t want certain members of their marriage to know about. Customers who desire this kind of service typically had one (somewhat abridged) statement mailed home, and another, more detailed (read: incriminating) statement mailed to another address.
When the firm instituted a major upgrade to its customer-facing portal, however, a database migration error slipped through the cracks. The customer’s home address was used for the full, unabridged account statements. The nature and character of the discussions between account holder and spouse regarding charges for hotel rooms, expensive jewelry, flowers, and dinners are left as an exercise for the imagination. According to a source inside the company, the firm lost a number of wealthy clients and nearly $450 million in managed assets as a result of the flub. But the real winners in this case, apparently, were the divorce lawyers.
Testing tip: In this case, it seems like the engineers who designed the upgrade didn’t fully understand the ramifications of what they were doing. The bank executives who maintain this house of cards were ultimately at fault. Communicate the intricacies of your customers’ business relationships to your site designers and follow through with continuous oversight to ensure clients’ dirty laundry, err, sensitive data out of public view.
<end article>
While the above examples are certainly large and adversely affected many people’s lives and finances, these are a very, very small tip of a really, really large iceberg. Our digital world is the modern-day Titanic, steaming ahead while we party to the tunes in our earbuds and believe the 3D we see is real…
We humans like to believe what we see, what we’re told, what we feel. Most of us are trusting of the information we receive every day with little consideration of its accuracy. Only when something doesn’t work, or some calamity occurs due to incorrect data, do we stop to ask ourselves, “Is that right?”
I’ll conclude these thoughts in my next posting, but I will leave you with a clue that will provide security and sanity in the face of potential digital uncertainty: use commmon sense.
Parasam 