Follow Blog via Email

Enter your email address to follow this blog and receive notifications of new posts by email.

Email Address:
Recent Posts
Archives
Categories
- 2nd screen
- Content Protection
- design
- fashion
- history
- IoT
- philosophy
- photography
- post-production
- Quality Control
- science
- security
- technology
- Uncategorized
Meta

Browsing Category Quality Control

Quality AND Quantity… or how can I have my cake and eat it too??

January 27, 2012 · by parasam

The relatively new ecosystem of large-scale distribution of digital content (movies, tv, music, books, periodicals) has brought many challenges – one of the largest being how in the world do we create and distribute so much of this stuff and keep the quality high?

Well… often we don’t… sad to say, most of the digital content that is made available online (and here I mean this in the largest sense: cable, satellite, telco as well as the internet) is of lower quality than what we enjoyed in the past with purely physical distribution. Compared to an HD Blu-ray movie, a fine photographic print, a hard-back book made by a good lithographer, a CD of music – the current crop of highly compressed video, music, etc. is but a reasonable approximation.

In many cases, the new digital versions are getting really, really good – for the cost and bandwidth used to distribute them. The convenience, low cost and flexibility of the digital distribution model has been overwhelmingly adopted by the world today – to the extent that huge physical distribution companies (such as Netflix – who only a few years ago ONLY distributed movies via DVD) now state that “We expect DVD subscribers to decline every quarter, forever.”

The bottom line is that the products delivered by digital distribution technology have been deemed “good enough” by the world at large. Yes, there is (fortunately) a continual quest for higher quality, and an appreciation of that by the consumer: nobody wants to go back to VHS quality of video – we all want HD (or as close to that as we can get). But the convenience, flexibility and low cost of streaming and downloaded material offsets, in most cases, the drop in quality compared to what can be delivered physically.

One only has to look at print circulation figures for newspapers and magazines to see how rapidly this change is taking place. Like it or not, this genie is way out of the bottle and is not going back. Distributors of books, newspapers and magazines have perhaps the most challenging adaption period ahead for two reasons:

#1: “Digital Print” can be effectively distributed at a very high quality to smartphones, tablets and laptops/PCs due to the high quality displays in use today on these devices and the relatively small data requirements to send print and smallish photos. This means that there is almost no differentiation experienced by the consumer, in terms of quality, when they move from physical to virtual consumption.

#2: With attention to the full range of possibilities of “Digital Print” media, including hyperlinking, embedded videos and graphics that are impossible (either technically or financially) to create in physical print, interactive elements, etc. – the digital versions are often more compelling than the physical versions of the same content.

The content creation and distribution model for print is moving very rapidly: it is very likely that within 5 years the vast majority of all contemporaneous print media (newspapers, magazines, popular fiction, reports, scientific journals, etc.) will be available only in digital format.

In addition to the convenience and immediacy of digital print, there are other issues to consider: newsprint requires lots of trees, water, energy, fuel to transport, etc. – all adding to the financial and ecological cost of physical distribution. The cost of manufacturing all our digital devices, and the energy to run them, is a factor in the balance, but that cost is far lower. Every morning’s paper is eventually thrown out, burned, etc. – while yesterday’s digital news is just written over in memory…

Music, movies, television, games, etc. are more difficult to distribute digitally than physically – for the equivalent quality. That is the crucial difference. High quality video and music takes a LOT of bandwidth, and even in today’s times that is still expensive. Outside of the USA, Western Europe and certain areas of AsiaPacific, high bandwidth either does not exist or is prohibitively expensive.

To offset this issue, all current video and audio content is highly compressed to save bandwidth, lower costs, and decrease transmission time. This compression lowers the quality of the product. Tremendous efforts have been successfully made over the years to drive the quality up and either keep the bandwidth the same, or even lower it. This has allowed us to experience tv on our cell phones now, a reasonable level of HD in the home via cable/satellite/telco, and thousands of songs in a postage-stamp sized device that can clip to your shirt pocket.

We assume that this trajectory will only keep improving, with eventually the quality of video and audio getting close to what physical media can offer today.

So what is the point of this observation? That a truly enormous amount of data is now being produced and distributed digitally – more than most ever envisioned. The explosion of digital consumer devices, and even more importantly the complete dependence by all business, governmental and military functions in the world on computers and interconnected data, has pushed data consumption to levels that are truly astounding.

An article in the Huffington Post in April 2011 estimated the annual data consumption of the world at 10 zettabytes per year. A zettabyte is a million petabytes. A petabyte is a million gigabytes. Ok, you know that your iPhone holds a few gigabytes… now you get the picture. And this amount of data is escalating rapidly…

Now we are getting to crux of this post: how in the world is this amount of data checked for accuracy and quality? As was shown in my last post, the consequences of bad data can be devastating, or at the least just annoying if your movie has bad pictures or distorted sound. It may seem obvious once you see the very large amounts of data created and moved every year (or to put it another way: 32,000 gigabytes per second is the current rate of data consumption on the planet Earth) – most of the data (whether financial data, movies, music, etc.) is simply not checked at all.

We rely – perhaps more than we should – on the accuracy of the digital systems themselves to propagate this data correctly. Most of the data that is turned loose into our digital world had some level of quality check at some point – an article was proof-read before publishing; a movie was put through a quality control (QC) process; photos were examined, etc. However, the very nature of our fragmented and differentiated digital distribution system requires constant and frequent transformation of data.

Whether a movie was transcoded (converted from one digital format to another); financial data moved from one database to another; music encoded from a CD to an MP3 file – all these data transformations are often done automatically and then sent on their way – with no further checks on accuracy. This is not completely due to a blatant disregard for quality – it’s just the laws of physics: there is simply no way to check this much data!

This is an insiduous problem – no one sat back 20 years ago and said, “We’re going to build a digital universe that will basically become unmanageable in terms of quality control.” But here we are…

On the whole, the system as a whole works amazingly well. The amount of QC done on the front end (when the data is first created), coupled with the relatively high accuracy of digital systems, has allowed the ecosystem to work rather well – on the average. The issue is that individual errors really do matter – see my last post for some notable examples. The National Academy of Science reported in 2006 that the total error rate for dispensation of prescription medicine that caused injury or death in the United States was 0.5% – now that seems small until you do the math. In 2006 there were 300 million people in the US, so there 1.5 million people affected by these errors. If you were affected by a data error that caused injury or death you might look a bit differently on the issue of quality control in digital systems.

So the issue boils down to this: is it possible to have a system (the ‘internet’ for lack of a better description – although here I really mean the conglomerated digital communications system for all forms of digital data) that can offer BOTH high quantities of data as well as ensure a reasonably high level of quality?

The short answer is yes, this is possible. It takes effort, attention and good process. I’ve coined a term for this – TS2 – (you can’t be a tekkie if you don’t invent acronyms… 🙂 meaning “Trusted Source / Trusted System”. At the highest level, if a data source (movie, article, financial data, etc.) is tested to a level of certainty before being introduced to a digital distribution system AND all of the workflows, transformative processes, etc. themselves are tested, then it can be mathematically proven that the end distributed result will have a high level of accuracy – with no further QC required.

This is not a trivial issue, as the hard part is testing and control of the system elements that touch the data as it travels to its destination. This cannot always be accomplished, but in many cases it is possible, and a well-managed approach can greatly increase the reliablity of the delivered content. A rigorous ‘change control’ process must be introduced, with any sugsequent changes to the system being well tested in a lab environment before turned loose in the ‘real world.’ Some examples in the previous post show what happens if this is not done….

So, some food for thought… or to put it another way, it is possible to eat your digital cake!

… but to really screw it up you need a computer…

January 26, 2012 · by parasam

We’ve all heard this one, but I wanted to share a few ‘horror stories’ with you as a prequel to a blog I will post shortly on the incredible challenges that our new world of digital content puts to us in terms of quality control of content. Today we are swimming in an ocean of content: movies, books, music, financial information, e-mail, etc. We collectively put an alarming level of trust in all this digital information – we mostly just assume it’s correct. But what if it is not?

The following ‘disasters’ are all true. Completely. They were originally compiled by Andrew Brandt of the IDG News Service in October 2008 as a commentary on the importance of good QA (Quality Assurance) teams in the IT industry.

<begin article>

Stupid QA tricks: Colossal testing oversights

What do you get when you add the human propensity to screw stuff up to the building of large-scale IT systems? What the military calls the force-multiplier effect — and the need for a cadre of top-notch QA engineers.

After all, if left unchecked, one person’s slip of the mouse can quickly turn into weeks of lost work, months of missing e-mails, or, in the worst cases, whole companies going bankrupt. And with IT infused in every aspect of business, doesn’t it pay to take quality assurance seriously?

Let’s face it. Everybody makes mistakes. Users, managers, admins – no one is immune to the colossally stupid IT miscue now and again. But when a fat-fingered script or a poor security practice goes unnoticed all the way through development and into production, the unsung heroes of IT, the QA engineers, take a very embarrassing center stage. It may seem cliché, but your IT development chain is only as strong as its weakest link. You better hope that weakest link is your QA team, as these five colossal testing oversights attest.

Code “typo” hides high risk of credit derivative

Testing oversight: Bug in financial risk assessment code

Consequence: Institutional investors are led to believe high-risk credit derivatives are highly desirable AAA-rated investments.

Here’s the kind of story we’re not hearing much about these days despite our present economic turmoil.

According to a report published in May 2008 in the Financial Times, Moody’s inadvertently overrated about $4 billion worth of debt instruments known as CPDOs (constant proportion debt obligations), due to a bug in its software. The company, which rates a wide variety of government bonds and obligation debts, underplayed the level of risk to investors as a result of the bug, a glitch that may have contributed to substantial investment losses among today’s reeling financial institutions. CPDOs were sold to large institutional investors beginning in 2006, during the height of the financial bubble, with promises of high returns — nearly 10 times those of prime European mortgage-backed bonds — at very little risk.

Internal Moody’s documents reviewed by reporters from the Financial Times, however, indicated that senior staff at Moody’s were aware in February 2007 that a glitch in some computer models rated CPDOs as much as 3.5 levels higher in the Moody’s metric than they should have been. As a result, Moody’s advertised CPDOs as significantly less risky than they actually were until the ratings were corrected in early 2008.

Institutional investors typically rely on ratings from at least two companies before they put significant money into a new financial product. Standard & Poor’s had previously rated CPDOs with its highest AAA rating, and stood by its evaluation. Moody’s AAA rating provided the critical second rating that spurred investors to begin purchasing CPDOs. But other bond-ratings firms didn’t rate CPDO transactions as highly; John Schiavetta, head of global structured credit at Derivative Fitch in New York, was quoted in the Financial Times in April 2007, saying, “We think the first generation of CPDO transactions are overrated.”

Among the U.S.-based financial institutions that put together CPDO portfolios, trying to cash in on what, in late 2006, seemed to be a gold rush in investments, were Lehman Brothers, Merrill Lynch, and J.P. Morgan. When first reported this past May, the Financial Times story described the bug in Moody’s rating system as “nothing more than a mathematical typo — a small glitch in a line of computer code.” But this glitch may have contributed in some measure to the disastrous financial situation all around us.

It’s kind of hard to come up with a snarky one-liner for a foul-up like that.

Testing tip: When testing something as critical as this, run commonsense trials: Throw variations of data at the formula, and make sure you get the expected result each time. You also have to audit your code periodically with an outside firm, to ensure that a vested insider hasn’t “accidentally” inserted a mathematical error that nets the insider millions. There’s no indication that such an inside job happened in this case, but such a scenario isn’t so far-fetched that it’s beyond the realm of possibility.

Sorry, Mr. Smith, you have cancer. Oh, you’re not Mr. Smith?

Testing oversight: Mismatched contact information in insurer’s customer database

Consequence: Blue Cross/Blue Shield sends 202,000 printed letters containing patient information and Social Security numbers to the wrong patients.

Of course, it sounded like a good idea at the time: Georgia’s largest health insurance company, with 3.1 million members, designed a system that would send patients information about how each visit was covered by their insurance. The EOB (explanation of benefits) letters would provide sensitive patient information, including payment and coverage details, as well as the name of the doctor or medical facility visited and the patient’s insurance ID number.

Most insurance companies send out EOBs after people receive medical treatment or visit a doctor, but the Georgia Blue Cross/Blue Shield system so muddled up its medical data management functionality that its members were sent other members’ sensitive patient information. According to The Atlanta Journal-Constitution, registered nurse Rhonda Bloschock, who is covered by Blue Cross/Blue Shield, received an envelope containing EOB letters for nine different people. Georgia State Insurance Commissioner John Oxendine described the gaffe to WALB news as “the worst breach of healthcare privacy I’ve seen in my 14 years in office.”

As for the roughly 6 percent of Georgia Blue Cross/Blue Shield customers who were affected, I’m sure they will be heartened by the statement provided by spokeswoman Cindy Sanders, who described the event as an isolated incident that “will not impact future EOB mailings.” It’s a mantra Georgia Blue Cross/Blue Shield customers can keep repeating to themselves for years as they constantly check their credit reports for signs of identity theft.

Testing tip: Merging databases is always tricky business, so it’s important to run a number of tests using a large sample set to ensure fields don’t get muddled together. The data set you use for testing should be large enough to stress the system as a normal database would, and the test data should be formatted in such a way to make it painfully obvious if anything is out of place. Never use the production database as your test set.

Where free shipping really, really isn’t free

Testing oversight: Widespread glitches in Web site upgrade

Consequence: Clothier J. Crew suffers huge financial losses and widespread customer dissatisfaction in wake of “upgrade” that blocks and fouls up customer orders for a month.

On June 28, 2008, engineers took down the Web site for clothes retailer J. Crew for 24 hours to perform an upgrade. In terms of the results of this effort, one might argue that the site did not in fact come back online for several weeks, even though it was still serving pages.

The company’s 10-Q filing summarized the problems: “During the second quarter of fiscal 2008 we implemented certain direct channel systems upgrades which impacted our ability to capture, process, ship and service customer orders.” That’s because the upgrade essentially prevented site visitors from doing anything other than look at photos of fashionable clothes.

Among the problems reported by customers was this whopper: A man who ordered some polo shirts received, instead, three child-size shirts and a bill for $44.97 for the shirts, plus $9,208.50 for shipping. And before you ask, no, they weren’t hand-delivered by a princess in an enchanted coach.

As a result, the company temporarily shut down e-mail marketing campaigns designed to drive business to the Web site. It also had to offer discounts, refunds, and other concessions to customers who couldn’t correct orders conducted online or who received partial or wrong orders.

But the biggest story is how the Web site upgrade affected the company’s bottom line. In a conference call with investors in August, CFO James Scully said, “The direct system upgrades did impact our second-quarter results more than we had anticipated and will also impact our third-quarter and fiscal-year results,” according to a transcript of the call.

Ouch.

Testing tip: When your company’s bottom line depends on the availability of your Web site, there’s no excuse for not running a thorough internal trial to probe the functionality of the entire site before you throw that update live to the world. Bring everyone on the Web team into the office, buy a bunch of pizzas, and tell them to click absolutely everything. And keep full backups of your old site’s front and back end, just in case you do somehow push a broken site update live and need to revert to save your company from unmitigated disaster.

Department of Corrections database inadvertently steps into the “user-generated” generation

Testing oversight: Trusted anonymous access to government database

Consequence: Database queries in URLs permit anyone with passing knowledge of SQL to pull down full personal information of anyone affiliated with the Oklahoma Department of Corrections, including prisoners, guards, and officers.

Anyone who’s ever been an employee of the Oklahoma prison system or an unwilling guest of the state now has an additional issue to worry about: identity theft. Thanks to a poorly programmed Web page designed to provide access to the Sexual and Violent Offender Registry, Web visitors were able to gain complete access to the entire Department of Corrections database.

Among the data stored in the database were names, addresses, Social Security numbers, medical histories, and e-mail addresses. But the problem was far worse that that: Anyone who knew how to craft SQL queries could have actually added information to the database.

Got an annoying neighbor who mows his lawn too early on a Sunday? How about a roommate who plays his music too loud, late into the night? Annoying ex-boyfriend or ex-girlfriend? Why not add them to the Sexual and Violent Offender Registry and watch them get rejected from jobs and be dragged off to the pokey after a routine traffic stop?

To add insult to injury, when Alex Papadimoulis, editor of dailywtf.com, alerted Oklahoma corrections officials about the security problem, they fixed it immediately — by making the SQL query case-sensitive.

So instead of adding “social_security_number” to the query string that retrieves that bit of information, it only worked if you used “Social_security_number.” Genius, huh? Nobody would ever have thought of that.

The database-on-a-Web-site issue is only a slice of the problems Oklahoma’s Department of Corrections faces when it comes to IT. An audit of the department published at the end of 2007 explains that the OMS (Offender Management System) is on the brink of collapse. “The current software is so out of date that it cannot reside on newer computer equipment and is maintained on an antiquated hardware platform that is becoming increasingly difficult to repair. A recent malfunction of this server took OMS down for over a full day while replacement parts were located. If this hardware ultimately fails, the agency will lose its most vital technology resource in the day-to-day management of the offender population.”

Testing tip: When you’re building an interface to a database that contains the sensitive data of hundreds or thousands of people, there’s no excuse for taking the least-expensive-coder route. Coding security into a Web application takes a programmer with practical experience. In this case, that didn’t happen. The money you spend on a secure site architecture at the beginning may save you from major embarrassment later, after some kid breaks your security model in five minutes. Remember, “security through obscurity” provides no security at all.

Busted big-time — by the bank

Testing oversight: Contact fields transposed during financial database migration

Consequence: Financial services firm sends detailed “secret” savings and charge card records made for mistresses to customers’ wives.

It’s hard to get away with an affair when the bank won’t play along. That’s what some high-roller clients of an unnamed financial services firm learned when the firm sent statements containing full details of account holders’ assets to their home addresses.

Although that might not sound like a recipe for disaster, this particular firm — which requires a $10 million minimum deposit to open an account — is in the business of providing, shall we say, a financial masquerade for those who wish to sock away cash they don’t want certain members of their marriage to know about. Customers who desire this kind of service typically had one (somewhat abridged) statement mailed home, and another, more detailed (read: incriminating) statement mailed to another address.

When the firm instituted a major upgrade to its customer-facing portal, however, a database migration error slipped through the cracks. The customer’s home address was used for the full, unabridged account statements. The nature and character of the discussions between account holder and spouse regarding charges for hotel rooms, expensive jewelry, flowers, and dinners are left as an exercise for the imagination. According to a source inside the company, the firm lost a number of wealthy clients and nearly $450 million in managed assets as a result of the flub. But the real winners in this case, apparently, were the divorce lawyers.

Testing tip: In this case, it seems like the engineers who designed the upgrade didn’t fully understand the ramifications of what they were doing. The bank executives who maintain this house of cards were ultimately at fault. Communicate the intricacies of your customers’ business relationships to your site designers and follow through with continuous oversight to ensure clients’ dirty laundry, err, sensitive data out of public view.

<end article>

While the above examples are certainly large and adversely affected many people’s lives and finances, these are a very, very small tip of a really, really large iceberg. Our digital world is the modern-day Titanic, steaming ahead while we party to the tunes in our earbuds and believe the 3D we see is real…

We humans like to believe what we see, what we’re told, what we feel. Most of us are trusting of the information we receive every day with little consideration of its accuracy. Only when something doesn’t work, or some calamity occurs due to incorrect data, do we stop to ask ourselves, “Is that right?”

I’ll conclude these thoughts in my next posting, but I will leave you with a clue that will provide security and sanity in the face of potential digital uncertainty: use commmon sense.