The relatively new ecosystem of large-scale distribution of digital content (movies, tv, music, books, periodicals) has brought many challenges – one of the largest being how in the world do we create and distribute so much of this stuff and keep the quality high?
Well… often we don’t… sad to say, most of the digital content that is made available online (and here I mean this in the largest sense: cable, satellite, telco as well as the internet) is of lower quality than what we enjoyed in the past with purely physical distribution. Compared to an HD Blu-ray movie, a fine photographic print, a hard-back book made by a good lithographer, a CD of music – the current crop of highly compressed video, music, etc. is but a reasonable approximation.
In many cases, the new digital versions are getting really, really good – for the cost and bandwidth used to distribute them. The convenience, low cost and flexibility of the digital distribution model has been overwhelmingly adopted by the world today – to the extent that huge physical distribution companies (such as Netflix – who only a few years ago ONLY distributed movies via DVD) now state that “We expect DVD subscribers to decline every quarter, forever.”
The bottom line is that the products delivered by digital distribution technology have been deemed “good enough” by the world at large. Yes, there is (fortunately) a continual quest for higher quality, and an appreciation of that by the consumer: nobody wants to go back to VHS quality of video – we all want HD (or as close to that as we can get). But the convenience, flexibility and low cost of streaming and downloaded material offsets, in most cases, the drop in quality compared to what can be delivered physically.
One only has to look at print circulation figures for newspapers and magazines to see how rapidly this change is taking place. Like it or not, this genie is way out of the bottle and is not going back. Distributors of books, newspapers and magazines have perhaps the most challenging adaption period ahead for two reasons:
#1: “Digital Print” can be effectively distributed at a very high quality to smartphones, tablets and laptops/PCs due to the high quality displays in use today on these devices and the relatively small data requirements to send print and smallish photos. This means that there is almost no differentiation experienced by the consumer, in terms of quality, when they move from physical to virtual consumption.
#2: With attention to the full range of possibilities of “Digital Print” media, including hyperlinking, embedded videos and graphics that are impossible (either technically or financially) to create in physical print, interactive elements, etc. – the digital versions are often more compelling than the physical versions of the same content.
The content creation and distribution model for print is moving very rapidly: it is very likely that within 5 years the vast majority of all contemporaneous print media (newspapers, magazines, popular fiction, reports, scientific journals, etc.) will be available only in digital format.
In addition to the convenience and immediacy of digital print, there are other issues to consider: newsprint requires lots of trees, water, energy, fuel to transport, etc. – all adding to the financial and ecological cost of physical distribution. The cost of manufacturing all our digital devices, and the energy to run them, is a factor in the balance, but that cost is far lower. Every morning’s paper is eventually thrown out, burned, etc. – while yesterday’s digital news is just written over in memory…
Music, movies, television, games, etc. are more difficult to distribute digitally than physically – for the equivalent quality. That is the crucial difference. High quality video and music takes a LOT of bandwidth, and even in today’s times that is still expensive. Outside of the USA, Western Europe and certain areas of AsiaPacific, high bandwidth either does not exist or is prohibitively expensive.
To offset this issue, all current video and audio content is highly compressed to save bandwidth, lower costs, and decrease transmission time. This compression lowers the quality of the product. Tremendous efforts have been successfully made over the years to drive the quality up and either keep the bandwidth the same, or even lower it. This has allowed us to experience tv on our cell phones now, a reasonable level of HD in the home via cable/satellite/telco, and thousands of songs in a postage-stamp sized device that can clip to your shirt pocket.
We assume that this trajectory will only keep improving, with eventually the quality of video and audio getting close to what physical media can offer today.
So what is the point of this observation? That a truly enormous amount of data is now being produced and distributed digitally – more than most ever envisioned. The explosion of digital consumer devices, and even more importantly the complete dependence by all business, governmental and military functions in the world on computers and interconnected data, has pushed data consumption to levels that are truly astounding.
An article in the Huffington Post in April 2011 estimated the annual data consumption of the world at 10 zettabytes per year. A zettabyte is a million petabytes. A petabyte is a million gigabytes. Ok, you know that your iPhone holds a few gigabytes… now you get the picture. And this amount of data is escalating rapidly…
Now we are getting to crux of this post: how in the world is this amount of data checked for accuracy and quality? As was shown in my last post, the consequences of bad data can be devastating, or at the least just annoying if your movie has bad pictures or distorted sound. It may seem obvious once you see the very large amounts of data created and moved every year (or to put it another way: 32,000 gigabytes per second is the current rate of data consumption on the planet Earth) – most of the data (whether financial data, movies, music, etc.) is simply not checked at all.
We rely – perhaps more than we should – on the accuracy of the digital systems themselves to propagate this data correctly. Most of the data that is turned loose into our digital world had some level of quality check at some point – an article was proof-read before publishing; a movie was put through a quality control (QC) process; photos were examined, etc. However, the very nature of our fragmented and differentiated digital distribution system requires constant and frequent transformation of data.
Whether a movie was transcoded (converted from one digital format to another); financial data moved from one database to another; music encoded from a CD to an MP3 file – all these data transformations are often done automatically and then sent on their way – with no further checks on accuracy. This is not completely due to a blatant disregard for quality – it’s just the laws of physics: there is simply no way to check this much data!
This is an insiduous problem – no one sat back 20 years ago and said, “We’re going to build a digital universe that will basically become unmanageable in terms of quality control.” But here we are…
On the whole, the system as a whole works amazingly well. The amount of QC done on the front end (when the data is first created), coupled with the relatively high accuracy of digital systems, has allowed the ecosystem to work rather well – on the average. The issue is that individual errors really do matter – see my last post for some notable examples. The National Academy of Science reported in 2006 that the total error rate for dispensation of prescription medicine that caused injury or death in the United States was 0.5% – now that seems small until you do the math. In 2006 there were 300 million people in the US, so there 1.5 million people affected by these errors. If you were affected by a data error that caused injury or death you might look a bit differently on the issue of quality control in digital systems.
So the issue boils down to this: is it possible to have a system (the ‘internet’ for lack of a better description – although here I really mean the conglomerated digital communications system for all forms of digital data) that can offer BOTH high quantities of data as well as ensure a reasonably high level of quality?
The short answer is yes, this is possible. It takes effort, attention and good process. I’ve coined a term for this – TS2 – (you can’t be a tekkie if you don’t invent acronyms… 🙂 meaning “Trusted Source / Trusted System”. At the highest level, if a data source (movie, article, financial data, etc.) is tested to a level of certainty before being introduced to a digital distribution system AND all of the workflows, transformative processes, etc. themselves are tested, then it can be mathematically proven that the end distributed result will have a high level of accuracy – with no further QC required.
This is not a trivial issue, as the hard part is testing and control of the system elements that touch the data as it travels to its destination. This cannot always be accomplished, but in many cases it is possible, and a well-managed approach can greatly increase the reliablity of the delivered content. A rigorous ‘change control’ process must be introduced, with any sugsequent changes to the system being well tested in a lab environment before turned loose in the ‘real world.’ Some examples in the previous post show what happens if this is not done….
So, some food for thought… or to put it another way, it is possible to eat your digital cake!
