|IN THE SPOTLIGHT|
A detailed look at how we assess and rate HD DVD and Blu-ray discs. Plus, beyond the jargon: descriptions and examples of the most common technical issues referenced in our reviews.
By Peter M. Bracke
Thursday March 22, 2007 at 1:00AM EST
More than any other feature on High-Def Digest, our software reviews generate by far the most feedback. Our forums (and our email box) are routinely flooded with reactions as varied as our readers themselves, whether they're writing in to praise our glowing review of a given title, or to lament how we in the world could possibly give another title a poor rating. Amid all this, the most oft-asked question we receive isn't about the details of any one particular review, but the hows and whys of our approach to disc reviewing itself.
To address all those inquiries, we've created this detailed overview of our Review Methodology. Breaking down each component of our reviews, we hope this will offer a thorough explanation of our systematic assessment of any given disc title. From how we approach dissecting the components of next-gen video and audio performance to the application of our (sometimes controversial) star ratings, it explains our philosophy toward balancing disc quality, artistic intent and supplemental content to formulate a ranking of overall value for your consumer dollar.
Since a cutting-edge technology like high-definition can also be so highly technical, this Methodology is also written to provide a primer on the most common terms we use in our review descriptions. We hope the following serves as an easy-to-understand introduction to how we assess Blu-ray and HD DVD disc quality, and helps further your enjoyment of our site -- and perhaps even of high-def discs in general.
Reviewing the Movie Itself
The "movie itself" section is the only category of our disc reviews that is 100% subjective -- the opinions expressed is the personal reaction of the reviewer, and the reviewer alone. We rate the movies how we see 'em -- the good, the bad and the ugly. Needless to say, our readers don't always agree, but that's the nature of film criticism -- one man's 'Showgirls' is another man's 'Citizen Kane.'
That's why we put the movie section right up front -- if you don't care what we think of the movie itself (or if you vehemently disagree with our opinion), you can skip right over it and jump down to the technical portions of our review. All of our reviews are divided into five categories -- Movie, Video, Audio, Supplements, High-Def Exclusives and Final Thoughts -- and a handy navigation box is accessible at the top right for quick access to each section. (Note that below supplements there is also an Easter Egg section, which includes instructions for accessing hidden features on discs that include them.)
Assessing the Video
Image quality is widely considered to be the foremost attribute of any great high-def release. It is certainly the most difficult area to assess, as the process of transferring the source material to high-definition (known as telecine) is a very complicated, highly technical craft, often entailingsubjective decisions for which there are no "right" or "wrong" answers. Not to mention all the very intimidating-sounding terms used in video reviews, which are often so arcane they sometimes leave even the most knowledgeable videophile scratching their head in confusion.
Artist Intention vs. A Little Razzle-Dazzle?
Before we dive in the techno-babble, first an explanation of our overall approach to rating a disc's quality. Without a doubt, one of the most hotly debated issues surrounding disc quality is whether a transfer should remain faithful to the original source material (and thus the filmmaker's original intentions), or simply look as "good" as it possibly can. Needless to say, the studios do not provide us with the original masters against which to compare their Blu-ray and HD DVD transfers; and even if they wanted to, there would be no practical way to implement such a comparison.
Thus, reviewing the video quality on any given disc requires a bit of blind faith. We here at High-Def Digest have received continued assurances from all of the major Blu-ray- and HD DVD-supporting studios that the original filmmakers are always consulted when striking a new high-def master of a title. In fact, according to current Directors Guild of America rules, a film's director and/or director of photography must be asked to participate in the process whenever a new video master of their film is made. Of course, they can choose not to participate, or may be unable to, but such occurrences are considered rare. So we have no reason to believe that the studios are not supportive of preserving the artistic integrity of the filmmaker's intended vision.
So when it comes to our reviews, we make pains to not detract from a disc's rating simply because a filmmaker's stylistic choices may not "look good" to our eyes. For example, take the HD DVD release of Steven Soderbergh's 'Traffic.' Intentionally schizophrenic in its visuals, Soderbergh employed various film stocks, excessive coloring and pumped-up "film grain" to create a thematic contrast between his narrative's interlocking stories. Though 'Traffic' could have easily been made to look "better" on HD DVD via today's digital processing techniques (such as grain removal, or by balancing colors to be more "natural," etc.), it would without question be a violation of the filmmaker's vision. Though again it is impossible to always know what is overt stylization of an image when assessing a disc's video transfer, our guiding principle when assessing video quality here at High-Def Digest is that the filmmaker's desires should come first.
However, when making our final rating assessment of a transfer, we can't ignore that intentional stylization can often result in an overall lessening of image quality. For example, if a filmmaker chooses to employ a particular style choice, or apply post-production tweaking during the telecine process, and it results in noticeable deficiencies elsewhere (such as high contrast washing out colors, or intensely saturated hues lessening detail, etc.), we feel it is essential to take this into account.
Ultimately, the goal is to provide a star rating for video quality that addresses both a transfer's faithfulness to the original source material, as well as how it compares in overall presentation to the current reference-bearers for the format. In other words, our video analysis and corresponding star rating is meant to assess not only how accurate a transfer is to the filmmaker's intent, but its "wow factor" and ranking in comparison to its peer titles.
Minting a Great Source
By far, we have found that the single most important ingredient to a great high-def transfer is having the highest-quality source material. We've seen more Blu-ray and HD DVD discs deep-sixed by poor prints, badly botched video masters or bizarre remastering decisions that can make mincemeat out of what could have been a sterling presentation.
Film-based titles are particularly susceptible. A freshly-minted print is essential -- not a dusty old 16mm dupe that has been sitting around in some studio exec's closet for the past three decades. Of course, cost is usually the main impediment: hauling out a film's negative, creating a new print and spending the required amount of time and effort to really bring out the best in the material can be quite expensive. Prints littered with splice marks, scratches, faded blacks, dirt and reel markers (those little circle- and oval-shaped splotches you see flash in the upper right corner of a movie, which were originally placed there to alert the theater projectionist it was time to change the reel) may have been fine in the days of low-res VHS tape, but they are simply unacceptable today. High-definition is completely unforgiving, and even the slightest blemish is not safe from the technology's eagle-eyed vision.
Note that there are occasions when print defects are simply unavoidable. Though it is rare, studios sometimes openly document particularly troublesome issues. For example, Lionsgate's Blu-ray release of 'Stargate' came with mention of problematic source material that was used for the "Director's Cut" of the film included on the disc. Though of course we'd all love for every transfer to be perfect, content providers score big points with us for giving consumers this kind of head's up, because at least we know that if miracles can't always happen, at least some care and concern was expended to ensure the best presentation possible. (We also usually do not downgrade a disc's tech ratings for such documented problems, in consideration of their often unavoidable nature).
While material originating from the video domain (such as shot-on-HD music titles and other special-interest programming) may not have the same "source issues" as film, they are hardly immune from their own complications. As of now, the vast majority of Blu-ray and HD DVD video-based releases were minted from full HD, 1080 masters. As such, we've found they look great (Nine Inch Nails' recent 'Beside You in Time' HD DVD and Blu-ray releases being a perfect example), and will likely continue to for the foreseeable future.
However, before the HD era, most non-film-based DVD releases (even if they were shot in high-definition) were usually converted to standard-def resolution before post-production, and then edited entirely in that realm. In many cases, once the source was transferred, it was discarded. (This also goes for the majority of pre-HD era television programming, which though often shot on film, was usually transferred and post-produced on standard-def video.) Alas, we anticipate the day when this older material starts to make the transition to HD, and those older elements can't be found. The only solution will be to "upconvert" those ancient masters. We suspect they'll look decent to dreadful, unless the content providers can go back and find the old elements, re-transfer them to HD, then completely reconstruct them in the editing room. (We won't hold our breath for this one.) In any case, for now we'll just continue to enjoy all the great HD-based video-originated titles currently hitting Blu-ray and HD DVD, and hope for the best in the future.
Components of Rating a Video Transfer
Once a transfer's source material is assessed, it's time to rate the individual components of image quality. Though every title has its own unique characteristics, there is always a set of definite basics that must be covered. Though deconstructing every single aspect of the telecine process is beyond the scope of this article, the following is an overview of the parameters we assess when reviewing every title, as well as as explanation of the terms that are most common in our analyses.
Compression Artifacts & Other Stories
Finally, the last area we look at when assessing video is the encoding, and any attendant compression anomalies.
We are not going to pick any favorites among the three compression codecs currently in use on Blu-ray and HD DVD -- MPEG-2, AVC MPEG-4 and VC-1 -- because the high-def formats just haven't been road-tested long enough. Each has their plusses and minuses, so the jury is still out on which one is "best." However, poorly-encoded video, paired with questionable decisions made during telecine (often to "improve" an image artificially), can result in a number of obvious defects we lump under the umbrella term "compression artifacts." Among the major offenders:
Assessing the Audio
Thankfully, assessing the audio is a bit less complicated than the video, but no less important. There is much less room for overt stylization of audio, and the goal is almost always to present an experience that is as transparent to the original studio master as is possible.
That Ol' Snap, Crackle and Pop
Of course, there are still many obvious ailments that can affect low-quality sources, particularly older titles whose original elements have either been lost, or proven too expensive to give an effective remaster. Audible hiss is perhaps the most prevalent (remember all those old, noisy cassette tapes you used to record your favorite songs off the radio on?). Distortion is another villain, with dialogue the frequent victim, resulting in the actors sounding as if they were all recorded in a dilapidated old phone booth. Crackles, pops and, in some cases, audio dropouts can also mar poor-quality mixes. And indicative of many soundtracks that pre-date the era of theatrical surround sound, low bass tones are often "flat," particularly on mono mixes, where the entire frequency range had to be compressed on to one single channel of audio.
Components of an Audio Soundtrack
Beyond the quality of the source material, following are the additional characteristics we take into account when reviewing a soundtrack:
Factoring in the Supplements and Exclusives
The final piece of the reviewing puzzle is weighing the quality of supplemental material. Though we feel that the video and audio presentation of a film is ultimately the most important aspect of any next-gen release, and as such, should never be sacrificed for supplemental features, bonus content is still a very important part of the mix, as most of us early adopters who are upgrading our DVD collections want at least the same amount of extras we received the first time around. Studio support of bonus features is also vital to mainstream acceptance of Blu-ray and HD DVD -- as the success of standard-def DVD proved, consumers now expect supplemental material on their discs, and nothing less than at least something will do.
Rating the quality of bonus features is a bit more subjective than our more technical approach to the video and audio. We strive not to offer any single one "right" viewpoint, so much as a general impression of how insightful, entertaining and comprehensive the supplemental content is. There are, however, a few general considerations we take into account when it comes to making-of material.
Standard-Def Extras, Say Hello to Blu-ray and HD DVD
First, are there any bonus features that have been ported over from previous standard-def versions, be it DVD, LaserDisc or VHS? Since all of this material has already been created, it is hard to see any reason for its lack of inclusion on a comparable Blu-ray and/or HD DVD version, other than a studio's desire to want to "double dip" the title with another high-def edition down the road. (To be fair, there are sometimes legal clearance issues with some extras, but shouldn't the studios have figured this out already?) We certainly give points to Blu-ray and HD DVD titles that faithfully replicate their standard-def counterparts, and seeing as how both formats are still being sold primarily to early adopters, they deserve at least the same number of extras they've already paid for once before (or more) on DVD.
Second, were these extras created exclusively for the DVD, or simply re-purposed from various electronic press kits (EPKs) or promotional television specials? Quite frankly, regurgitated promo fluff can be pretty awful. Such material was almost always created during production, so it usually sheds absolutely no light on a film's reception, critical reaction and legacy. Though such material is a nice bonus for completists, we certainly don't rate it as highly as DVD-exclusive content.
Note that we are, however, big fans of stand-alone television documentaries and other types of third-party material. Warner, for example, has been quite supportive of these types of features, particularly on their classic catalog titles. Whether it be an hour-long American Movie Classics documentary on the art of film editing, or that way-cool retrospective feature on the life of Joan Crawford that graced Turner Classic Movies, these types of specials can be fantastic, and make wonderful additions to a Blu-ray and/or HD DVD release. These are not to be confused with more EPK-type material (such as those dreadful "HBO First Look" specials -- bleech!).
Fresh to Blu-ray and HD DVD is the concept of supplements exclusive to high-definition. Whether it be additional documentaries and featurettes produced in full 1080i/p HD video, or new fangled interactivity and web connectivity, this is an exciting new frontier. Though we almost always give points for exclusive HD extras, as the content is still in its infancy, there are a few variables we consider.
By far the most important factor is whether an HD exclusive is truly informational, or simply a gimmick. Give us the choice between a picture-in-picture video commentary, or an interactive game where you have to diffuse bombs before Keanu Reeves' bus explodes, and we'll always pick the former. Though we'll certainly award at least half a star to any title that boasts some true HD exclusives (we applaud content providers that are being progressive with the format), the content ultimately needs to be something genuine (and not merely marketing fluff) to earn a higher rating.
Then there is the question of basic format interactivity. Just about every Blu-ray and HD DVD title has (or should have) standard navigational upgrades over standard-def DVD, such as the ability to access menus on-the-fly during playback of the main feature. Custom bookmarks are also finding great favor, particularly on HD DVD titles from Universal and Warner. The latter studio even offers the ability to pan-and-zoom a disc's image as a standard feature on all of its HD DVD releases. However, while we mention the inclusion of such features in our reviews, we don't award more than half-a-star for such basic functions unless they go above and beyond the status quo.
So Many Supplements, So Many Stars
So, how do we actually give star ratings for extras, whether ported-over from the standard-def releases or true HD exclusives? Though again this is a more subjective area than the audio/video, we do adhere to the "crap in, crap out" principle -- even if a disc has ten commentaries, if they are all just a bunch of dead air, we aren't giving 'em any points. But in general, we award the following stars for supplements:
Additional materials, such as vintage archival footage, audio-only elements (such as isolated soundtracks, etc.) and blooper reels are all subjectively rated. Again, though, the general rule of thumb is to award a half-star for each half-hour's worth of material, depending on its quality. As for easter eggs, we also don't usually award any additional stars for such content, as it tends to be very minor in nature. However, should an egg be substantial (such as unlocking an alternate version of the movie, for example), we will boost the star rating accordingly.
In the case of any standard-def-like features that are exclusive to the Blu-ray and/or HD DVD release, such as commentaries or featurettes, the same general approach to ratings as described above applies. For material that is truly original to HD, we add the following star ratings:
Achieving an Overall Rating
Now, it's time to put it all together.
Once each category of a disc release has been rated, it's time to add 'em all up and average it out. Our current formula for calculating our Overall rating is fairly straightforward, weighing each of the four major categories (Movie, Video, Audio, and Supplements) equally.
For example, if a disc were to receive a two-and-a-half star Movie rating, four stars for Video, four-and-a-half stars for Audio, and a combined three-and-a-half stars for Supplements, the result would be:
2.5 (Movie) + 4 (Video) + 4.5 (Audio) + 3.5 (Supplements) = 14.5. Then: 14.5 / 4 (# of Categories) = 3.65
Using the general formula for rounding, the final rating would then be a 3.5.
Note that because they are still relatively rare, we do not include the High-Def Exclusives rating in our average. Instead, we use it as a form of "bonus points," ie: for discs that do include high-def exclusives, usually we'll add a half a star or more to the overall grade based on the quality and quantity the content. In the future, as exclusive content becomes more common and expected on high-def titles, we'll begin averaging in exclusives along with the other four categories. But for now, we're keeping them as that extra little gold star to boost top-tier titles.
Note that the Overall rating is intended only as a guide. We welcome all of our readers to apply their own preferences. For example, many simply do not care about our opinion of a movie, or about the supplemental features. So we've designed our rating system to work just as easily with any of the rating categories eliminated.
Take the above example. If one didn't care about our movie preferences, simply eliminating that portion and re-averaging would yield a higher rating, consistent with our overall findings of better technical merit versus artistic:
4 (Video) + 4.5 (Audio) + 3.5 (Supplements) = 12.0. Then: 12.0/3 (# of Categories) = 4.00
A 3.5-star disc now becomes a 4.0-star disc.
Finally, we also sometimes adjust an overall rating after averaging based on additional external factors. For example, if a disc seems priced outlandishly high it may warrant lowering its composite rating. Conversely, if a disc is a real bargain, its overall value for money thus would earn it a boost. In any case, whenever we do adjust an Overall rating for such outside factors, we'll always note it in the reviews concluding text.
There you have it. As always, we welcome your feedback on all of our reviews here at High-Def Digest. And as both the Blu-ray and HD DVD formats continue to mature, we will certainly be fine-tuning and revising our review methods. Please feel free to share your thoughts with us via our handy Feedback Form, or in our forums area, where we've set up a dedicated thread in our feedback forum for discussion of our review methodology.