|IN THE SPOTLIGHT|
Our Review Methodology
A detailed look at how we assess and rate HD DVD and Blu-ray discs. Plus, beyond the jargon: descriptions and examples of the most common technical issues referenced in our reviews.
By Peter M. Bracke
Thursday March 22, 2007 at 1:00AM EST
More than any other feature on High-Def Digest, our software reviews generate by far the most feedback. Our forums (and our email box) are routinely flooded with reactions as varied as our readers themselves, whether they're writing in to praise our glowing review of a given title, or to lament how we in the world could possibly give another title a poor rating. Amid all this, the most oft-asked question we receive isn't about the details of any one particular review, but the hows and whys of our approach to disc reviewing itself.
To address all those inquiries, we've created this detailed overview of our Review Methodology. Breaking down each component of our reviews, we hope this will offer a thorough explanation of our systematic assessment of any given disc title. From how we approach dissecting the components of next-gen video and audio performance to the application of our (sometimes controversial) star ratings, it explains our philosophy toward balancing disc quality, artistic intent and supplemental content to formulate a ranking of overall value for your consumer dollar.
Since a cutting-edge technology like high-definition can also be so highly technical, this Methodology is also written to provide a primer on the most common terms we use in our review descriptions. We hope the following serves as an easy-to-understand introduction to how we assess Blu-ray and HD DVD disc quality, and helps further your enjoyment of our site -- and perhaps even of high-def discs in general.
Reviewing the Movie Itself
The "movie itself" section is the only category of our disc reviews that is 100% subjective -- the opinions expressed is the personal reaction of the reviewer, and the reviewer alone. We rate the movies how we see 'em -- the good, the bad and the ugly. Needless to say, our readers don't always agree, but that's the nature of film criticism -- one man's 'Showgirls' is another man's 'Citizen Kane.'
That's why we put the movie section right up front -- if you don't care what we think of the movie itself (or if you vehemently disagree with our opinion), you can skip right over it and jump down to the technical portions of our review. All of our reviews are divided into five categories -- Movie, Video, Audio, Supplements, High-Def Exclusives and Final Thoughts -- and a handy navigation box is accessible at the top right for quick access to each section. (Note that below supplements there is also an Easter Egg section, which includes instructions for accessing hidden features on discs that include them.)
Assessing the Video
Image quality is widely considered to be the foremost attribute of any great high-def release. It is certainly the most difficult area to assess, as the process of transferring the source material to high-definition (known as telecine) is a very complicated, highly technical craft, often entailingsubjective decisions for which there are no "right" or "wrong" answers. Not to mention all the very intimidating-sounding terms used in video reviews, which are often so arcane they sometimes leave even the most knowledgeable videophile scratching their head in confusion.
Artist Intention vs. A Little Razzle-Dazzle?
Before we dive in the techno-babble, first an explanation of our overall approach to rating a disc's quality. Without a doubt, one of the most hotly debated issues surrounding disc quality is whether a transfer should remain faithful to the original source material (and thus the filmmaker's original intentions), or simply look as "good" as it possibly can. Needless to say, the studios do not provide us with the original masters against which to compare their Blu-ray and HD DVD transfers; and even if they wanted to, there would be no practical way to implement such a comparison.
Thus, reviewing the video quality on any given disc requires a bit of blind faith. We here at High-Def Digest have received continued assurances from all of the major Blu-ray- and HD DVD-supporting studios that the original filmmakers are always consulted when striking a new high-def master of a title. In fact, according to current Directors Guild of America rules, a film's director and/or director of photography must be asked to participate in the process whenever a new video master of their film is made. Of course, they can choose not to participate, or may be unable to, but such occurrences are considered rare. So we have no reason to believe that the studios are not supportive of preserving the artistic integrity of the filmmaker's intended vision.
So when it comes to our reviews, we make pains to not detract from a disc's rating simply because a filmmaker's stylistic choices may not "look good" to our eyes. For example, take the HD DVD release of Steven Soderbergh's 'Traffic.' Intentionally schizophrenic in its visuals, Soderbergh employed various film stocks, excessive coloring and pumped-up "film grain" to create a thematic contrast between his narrative's interlocking stories. Though 'Traffic' could have easily been made to look "better" on HD DVD via today's digital processing techniques (such as grain removal, or by balancing colors to be more "natural," etc.), it would without question be a violation of the filmmaker's vision. Though again it is impossible to always know what is overt stylization of an image when assessing a disc's video transfer, our guiding principle when assessing video quality here at High-Def Digest is that the filmmaker's desires should come first.
However, when making our final rating assessment of a transfer, we can't ignore that intentional stylization can often result in an overall lessening of image quality. For example, if a filmmaker chooses to employ a particular style choice, or apply post-production tweaking during the telecine process, and it results in noticeable deficiencies elsewhere (such as high contrast washing out colors, or intensely saturated hues lessening detail, etc.), we feel it is essential to take this into account.
Ultimately, the goal is to provide a star rating for video quality that addresses both a transfer's faithfulness to the original source material, as well as how it compares in overall presentation to the current reference-bearers for the format. In other words, our video analysis and corresponding star rating is meant to assess not only how accurate a transfer is to the filmmaker's intent, but its "wow factor" and ranking in comparison to its peer titles.
Minting a Great Source
By far, we have found that the single most important ingredient to a great high-def transfer is having the highest-quality source material. We've seen more Blu-ray and HD DVD discs deep-sixed by poor prints, badly botched video masters or bizarre remastering decisions that can make mincemeat out of what could have been a sterling presentation.
Film-based titles are particularly susceptible. A freshly-minted print is essential -- not a dusty old 16mm dupe that has been sitting around in some studio exec's closet for the past three decades. Of course, cost is usually the main impediment: hauling out a film's negative, creating a new print and spending the required amount of time and effort to really bring out the best in the material can be quite expensive. Prints littered with splice marks, scratches, faded blacks, dirt and reel markers (those little circle- and oval-shaped splotches you see flash in the upper right corner of a movie, which were originally placed there to alert the theater projectionist it was time to change the reel) may have been fine in the days of low-res VHS tape, but they are simply unacceptable today. High-definition is completely unforgiving, and even the slightest blemish is not safe from the technology's eagle-eyed vision.
Note that there are occasions when print defects are simply unavoidable. Though it is rare, studios sometimes openly document particularly troublesome issues. For example, Lionsgate's Blu-ray release of 'Stargate' came with mention of problematic source material that was used for the "Director's Cut" of the film included on the disc. Though of course we'd all love for every transfer to be perfect, content providers score big points with us for giving consumers this kind of head's up, because at least we know that if miracles can't always happen, at least some care and concern was expended to ensure the best presentation possible. (We also usually do not downgrade a disc's tech ratings for such documented problems, in consideration of their often unavoidable nature).
While material originating from the video domain (such as shot-on-HD music titles and other special-interest programming) may not have the same "source issues" as film, they are hardly immune from their own complications. As of now, the vast majority of Blu-ray and HD DVD video-based releases were minted from full HD, 1080 masters. As such, we've found they look great (Nine Inch Nails' recent 'Beside You in Time' HD DVD and Blu-ray releases being a perfect example), and will likely continue to for the foreseeable future.
However, before the HD era, most non-film-based DVD releases (even if they were shot in high-definition) were usually converted to standard-def resolution before post-production, and then edited entirely in that realm. In many cases, once the source was transferred, it was discarded. (This also goes for the majority of pre-HD era television programming, which though often shot on film, was usually transferred and post-produced on standard-def video.) Alas, we anticipate the day when this older material starts to make the transition to HD, and those older elements can't be found. The only solution will be to "upconvert" those ancient masters. We suspect they'll look decent to dreadful, unless the content providers can go back and find the old elements, re-transfer them to HD, then completely reconstruct them in the editing room. (We won't hold our breath for this one.) In any case, for now we'll just continue to enjoy all the great HD-based video-originated titles currently hitting Blu-ray and HD DVD, and hope for the best in the future.
Components of Rating a Video Transfer
Once a transfer's source material is assessed, it's time to rate the individual components of image quality. Though every title has its own unique characteristics, there is always a set of definite basics that must be covered. Though deconstructing every single aspect of the telecine process is beyond the scope of this article, the following is an overview of the parameters we assess when reviewing every title, as well as as explanation of the terms that are most common in our analyses.
- Black level: Are a transfer's blacks solid, and not gray? Are they consistent throughout? Poor blacks can result in washed out color, an overly bright picture, and poor contrast.
- Contrast: Is there an appreciable difference between the darkest and lightest areas of the picture? In other words, are blacks black, and whites white? And does contrast look consistent across the entire grayscale? Or are whites so "blown-out" that they bloom, and the image looks way too bright?
- Colors: Barring any stylistic intentions, are colors vibrant but not smeary, blurry or fuzzy? Or are they over- or under-saturated? And are primary colors natural and without tint problems?
- Fleshtones: Typically overdone colors or weak saturation reveals themselves most noticeably in human skin, as we are most sensitive to unnaturalness in what we know most intimately. Oversaturated colors often result in skin that looks impossibly flush, with fine details (such as the ability to see pores or hairs on flesh) lost to a wash of color. Tint issues are also obvious here, with skin appearing either too pink (the dreaded "pig-face" syndrome) or too green (for that Linda Blair "pea soup" look).
- Detail: How well are fine textures and objects visible in an image? Do close-ups look natural, life-like and vivid? And in long-shots, can you make out far-away-objects, or are they just a mass of undefined blobs?
- Shadow delineation. Is fine detail visible even in the darkest (but not true black) areas of the picture? Usually, if the "fall-off" in the low end of the grayscale is too steep (also called "black crush") fine details are lost completely. This results in an image that, while it may "pop" with vibrancy, is too contrasted and, again, lacks fine detail.
Compression Artifacts & Other Stories
Finally, the last area we look at when assessing video is the encoding, and any attendant compression anomalies.
We are not going to pick any favorites among the three compression codecs currently in use on Blu-ray and HD DVD -- MPEG-2, AVC MPEG-4 and VC-1 -- because the high-def formats just haven't been road-tested long enough. Each has their plusses and minuses, so the jury is still out on which one is "best." However, poorly-encoded video, paired with questionable decisions made during telecine (often to "improve" an image artificially), can result in a number of obvious defects we lump under the umbrella term "compression artifacts." Among the major offenders:
- Noise: Often misunderstood, video noise is not the same thing as film grain. The technical definition of noise is a collection of random pixels of low contrast around edges of high contrast. The result is a "buzzing" of pixels, usually only perceptible on solid areas of the picture (such as a similar-colored wall, or a sky) that looks like a swarm of insects (hence the term "mosquito noise").
- Macroblocking: Sometimes confused with noise, macroblocking is usually a reference to the actual break-up of screen pixels due to poor encoding/decoding. Though quite rare in pre-recorded formats like Blu-ray and HD DVD, it is rather common in over-the-air HD broadcasts due to bandwidth limitations. Fast-action is most susceptible, where quick-cut and fast-action shots break down into big blocks of pixels, as the encoder and/or decoder can't keep up with the increased bit rate.
- Posterization. Another word for color banding. Gradients of color look like stripes instead of a smooth, clean transition. Note that the blame for posterization is often hard to place, as cheaper players and display devices sometimes have trouble resolving fine gradations on otherwise perfectly encoded discs.
Assessing the Audio
Thankfully, assessing the audio is a bit less complicated than the video, but no less important. There is much less room for overt stylization of audio, and the goal is almost always to present an experience that is as transparent to the original studio master as is possible.
That Ol' Snap, Crackle and Pop
Of course, there are still many obvious ailments that can affect low-quality sources, particularly older titles whose original elements have either been lost, or proven too expensive to give an effective remaster. Audible hiss is perhaps the most prevalent (remember all those old, noisy cassette tapes you used to record your favorite songs off the radio on?). Distortion is another villain, with dialogue the frequent victim, resulting in the actors sounding as if they were all recorded in a dilapidated old phone booth. Crackles, pops and, in some cases, audio dropouts can also mar poor-quality mixes. And indicative of many soundtracks that pre-date the era of theatrical surround sound, low bass tones are often "flat," particularly on mono mixes, where the entire frequency range had to be compressed on to one single channel of audio.
Components of an Audio Soundtrack
Beyond the quality of the source material, following are the additional characteristics we take into account when reviewing a soundtrack:
- Dynamic Range: In simplest terms, a measurement of output before it either is overloaded or becomes distorted. In other words, does the "high end" (say, the blare of a trumpet, or a high-pitched scream) sound clean, bright and natural? Do the middle tones (the pleasing sound of a violin being strung, for example, or the purr of a kitten) seem pleasing, warm and spacious? And is low bass strong and solid -- not flat and wimpy, or distorted or too "thumpy?" If you have ever been to a rock concert where the booming bass results in feedback or static in the speakers, that's an example of dynamic range being exceeded. Not fun.
- Directionality: A general term used to describe how accurately sounds are placed in the various speakers, and how clearly they can be localized by your ears. When Peter Sellers slips on that banana peel and it flies off-screen, does the corresponding sound effect get directed to the correct speaker? And are specific sounds audible and separate from each other, or does it all feel like just a bunch of sonic mush, with no sense of clear placement?
- Imaging: Also often referred to as "transparency," or "seamless pans." A term for the approximation of "movement" of sounds between two or more channels. This should feel natural and smooth, as if there are not specific speakers in the room, but a natural and immersive soundfield that surrounds the listener. A well-designed mix will create a truly active experience that sustains the illusion that sounds are coming from all around the listener in a full 360 degrees. This type of highly-enveloping soundtrack is often said to be one that creates a "wall of sound," and is particularly indicative of high-resolution audio formats like Dolby TrueHD, DTS-HD Lossless Master Audio, and uncompressed PCM.
- Balance. Also called "Volume Balance" or "Volume Matching." Are dialogue, effects and score equal in terms of their prominence in the mix? Or is dialogue smothered by effects that sound too loud, or by a booming score? Note that poor balance can be a result of a improperly-calibrated home theater set-up (such as a center channel whose individual volume level has not been matched properly, resulting in quiet dialogue), so it is important to always check your system settings. However, low dialogue and overbearing effects are common problems with many modern soundtracks, especially action flicks. Unfortunately, some content providers haven't yet learned that louder explosions are not necessarily better explosions.
Factoring in the Supplements and Exclusives
The final piece of the reviewing puzzle is weighing the quality of supplemental material. Though we feel that the video and audio presentation of a film is ultimately the most important aspect of any next-gen release, and as such, should never be sacrificed for supplemental features, bonus content is still a very important part of the mix, as most of us early adopters who are upgrading our DVD collections want at least the same amount of extras we received the first time around. Studio support of bonus features is also vital to mainstream acceptance of Blu-ray and HD DVD -- as the success of standard-def DVD proved, consumers now expect supplemental material on their discs, and nothing less than at least something will do.
Rating the quality of bonus features is a bit more subjective than our more technical approach to the video and audio. We strive not to offer any single one "right" viewpoint, so much as a general impression of how insightful, entertaining and comprehensive the supplemental content is. There are, however, a few general considerations we take into account when it comes to making-of material.
Standard-Def Extras, Say Hello to Blu-ray and HD DVD
First, are there any bonus features that have been ported over from previous standard-def versions, be it DVD, LaserDisc or VHS? Since all of this material has already been created, it is hard to see any reason for its lack of inclusion on a comparable Blu-ray and/or HD DVD version, other than a studio's desire to want to "double dip" the title with another high-def edition down the road. (To be fair, there are sometimes legal clearance issues with some extras, but shouldn't the studios have figured this out already?) We certainly give points to Blu-ray and HD DVD titles that faithfully replicate their standard-def counterparts, and seeing as how both formats are still being sold primarily to early adopters, they deserve at least the same number of extras they've already paid for once before (or more) on DVD.
Second, were these extras created exclusively for the DVD, or simply re-purposed from various electronic press kits (EPKs) or promotional television specials? Quite frankly, regurgitated promo fluff can be pretty awful. Such material was almost always created during production, so it usually sheds absolutely no light on a film's reception, critical reaction and legacy. Though such material is a nice bonus for completists, we certainly don't rate it as highly as DVD-exclusive content.
Note that we are, however, big fans of stand-alone television documentaries and other types of third-party material. Warner, for example, has been quite supportive of these types of features, particularly on their classic catalog titles. Whether it be an hour-long American Movie Classics documentary on the art of film editing, or that way-cool retrospective feature on the life of Joan Crawford that graced Turner Classic Movies, these types of specials can be fantastic, and make wonderful additions to a Blu-ray and/or HD DVD release. These are not to be confused with more EPK-type material (such as those dreadful "HBO First Look" specials -- bleech!).
Fresh to Blu-ray and HD DVD is the concept of supplements exclusive to high-definition. Whether it be additional documentaries and featurettes produced in full 1080i/p HD video, or new fangled interactivity and web connectivity, this is an exciting new frontier. Though we almost always give points for exclusive HD extras, as the content is still in its infancy, there are a few variables we consider.
By far the most important factor is whether an HD exclusive is truly informational, or simply a gimmick. Give us the choice between a picture-in-picture video commentary, or an interactive game where you have to diffuse bombs before Keanu Reeves' bus explodes, and we'll always pick the former. Though we'll certainly award at least half a star to any title that boasts some true HD exclusives (we applaud content providers that are being progressive with the format), the content ultimately needs to be something genuine (and not merely marketing fluff) to earn a higher rating.
Then there is the question of basic format interactivity. Just about every Blu-ray and HD DVD title has (or should have) standard navigational upgrades over standard-def DVD, such as the ability to access menus on-the-fly during playback of the main feature. Custom bookmarks are also finding great favor, particularly on HD DVD titles from Universal and Warner. The latter studio even offers the ability to pan-and-zoom a disc's image as a standard feature on all of its HD DVD releases. However, while we mention the inclusion of such features in our reviews, we don't award more than half-a-star for such basic functions unless they go above and beyond the status quo.
So Many Supplements, So Many Stars
So, how do we actually give star ratings for extras, whether ported-over from the standard-def releases or true HD exclusives? Though again this is a more subjective area than the audio/video, we do adhere to the "crap in, crap out" principle -- even if a disc has ten commentaries, if they are all just a bunch of dead air, we aren't giving 'em any points. But in general, we award the following stars for supplements:
- Audio commentaries. Usually receive one star for the first commentary, then a half-a-star for each additional commentary. Of course, this assumes that they are actually informative and substantial, and not just overkill.
- Documentaries: As a general rule of thumb, we usually award a half-star for every half-hour of featurette or documentary material. Again, provided it is of sufficient quality, and not pure fluff.
- Deleted Scenes: On average, a decent assortment of excised footage rates a half-star.
- Still Galleries. Also includes storyboard comparisons, animatics, etc. On average, these receive a half-star (permitting they are substantial in nature, and not just a handful of promo shots you'd find on the imdb).
- Trailers: While we mention them in the text of our reviews, we generally don't award points for trailers. They have become so common now on all disc-based formats that they hardly seem like the bonuses the once did back in the days of LaserDisc.
Additional materials, such as vintage archival footage, audio-only elements (such as isolated soundtracks, etc.) and blooper reels are all subjectively rated. Again, though, the general rule of thumb is to award a half-star for each half-hour's worth of material, depending on its quality. As for easter eggs, we also don't usually award any additional stars for such content, as it tends to be very minor in nature. However, should an egg be substantial (such as unlocking an alternate version of the movie, for example), we will boost the star rating accordingly.
In the case of any standard-def-like features that are exclusive to the Blu-ray and/or HD DVD release, such as commentaries or featurettes, the same general approach to ratings as described above applies. For material that is truly original to HD, we add the following star ratings:
- Picture-in-Picture: As video commentaries encompass multiple media (essentially functioning as an audio commentary, video documentary and still gallery all in one), and can offer an unmatched depth of content, we reward them quite favorably in our star ratings. In general, a quality P-I-P track (also known as the "In-Movie Experience" or "Total Experience") will get two to two-and-a-half stars alone.
- U-Control: Currently exclusive to Universal releases, the ability to toggle exclusive features on and off is a cool trick. So we award these releases at least one extra star. Though as this technology is still in its infancy, that could change, with future star ratings adjusted higher or lower...
- Blu-Wizard: Same as above. Currently included on only two Sony titles ('Black Hawk Down' and Resident Evil: Apocalypse, it is just too soon to give this burgeoning technology a standard rating.
- Blu-Scape: Exclusive to Disney, these animated short films (included on discs such as 'The Haunted Mansion,' 'Dinosaur,' and 'Flightplan') are pretty to look at, and wonderfully constructed by award-winning filmmaker Louie Shwartzberg. Still, given their limited replay value, we currently rate them only one star at most.
Achieving an Overall Rating
Now, it's time to put it all together.
Once each category of a disc release has been rated, it's time to add 'em all up and average it out. Our current formula for calculating our Overall rating is fairly straightforward, weighing each of the four major categories (Movie, Video, Audio, and Supplements) equally.
For example, if a disc were to receive a two-and-a-half star Movie rating, four stars for Video, four-and-a-half stars for Audio, and a combined three-and-a-half stars for Supplements, the result would be:
2.5 (Movie) + 4 (Video) + 4.5 (Audio) + 3.5 (Supplements) = 14.5. Then: 14.5 / 4 (# of Categories) = 3.65
Using the general formula for rounding, the final rating would then be a 3.5.
Note that because they are still relatively rare, we do not include the High-Def Exclusives rating in our average. Instead, we use it as a form of "bonus points," ie: for discs that do include high-def exclusives, usually we'll add a half a star or more to the overall grade based on the quality and quantity the content. In the future, as exclusive content becomes more common and expected on high-def titles, we'll begin averaging in exclusives along with the other four categories. But for now, we're keeping them as that extra little gold star to boost top-tier titles.
Note that the Overall rating is intended only as a guide. We welcome all of our readers to apply their own preferences. For example, many simply do not care about our opinion of a movie, or about the supplemental features. So we've designed our rating system to work just as easily with any of the rating categories eliminated.
Take the above example. If one didn't care about our movie preferences, simply eliminating that portion and re-averaging would yield a higher rating, consistent with our overall findings of better technical merit versus artistic:
4 (Video) + 4.5 (Audio) + 3.5 (Supplements) = 12.0. Then: 12.0/3 (# of Categories) = 4.00
A 3.5-star disc now becomes a 4.0-star disc.
Finally, we also sometimes adjust an overall rating after averaging based on additional external factors. For example, if a disc seems priced outlandishly high it may warrant lowering its composite rating. Conversely, if a disc is a real bargain, its overall value for money thus would earn it a boost. In any case, whenever we do adjust an Overall rating for such outside factors, we'll always note it in the reviews concluding text.
There you have it. As always, we welcome your feedback on all of our reviews here at High-Def Digest. And as both the Blu-ray and HD DVD formats continue to mature, we will certainly be fine-tuning and revising our review methods. Please feel free to share your thoughts with us via our handy Feedback Form, or in our forums area, where we've set up a dedicated thread in our feedback forum for discussion of our review methodology.