Our approach to testing things like sound quality is to use scientific testing methods. While many other testers rely on "golden ears" (those with good hearing who subjectively judge headphones), we use proven scientific test methods developed by industry experts. This means that we can not only quote specific numbers for tests (instead of saying that the music sounded bassy, we can quote a specific frequency response), but that we can run exactly the same tests and get the same results year by year.
Audio performance is the most important thing we test on headphones, so we put a lot of time and effort into testing the quality of the sound that headphones produce. We evaluate the electroacoustic performance of headphones (meaning the details of the sound produced) objectively using a test system from Listen, Inc which is compromised of a SoundCheckTM, electroacoustic test system, and a Brüel & Kjær Head and Torso Simulator (HATS). This enables us to do precise, scientific testing of headphones, rather than the subjective testing that is often used. These components of our testing system are described in more detail below.
Our Testing Rig
SoundCheck
 SoundCheck, from Listen, Inc. is an electroacoustic measurement and analysis package widely used for testing audio devices, both on the production line and in R&D applications. It is a PC and sound card based system which communicates with both analog and digital equipment using standard, non-proprietary interfaces.
The Brüel & Kjær hardware is directly controlled through SoundCheck, with the test sounds passing from the soundcard to the high-end audio amplifier we use to drive the headphones (a Crown D45). Tests are fully programmable, enabling the sound signal, various analysis methods and result output format to be selected.
 Head and Torso Simulator
The Head and Torso Simulator (HATS) is a mannequin with built-in ear and mouth simulators that provides a realistic reproduction of the acoustic properties of an average adult human head and torso, including how it absorbs and reflects sound. It incorporates an artificial mouth and ear with pinna (the visible part of your ear that acts to gather sound). We mount the headphones on the HATS in the same way that they are mounted on a human head, and the sensitive microphones in the ears of the HATS pick up the sound from the headphones and feed this back to the SoundCheck system through two high-end microphone amplifiers. The entire system is calibrated and tested on a regular basis.
The testing rig.
Our Tests
Our audio quality tests are broken down into 6 sections: frequency response, distortion, tracking, sound pressure level, isolation and leakage. Below, we describe the tests we do for each of these sections in turn.
Frequency Response
Frequency response describes how a set of headphones emphasizes different frequencies. To test this, we first put the headphones on HATS, and make sure they have a proper fit. We then use SoundCheck to send a frequency sweep through the headphones, which covers a range from 20 to 20,000Hz. This frequency range covers all the frequencies the human ear can hear. The sound is played through the headphones and into HATS' precision microphone ears. HATS records the playback and sends the data back to SoundCheck. Now we can compare the original sound file to what HATS recorded to determine how the headphones have altered the original sound file.
This process produces a graph like the one below, which shows the frequency response of the headphones from low frequencies (at the left) to high frequencies (at the right). The two lines are for the left (in green) and right (in red) channels of the headphones. The response curve we feature includes a diffuse free field correction, which corrects for the effects of the head, torso, pinna and ear canal on the response curve.
There is no ideal response for headphones, so there is not a single response curve that we test against. Instead, we set a range of +/- 6 dBSPL in the frequency range of 500Hz to 9Khz, and our testing system puts these these limits against the response curve of the headphones. Our scores are then based on how much the headphones go outside these limits, so we are not scoring on the exact curve, but rather on the smoothness of the curves. Although headphones are often regarded as being a matter of personal taste, a good pair of headphones should produce a good, clean frequency response that does not overly exaggerate or diminish nearby frequencies. We're looking for the headphones to accurately reproduce the music you're putting through them.
We test the headphones frequency response at an average level of 78 dbSPL, measured at 1KHz. This represents a typical listening level for someone listening in a quiet room or other low-noise environment.
Distortion
Distortion is a measure of how accurately the headphones reproduce the waveforms that make up the music you listen to; if they clip, compress or otherwise mangle the waveform, your music won't sound the way it should. Using the SoundCheck system, we measure something called Total Harmonic Distortion (THD), which relates to how well the headphones reproduce not only the fundamental frequency itself, but the higher frequency harmonics that accompany it. A decent set of headphones should be able to accurately reproduce both the fundamental frequency and the harmonics at the same level as they are present in the original sound.
Our testing system measures the THD of headphones at 90 dBSPL and produces a graph like the one below. On the graph, you can see the THD for both channels: the left channel is the green line, and the right is in red. Big peaks in this graph indicate that the headphones are having a particular problem with the harmonics of that frequency: notes at that frequency may sound buzzy or rough, like a distorted guitar. THD is measured as a percentage, indicating how much the original waveform is distorted. The bottom line is that lower is better here: you don't want either the red or green lines to peek above the zero line.
Our distortion score is based on the total amount of distortion across the frequency range; the higher the graph (and any peaks on it), the lower the score. However, it is worth remembering that in the complex assortment of frequencies that is music, you are very unlikely to be able to detect low levels of distortion. The distortion won't be particularly noticeable until around the 3% THD mark.
Tracking
Headphones have two channels, one for each ear. Both of these channels should sound the same, so we test this to make sure. Our testing system analyzes the sound both channels produce and creates a graph to show any differences between the two as a percentage of the frequency response. Again, we're testing the audible range, from 20 to 20,000Hz.
If both channels were producing the same sound, the blue line would stay along the zero line. When the left channel is louder, the blue line will rise above zero; when the right channel is louder, the blue line will dip below zero. A difference of a couple of percentage points here isn't a problem, but we don't expect to see anything above 4 to 5 percent; that could make the sound feel unbalanced and unpleasant.
Our score for this test is based on the maximum imbalance between the two channels; the bigger the peak or valley on the graph, the lower the score.
Maximum Usable Volume
In this test, we examine how high we can increase the volume on the headphones before they reach a peak THD of 3%. The SoundCheck test sequence gradually increases the level of the stimulus and analyzes the THD until it reaches 3 percent, or the level of the sound reaches 120 dBSPL. We don't test any higher than 120 dBSPL, because that's the ceiling of safe headphone use: anything higher could cause permanent hearing damage with long term exposure.
Isolation
In many situations—especially if you're looking for portable headphones—you'll want to block out the noise of the outside world. Public transit, air travel, traffic noise, or obnoxious roommates can be distracting and interrupt your music. We therefore test how much external noise the headphones can block out. We do this by placing a speaker next to the HATS and playing pink noise, which has equal power across the entire frequency range.
SoundCheck measures and records the noise levels. We next put the headphones on HATS and blast both with noise again. Afterwards, we run the test again with active noise cancellation turned on (if applicable). Once we're done, SoundCheck compares the three curves: without headphones, with headphones, and with active cancellation on. We subtract the latter two curves from the former, which produces a graph like this:
This shows how much of the sound was blocked; the higher the line, the more sound was prevented from reaching the ears.
In the case of headphones that include active noise cancellation (such as the Bose QuietComfort 15s and AblePlanet Clear Harmony NC200s), we feature a graph with two lines; one for the active noise cancellation disabled (the green) and one for the noise canceling enabled (the blue).
Our score is based on how well the headphones block the noise; the more they block, the better the score. With active noise canceling headphones, we use whichever test provides the best results as the basis for our score. The best results are almost invariably with the active cancellation enabled, but the feature often creates just as much noise as it cancels out.
Leakage
If you're trying to block out the world by listening to music, it's only polite to return the favor: the person sitting next to you on the bus wants to hear their own conversation, not your music. Leakage is also an important figure to know if you're planning on being in a quiet public place, like a library or museum.
Our test uses a senstive microphone, placed a set distance away from HATS. We first take a measurement of the ambient noise of the room. Next, we measure the sound level with about 90 dBSPL of pink noise playing back over the headphones. We then calculate the difference between these two levels; the bigger the difference, the more the headphones contribute to the aural pollution of our world. Our scoring for this test is based on this difference: the more sound the headphones leak, the lower the score.
We feel our comprehensive set of tests is the most thorough and comprehensive in the industry. It also allows us to do direct comparisons of headphones, even if they are not tested at the same time.
-  |
