How does Rob run listening tests, which tracks does he use, and how should I run my own?

Two kinds of listening

In October 2014 Watts said there are "specific tests" and "general tests". Specific tests are for "extremely small AB type changes" where he knows which parameter he is adjusting. General tests come in his words "at the end of the cycle", and "it just involves using the unit" Head-Fi, Oct 2014. He added that the results of a specific test are reusable: once he has optimized a filter, he can reuse the same code and "don't need to re-run the listening" Head-Fi, Oct 2014. In a podcast published in October 2026 (recording date not stated) he again described two types of listening: listening to enjoy the music, which he said is not a good way of hearing very small things, and listening tests on very subtle changes, where he assesses each attribute independently Part-Time Audiophile 5:58Part-Time Audiophile 8:21.

The test loop, 2014 to 2022

In October 2014 he described his usual protocol for tiny changes as "to use 30 seconds over and over again" with "3 tracks": "one for depth, one for separation, timbre and vocal, and another for really complex busy music" Head-Fi, Oct 2014. In his March 2016 blog post he described listening "for 20 to 30 seconds" and then going back and forth until he could assess the performance Head-Fi, Mar 2016. In August 2022 he said that, for depth tests on very small differences, he would "listen backwards and forwards many times, just using the first 10 seconds", and that when effects are really small you need to go back and forth many times to get a consistent view Head-Fi, Aug 2022. In an AVstore interview published in August 2022 (recording date not stated) he said that when a change is subtle and close to the optimum, it can take fifty back-and-forth listens to one track to decide which is slightly better AVstore 23:35. He also said in that interview that a specific variable can normally be heard within five minutes across all his tracks AVstore 23:35.

Which tracks, by date

October 2014 (Hugo thread). Three Redbook tracks: an organ piece recorded in a cathedral for sound stage depth; Norah Jones, Cold cold heart, for vocals and timing (the double bass); and Moby, "18 - in this world", for refinement Head-Fi, Oct 2014.

April 2016 (Watts Up blog). He said his depth track "came from an audio company" in a selection of music, and he did not know where the track comes from Head-Fi, Apr 2016. In June 2019 he said his depth track was from an album he linked, but the archived text does not name the album Head-Fi, Jun 2019. In the same month he named the Mussorgsky Pictures at an Exhibition track Gnomus as one he had used "over and over and over" for the subtlest changes in depth Head-Fi, Jun 2019. In July 2020 he called his standard depth piece an organ in a cathedral and said that in a depth test he listens only to depth, asking whether the organ sounds deeper with A than B Head-Fi, Jul 2020.

January 2022 (podcast). He said he uses three tracks for depth, including an organ track and two electronic pieces, plus a fast, busy guitar piece for hearing each plucked note, and a Norah Jones track for timbre variation; he said it is "a track I've used for fifteen, twenty years" Darko.Audio 32:21Darko.Audio 32:52.

August 2022 (Watts Up blog, Head-Fi). A six-track list, written because his mind went blank when asked in a video which tracks he used Head-Fi, Aug 2022. The six, with the attributes he assigned to each: Norah Jones, Cold cold heart, Come Away with Me, track 3 (instrument separation and focus, bass pitch on the double bass); J V A Guillou's arrangement of Pictures at an Exhibition, Gnomus, track 2, Dorian Records (depth, bass pitch); Hildegard von Bingen, O Tu Suavissima Virga, Vox Cosmica, track 1 (timbre variation, clarity); Max Richter, Journey Sequence, Ad Astra, track 8 (depth); Matiegka, Guitar Sonata in B minor, Brilliant Classics (instrument separation and focus, starting and stopping of notes, clarity); Sohn, Ransom notes, Tremors, track 5 (depth) Head-Fi, Aug 2022. As an example of use, to compare a 330 dB and a 350 dB noise shaper for depth he would use tracks 2, 4 and 6 of that list Head-Fi, Aug 2022. In the AVstore interview published four days later (recording date not stated) he said he has "about nine different tracks that I use" and had highlighted six, some for depth only, some for bass pitch, some for clarity and transparency AVstore 21:36.

September 2022 onward. At CanJam SoCal 2022 he said "The actual tracks that I use are pretty exotic" The Source AV 23:43. In June 2024 he wrote that for specific attributes he uses "a fixed set of test tracks", and that his ultimate reference tracks are ones he recorded himself, because he knows what the original performance sounded like Head-Fi, Jun 2024. In an audio-2G interview published February 2025 (probably recorded at Munich High End in May 2024; unverified) he said he has three or four test files for each parameter, and that his test source is always local files on an SSD because with streaming you do not know whether the source has been changed audio-2G 10:31audio-2G 11:57. In an interview published in October 2026 (recording date not stated; its transcript places it before May 2026), describing his DAVE-era noise-shaper tests, he said "in those days I used to use an organ track to test my depth" HiFi Buys 1:19:47.

The descriptions of the Norah Jones track differ by date: vocals and timing in 2014 Head-Fi, Oct 2014, timbre variation in the January 2022 podcast Darko.Audio 32:52, and instrument separation and focus plus bass pitch in the August 2022 list Head-Fi, Aug 2022. These are different descriptions of what he listens for on one track, not a stated change of method.

What he listens for

In March 2016 he listed nine factors to observe: instrument separation and focus, detail resolution, inner detail, sound-stage depth, sound-stage placement focus, timbre, starting and stopping of notes, pitch and rhythm, and refinement Head-Fi, Mar 2016. In August 2022 he said that "Something that has become more important to me recently is best described by the term clarity" Head-Fi, Aug 2022.

His yardstick for whether a change is real is variation. In October 2014 he wrote "The key to evaluating transparency is variation" Head-Fi, Oct 2014. He gave the same reason at CanJam RMAF in October 2017 (video published December 2017), in the AVstore interview published August 2022 and in a podcast published October 2026 (recording date not stated): recordings come with unknown provenance, so he cannot judge them against a known original, and instead asks whether the variation is larger RMAF 7:43AVstore 9:06Part-Time Audiophile 6:40Part-Time Audiophile 7:33. In the AVstore interview he named the variation he listens for as variation in timbre, space, dynamics and instrument separation and focus AVstore 9:51, and he gave similar lists in 2023 and 2024 talks and in the audio-2G interview The Source AV 16:21HIFICLUB 16:20audio-2G 8:58. In the HiFi Buys interview (published October 2026, recording date not stated) he put it as "is this more varied than before" rather than whether it is better than something else HiFi Buys 48:37.

He treats a brighter sound as a warning. In March 2016 he wrote that noise floor modulation can give a false impression of more detail resolution Head-Fi, Mar 2016. In a November 2018 open-day talk (uploaded December 2018) he said that checking timbre variation will show when detail seems better but timbre variation is worse, "and that will be a red flag to tell you something else is going wrong" Pursuit Perfect System 25:55.

He values tests where nothing changes. In 2014 he wrote "Null results with specific testing is good" Head-Fi, Oct 2014. In March 2016 he wrote that "my listening tests are at their best when I can hear no change when adjusting a variable" Head-Fi, Mar 2016. He repeated the idea in October 2017 and March 2024 transcripts miceblue425 34:45MFG Audio Reviews 54:35.

Conditions and rules he gives

His March 2016 blog post (Watts Up, Head-Fi) is the fullest list. It tells the listener to be a neutral observer with no expectations, to "Minimize variables" so that only one is changed in a specific test, "Don't listen to your wallet", "Don't listen to the brand", and to ignore what the product looks like Head-Fi, Mar 2016. It says "Do blind listening tests": if unsure, or wanting confirmation, run a single blind test where the other listener is told to listen to A or B, with no leaked expectation and no request for value judgments, just a description of the sound Head-Fi, Mar 2016. It warns that abilities change: "more than 4 hours of structured AB listening tests means I lose the desire to live", unusual circumstances reduce sensitivity "by an order of magnitude", and judgments at shows should be treated with care Head-Fi, Mar 2016. It also says "Don't listen with gross errors": he calls it perhaps only appropriate for a design process, but pointless to run listening tests when there are measurable problems Head-Fi, Mar 2016. He added that these tests "are by no means perfect" and that listening tests are "very hard and extremely easy to get wrong" Head-Fi, Mar 2016.

Volume matching. In January 2021 (Hugo TT 2 thread, in a post about its XLR and single-ended outputs) he wrote that the most likely explanation for some reported differences was that people were not turning down the volume by 6dB when doing the comparisons, adding "listening tests must be volume matched" Head-Fi, Jan 2021. In the same week he said he spends lots of time making sure the test is volume matched and set up correctly, and that with subtle changes a test can take several hours Head-Fi, Jan 2021. In May 2017 (DAVE thread), before the Hugo M Scaler existed, he gave the insertion loss of the M scaler then in development (the post compares 0.5M and 1M tap prototypes and a video mode) as 2.76 dB, so "you need to increase the volume by 3 dB to volume match" Head-Fi, May 2017. In May 2018 he said his Hugo TT 2 versus Hugo 2 comparison was volume matched Head-Fi, May 2018.

Cable placement and setup. In April 2016 he found that a preference between two USB cables on DAVE came down to the headphone cable touching the USB cable, and wrote "Then I realised I had made a school boy error" Head-Fi, Apr 2016. In August 2018 he reported the same kind of inconsistency with BNC cables on a Hugo 2 until the headphone cable was kept away from the digital cable Head-Fi, Aug 2018. In September 2018 he wrote that he had inconsistency on AB tests of digital cables that came from cable placement, and advised keeping digital and mains cables away from analog cables Head-Fi, Sep 2018.

Stress. In December 2018 he wrote that "stress when you are doing an AB tests is a sensitivity killer" Head-Fi, Dec 2018.

Session length and repeats. In the AVstore interview (published August 2022) he said that after about four or five hours of listening to his tracks you lose the will to live, and that a specific variable can normally be heard within five minutes AVstore 22:47AVstore 23:35. In September 2023 he wrote that at a 100ppm change in a WTA variable two AB listens on each track can reach a consensus, whereas at 10ppm "sometimes you have to go back and forth 50 times to get a consensus" Head-Fi, Sep 2023. A few days later he said that after about 4 hours of very small AB differences you need to stop Head-Fi, Sep 2023.

Equipment. In November 2019 he said that in the past he never used headphones for depth tests, only his main loudspeakers (B and W 803 D3), but since the Aeons he uses them for depth too, and that if a test is inconclusive and he suspects the headphones he repeats it on the loudspeakers Head-Fi, Nov 2019. In January 2022 he said most of his recent listening was on headphones, while loudspeakers still give better sensitivity and accuracy Darko.Audio 30:28. In April 2022 he said that where possible he builds the listening options into the FPGA so he can use a bit switch, so only the variable under test changes Head-Fi, Apr 2022. In the HiFi Buys interview (published October 2026, recording date not stated) he described the same practice of one FPGA design with a toggle switch between two noise shapers, so power consumption, operating conditions and place and route stay the same HiFi Buys 1:19:04HiFi Buys 1:19:38.

Blind testing

2014. "I don't usually do it blind - except for when I am hearing something extraordinary, something that makes no sense, then I will do single blind listening tests in order to confirm something really unusual" Head-Fi, Oct 2014. Later that month he wrote that when a result is extraordinary he re-tests single blind, and if the observer under blind conditions comes back with identical impressions, "you know that what you are hearing is real" Head-Fi, Oct 2014.

2016 to 2018. The March 2016 blog advises single blind tests for confirmation, and says double blind testing where stress is put on listeners can degrade sensitivity by two orders of magnitude Head-Fi, Mar 2016. In October 2017 he said the only time he gets other people to listen is when something enormous or with extraordinary consequences has happened; his example was a 1980s mains cable comparison where his girlfriend took a blind test that "Wasn't double blind, it was blind" miceblue425 53:06miceblue425 53:52. In reply to a questioner at the same session he said he has big problems with double blind testing, because once people are confused about what they are listening to they get stressed and sensitivity falls apart miceblue425 1:00:28miceblue425 1:00:34. In September 2018 he wrote that, as another poster had pointed out, a single blind test with A and B identified neutrally only as A or B is very powerful because bias is then eliminated, and said he often does this when a test reveals something extraordinary Head-Fi, Sep 2018. The same month he wrote that his listening tests "are about proving to me that something is audible and important", not about proving it to other people, and that when something extraordinary comes along he performs single blind tests Head-Fi, Sep 2018.

2020 to 2023. In April 2020 he described a single blind test with his son on a high-frequency filter, in which the son identified it as warmer Head-Fi, Apr 2020. In May 2021 he described a blind test with his son on interpolator aliasing, in which the son's description agreed with his own Head-Fi, May 2021. In December 2021 he wrote "when I get extraordinary results I run blind tests with my son" Head-Fi, Dec 2021. In April 2022 he wrote that his listening tests "would be considered anecdotal and not scientific evidence; but my job is to close the gap from live to recorded sound, and the tests I do are extremely carefully done and objective…" and that for results that defy explanation he does a single blind test with disinterested parties; he had not published his results, for example in the AES Head-Fi, Apr 2022. In September 2022 he wrote that blind testing has its place if the components are neutrally identified, but "the minute it's unidentified, then sensitivity falls apart" Head-Fi, Sep 2022. In another September 2022 post he wrote that you do not need double blind listening tests to hear things that "hit you in the face" and that the problem is the interpretation of sound quality changes Head-Fi, Sep 2022.

2023 onward. In January 2023 he wrote "On the design/development side the listening tests are just me", with blind tests usually with his son Daniel when something defies rational explanation, and that Chord conducts its own tests when a product is close to production Head-Fi, Jan 2023. In 2023 and 2024 interviews he said his son is trained to describe the sound in the same terms, is not told what he is listening to, and "always" or nearly always gives the same description he had reached Eric's Hi-Fi 35:03Eric's Hi-Fi 35:38The Source AV 35:31HIFICLUB 21:57. In the HiFi Buys interview (published October 2026, recording date not stated), about the noise-shaper improvements he had just described (from 200 dB to 220 dB and beyond), he said that when other listeners do not know what they are hearing they report the same depth difference HiFi Buys 1:22:59. In March 2023 he wrote that subjective assessment "needs to be done objectively", with conclusions evaluated "either by repeated (blind) listening tests or through repeated tests sometimes many years later" Head-Fi, Mar 2023. In November 2023 he added that there is always a possibility of error, that he repeats tests and reappraises, and that "Absolute certainty with subjective testing (no matter how objectively done) is an illusive thing" Head-Fi, Nov 2023. In March 2023 he said he has made listening test mistakes in the past and re-evaluates old conclusions Eric's Hi-Fi 34:16.

The engineer anecdote. He told several times of a listening test with a non-audiophile engineer from a chip company, comparing two noise shapers of 180 dB and 200 dB performance; the engineer was surprised at how easy the difference was to hear Head-Fi, Mar 2016RMAF 4:45The Headphone Show 39:27. In September 2018 he said he conducted that test single blind Head-Fi, Sep 2018. The 2016, 2017 and 2023 accounts cited here do not say whether it was blind.

Who listens

In February 2018 he said that in design it is just him doing his listening tests, with his sons sometimes involved MajorHiFi 21:37. In October 2017 he said "I'm designing for me" and that only he can answer what is best miceblue425 50:04. In October 2022 he wrote that if his hearing declined his son could take over his objective listening tests, as he has good sensitivity and has been trained by him Head-Fi, Oct 2022. In March 2023 he said it is often very good to get a non-audiophile to listen to your gear Eric's Hi-Fi 39:05. He also said in March 2023 that he ignores forum comments that say only that something sounds better, but takes seriously ones that describe in detail what is better and what is worse Eric's Hi-Fi 36:27. In the same audio-2G interview, asked whether he conducts his WTA algorithm tests himself or relies on a panel, he answered "No, just myself" audio-2G 10:19audio-2G 10:24. In September 2024, about Quartet, he said that after years of small changes he does not get a good feeling of scale, and that plugging it into somebody else's system and getting their reaction gives the real picture The Source AV 14:19.

Judging scale and musicality

In October 2014 he gave a rule of thumb for scale: if you do not notice an improvement the next day it was pretty small; still hearing it after one week means substantial, after a month very substantial, and after three months a very rare breakthrough Head-Fi, Oct 2014. In August 2022 he repeated the idea: if after a day you have gotten used to an improvement it is a small change, and if after 3 months you are still hearing improvements it is something huge Head-Fi, Aug 2022. In the same post he said the AB tests are perfect for very subtle changes, but "they won't tell you if it's more musically enjoyable", which he tests by finishing the module and listening normally Head-Fi, Aug 2022. The musicality check is slow: "many days or weeks" in March 2016 Head-Fi, Mar 2016, "several weeks" in the HiFi Buys interview (published October 2026, recording date not stated) HiFi Buys 44:25. In December 2019 he said blind tests are not very sensitive and are useless for determining musicality, for which, about the M scaler, he said simply plug it in and keep it if you enjoy your music more Head-Fi, Dec 2019.

If you want to try it yourself

Watts's own suggestions to owners are narrower than his design process. In December 2019 he told an owner choosing between two options to use a test track that has good depth ("I use an organ track"), and that the option with the best depth plus the warmest tone is the more accurate one Head-Fi, Dec 2019. In April 2019 he told an owner to try before buying, be aware of placebo, and make sure to do thorough and careful AB listening tests Head-Fi, Apr 2019. In March 2022 he wrote that placebo is very real, that you must be a completely neutral observer, and that you should analyze all aspects of sound quality and be prepared to repeat listening tests Head-Fi, Mar 2022. In October 2018, about the M scaler, he wrote that because the listener is the biggest variable, "always try to listen first", with enough time to see how much it changes your musical enjoyment Head-Fi, Oct 2018. In August 2023 he told an owner comparing PGGB with the M scaler to re-listen to some tracks and compare low frequency pitch ("double bass is good for this"), instrument separation and focus on a busy track, and timbre variation on a track with sax and trumpet Head-Fi, Aug 2023.

What he has not said

The archive does not contain a step-by-step protocol written for other listeners beyond the points above, trial counts or pass criteria for blind tests, a statement endorsing ABX or double blind testing, or published data from his own listening tests; the nearest statements are that his tests are unpublished and would be considered anecdotal Head-Fi, Apr 2022. In December 2024 he announced a seminar on how to objectively assess sound quality Head-Fi, Dec 2024, but the archived post gives only that introduction, so the seminar's content is not in this archive.

Quotes

“I don't usually do it blind - except for when I am hearing something extraordinary, something that makes no sense, then I will do single blind listening tests”2014-10-12 Head-Fi, Oct 2014

“listening tests must be volume matched”2021-01-20 Head-Fi, Jan 2021

“listen backwards and forwards many times, just using the first 10 seconds”2022-08-15 Head-Fi, Aug 2022

“On the design/development side the listening tests are just me.”2023-01-04 Head-Fi, Jan 2023

Dated positions

Caveats and open points

• His track descriptions differ by date and venue (2014 forum post, 2022 podcast, 2022 forum list); the archive does not show whether the lists describe the same set or a changed set.
• The 2016 and 2017 accounts of the engineer's noise shaper test do not say whether it was blind; the September 2018 post says it was single blind.
• The German Hifistatement interview (RWA-20160129) is the publisher's translation and is not quoted here.
• The December 2024 post announcing his objective-assessment seminar is archived only as an introduction, so the seminar content is not covered.
• His blind tests are described as single blind, mostly with one trained listener who is his son; he calls his tests anecdotal and has not published results.
• Advice to owners in this archive is scattered across replies (2018 to 2023) rather than a written protocol for home listeners.

Topics: Listening test methodology, Recordings, mastering and the production chain