Musepack Forums

Musepack Forums (
-   General (
-   -   Comparing rankings of different listening tests (

user 13 April 2006 12:30 pm

Comparing rankings of different listening tests
See the topic at mpc general forum, about: "is mpc better than mp3 ?"

kwanbis made a combined graph by simply putting 2 graphs side-by-side, one graph with mpc (1.14 iirc, maybe even older mpc version) tested in an older 128k multiformat test (mpc ranked there on par with vorbis ca. 4.5 iirc), against the latest 128k multiformat test, which didn't include new version of mpc 1.15v.
Vorbis got 4.7 this time, so the ha crowd starts arguing, ogg would be better than mpc.

I have argued, that it is impossible to make such conclusions by putting those test graphs simply side by side.
(not to mention, that the old graph uses an older mpc version than we have today).

What do say to the validity of using transparency = 5.0 as anchor to compare absolute rankings of different multiformat tests (without including one comparable anchor) ?

Shy 13 April 2006 01:58 pm

1. In the words of Frank Klemm, "1.14 is uninteresting" and it's not as good as 1.15 versions as far as overall as well as low bitrate quality goes.

2. Graphs, those are often uninteresting as well since most people don't even know how to interprate them. Roberto himself got fed up with people's misinterpretations of his 128kbps tests' graphs, where Musepack never went below 1st place. "4.7" or "4.5", the difference is meaningless according to the ANOVA analysis method used, and people don't, and sometimes "won't" understand it.

3. 128kbps, as I've mentioned before, is a senseless and ridiculous bitrate to consider an anchor by which to measure quality of audio codecs. The reason I and anyone concerned with the actual quality of a codec would say so, is that in the range of 128kbps, no codec comes near being transparent on avarage. It generally provides low sound quality, which is not what we're after.
The year is 2006 and still people insist on staying in the buttom of the ocean, tied to an anchor that should never have even been created. It's mainly Fraunhofer's and MPEG's fault, ever since they decided to go with the MP3 format, on which their focus was intentionally and knowingly on low, non transparent bitrates.

The "crowds" are not who we're concerned with. If most people care to focus on low quality sound, good for them, but Musepack's main focus is on high quality, and high quality means transparent quality. Obviously, Musepack also focuses on good quality for non transparent audio, which anyone can test for themselves, and as has been shown in too many public tests.

Today, with Musepack, Vorbis, AAC, the avarage bitrate range to consider a true transparency balance should be around 225-245kbps, and for providing high, mostly transparent quality, around 180-200kbps. Of course, the success of each codec varies at those bitrates with various audio content. Musepack's success is great, whereas others may have similar or less success, but I'm saying what the anchors should be.

You would find that it's hard to get any mention from other codecs' people on what they consider high or transparent quality bitrates for their codecs. This is a rough estimate of ours.

user 13 April 2006 02:30 pm


ad 1.:
Of course, those people, ie. kwanbis, R. Amorim, should know exactly, that the old test contained 1.14, that we have 1.15 for long time,
so there has been developemnt for mpc.
This makes the graph comparison even more invalid.

ad 2.: yes.
Roberto wrote explicitly in that ha topic, that this artificially constructed graph has the purpose to demonstrate in a simplified way, why mpc is inferior to the new ogg, aac....

ad 3.:

well, you and me some others know, that mpc deserves the attention, and know, where transparency starts.

The problem coming for mpc, and I see it already as existant,
those advocates of new aac, ogg, even new lame, they are on a publicity train to write mpc as dead horse, qualitywise!

This started in ignoring mpc in latest 128k test,
(of course, mpc@128 cannot be the goal, though mpc had proven in 2 older 128k tests, that it has top quality even there in comparison with the other contenders),
and now, those guys, who have a leadership of HA's group opinion, use even invalid, unfair, artifically constructed, wrong pseudo-scientific graphs, to "prove", what they are dreaming of, mpc being qualitywise inferior.

I think, if we let above picture slip through, mpc is lost.
I think of qualitywise interested people, who find out about lossy formats, find HA, and will deal with the other formats, not mpc, as "HA" buried mpc.
Buried based on wrong graphs.
The really high quality interested guys will then choose ogg/aac at high bitrates instead of mpc.

I understand fully, that opened these own forums for developemnt and user-support, as the atmosphere at HA (mpc-)-forums was way earlier disturbed by some anti-mpc zealots.

As my native language isn't English and I gave already my reasons in that topic, maybe some more people can write there against the abuse of those graphs.
The most hilarious issue is now the justification of that graph, the 5.0 = transparency level would be enough as comparable anchor (anchor to compare the perfomance of those old and new encoders in old and new tests).
Garf wrote some theory, that it might be possible, to compare those absolute values between independent old and new tests, but it depends on the statistics of course.
IMO, this graph needs to be erased from HA, as the creator kwanbis has not given the maths, with which statistical validity/probability/safety the graphs can be compared.

If we don't attack what happened there, mpc is lost at ha, and then nearly everywhere besides here.
It is format politics what some people over there are doing...

Shy 13 April 2006 02:57 pm

We are well aware of anti Musepack zealotry by people with this or that special interest. Regardless of what Roberto wrote in that thread, in other threads he himself emphasized the codecs are tied. However, he never misses a chance to post "controversial" things against Musepack. In nearly every Musepack thread, the format he allegedly has so little interest in, you'd see interesting comments from him, and others (including developers of other formats or "strong supporters", "surprisingly").

We created this forum long ago to be able to avoid trolling and useless, endless fights as well as provide proper support for our users and people interested in development. No one who regularly trolls in forums such as Hydrogenaudio's dares to come here and post similar misleading, ridiculous stuff. I could say there have been some attempts to turn this forum into another battle ground, which of course had failed miserably.

Our disinterest in participating in discussions filled with trolls has proven to be fruitful and not the least harmful. Musepack use and support hasn't been declining, but only rising. People that have so "articulately" been arguing that Musepack is bad because of this and that, can today do flips and flops when they hear things like you can play MPC files on iRiver, iPod, and various platforms.

This is indeed an opportunity to comment on the decision to not include Musepack in that last 128kbps test. No one has given me or anyone any sensible explanation as to why not to include Musepack, because there isn't any. You can form your own conclusions on this matter. Let me remind again that Musepack has never come less than 1st in any of those tests. Yet it was omitted. The only sensible explanation is this: People want to get rid of the competition. Musepack will always come first, because it is as good and better than the others. It is a sad example of how an "open community" accepts practices that today are sometimes not even used by closed, industry giants.

We don't participate in hopeless fights against masses of people with special interests. We don't try to drive people away from using other formats, as we don't do trolling, and we don't bother commenting in ridiculous threads filled with trolls, because it is hopeless. Anyone who has enough sensibility to check matters, will reach our site, our forum, will download our software and decide for themselves, like the hundreds of thousands who have already.

vinnie97 24 May 2006 01:03 pm


The reason I and anyone concerned with the actual quality of a codec would say so, is that in the range of 128kbps, no codec comes near being transparent on avarage.
I think that's where you're mistaken, because on average and for the average listener, it would seem that 128 kbps is definitely approaching transparency with Vorbis potentially sounding the best at said bitrate (I will admit, statistically tied). And this "near" (IMO) transparency is what makes 128 kbps so attractive (I'm happy with 80 kbps on my Rockboxed Nano even though transparency is in no way being reached...the artifacts just have very little affect on my tin ears and in a lowfi portable solution) for current DAPs, especially flash players, which currently max out at 6GB. For portability and mobility, the higher-bitrate transparency of which you speak is unneccessary and impractical and most likely not detectable by most listeners, especially when environmental noise, sound chip/amp quality and headphone quality are factored in. In the case of HD DAPs, however, I find your higher bitrate more suitable since space limitations are nil and such devices are more commonly used on HiFis.

Even so, 4.79/5 (Vorbis' score on the public 128 kbps test) certainly can't be far from transparency, can it? I think the answer is very subjective and individual but the fact that the test is public and thus indicative of a cross-section of folks bears consideration for possible extrapolation of just what bitrate is reaching transparency (or near transparency if you prefer) on average in 2006.

Back to MPC (sorry, went off on a tangent :P): Proving the victor of greatest transparency at the higher bitrates you mentioned is rather difficult unless you have spectacular hearing and/or are trained in the ways of artifact detection.

Additionally, I know of one such test conducted last year by golden ears himself, Guruboolz, featuring orchestral music only at ~180 kbps with MPC 1.15v. Statistically, the 2 contenders are locked in a tie with Vorbis showing a slight edge. Granted, I know this isn't general music (pop) and that it says very little in the way of the formats' performance at your sweet spot of 225-245 kbps but it would appear that a trend is emerging (which appears in every properly conducted listening test from 2005 and forward that has included MPC, 180 kbps by the Guru, namely...I won't include that aforementioned public 128 test since MPC was omitted). I would actually like to see a public open test with MPC at 1.1v at 128 (granted, I guess I shouldn't expect superiority at this bitrate at this juncture since MPC is intended for greater bitrates) so that the issue can be more succinctly settled (anything above that wouldn't yield any useful results going on how high all the contenders rated at 128 on the last go). Additionally, I'd like to see a Guru test of pop music around 180 kbps as well to further delineate current quality trends.

I think any perceived anti-MPC zealotry you see is more of a backlash to the oft-repeated lofty claims of MPC superiority over the years and still being spouted in some circles. Now that other contenders are measuring up to (and in some cases surpassing) MPC at ~180 kbps (at least) and perhaps below, those claims are no longer being taken at face value and this leaves everyone with more choices.

I will certainly contend that User's objection to the graph comparison is warranted. The comparison says nothing definitive due to the already mentioned anchor problem and questionable data extrapolation.

Shy 25 May 2006 03:12 am

vinnie97, instead of responding to your post as to a normal post, I will present some points free of codec specific zealotry:

1. Since you're very fond of non transparent, low quality low bitrate audio, this is probably not the place for you, nor is Musepack.

2. The 128kbps tests are composed of easy to encode samples that don't sound horribly bad at 128kbps.

3. You think the "higher bitrate transparency of which I speak" is unnecessary? Fine. And "impractical"? Yes, the "extra" ~50 kbps you get with --standard is simply too much for today's limited hardware.

4. "Most likely not detectable by most listeners"? Once you do an extensive research on that, feel free to make that claim. So far, all tests, including MPEG's ridiculous 128kbps tests, show that it's an insufficient bitrate.

5. High quality audio hardware is nice, but if you think "consumer quality" hardware is so poor that it camouflages audio codec artifacts so well, you should recheck your audiophile theories since they are monster cable wrong.

6. "Spectacular" hearing is not needed to spot artifacts at "high" (over 128kbps) bitrates. If you choose to rely on people you consider idols such as guruboolez instead of your own ears, have fun, but don't come claiming nonsense based on tests which no one even cares to verify.

7. "I guess I shouldn't expect superiority at this bitrate at this juncture since MPC is intended for greater bitrates" Interesting that you say that, because every single test shows that you should. "Now that other contenders are measuring up to (and in some cases surpassing) MPC at ~180 kbps" Allow me to correct you - yes, codecs have been showing some improvement (as well as backsliding in some cases with LAME), and no, not a single one surpasses Musepack, which has far less problem cases than any other codec, and as you can see in any public test, is as good or better than the others.

If you'd like to continue pimping other audio codecs, the opinion that transparency is unneeded, or idols which you blindly rely on, feel free to continue doing that, somewhere else.

This thread has covered its subjects and my extensive forum experience tells me a worsening of posting quality is assured, therefore, it's now closed.

All times are GMT. The time now is 07:24 pm.

Powered by vBulletin® Version 3.8.11 Beta 2
Copyright ©2000 - 2020, vBulletin Solutions Inc.