Musepack Forums  

Go Back   Musepack Forums > Main > General

Reply
 
Thread Tools Search this Thread Display Modes
Old 08 August 2005, 09:10 am   #1
pieroxy
Member
 
Join Date: Aug 2005
Posts: 5
Default Listening tests

On the main page, I read: "Musepack is always at the top". And then a few links are provided to listening tests, which shows that:
A. Musepack isn't always at the top;
B. These tests are heavily biased since thay are supposed to compare equivalent bitrates. On one of them, MPC is around 146kbps where most of the rest is 128kbps. Any codec would sound better than the rest with a bitrate almost 15% higher...

Just curious if this page is just out of date or something....
pieroxy is offline   Reply With Quote
Old 08 August 2005, 10:26 am   #2
Lefungus
Procrastinator
 
Lefungus's Avatar
 
Join Date: Jul 2004
Posts: 131
Default

A. It's always at the top or tied at the top with another codec (except for some tests where you can't say anything apart from 'it seems all codecs are better than Xing')
B. Those tests aren't supposed to compare equivalent bitrates on tiny samples

So you need to:
A. Read those tests again, and, that's important ! read how to interpret those results instead of jumping to conclusions
B. Read those tests again, and, that's important ! read why instant bitrates on small samples isn't important compared to average bitrates on full albums

After you've done all that, if you still think those tests are wrong, goto hydrogenaudio forums and use the search button. Lots of people already talked about that here.
Lefungus is offline   Reply With Quote
Old 08 August 2005, 11:22 am   #3
pieroxy
Member
 
Join Date: Aug 2005
Posts: 5
Default

Thanks for your answer. I know MPC is close to the top every time, but I was being picky here... It's not ALWAYS on top! ;-)

As far as the bitrate is concerned, I understand that the bitrate on a tiny sample might vary even though it might very well be 128 on a full album.

But it still remains that in ALL the tests (but the 2nd one, couldn't find infos on the bitrates there) MPC is above 140kbps where all the other codecs are closer to 128kbps. So as far as these tests are concernede, the proper conclusion to draw would be : "MPC@140 is always on top vs Others@128". Note that Ogg also errs on the same side.

And if with the same parameters it averages out at 128kbps on a typical full album, then why choosing samples that consistently gives 140+ kbps?

The fact is and remains, I have yet to see a listening test with MPC on top at the same bitrates than the others. Your argument just doesn't satisfy me...

Now don't get me wrong, I like MPC (and Ogg and others). But it's been a few years I've been reading about these listening tests, and I still don't get it...
pieroxy is offline   Reply With Quote
Old 08 August 2005, 05:17 pm   #4
Lefungus
Procrastinator
 
Lefungus's Avatar
 
Join Date: Jul 2004
Posts: 131
Default

The samples used in such tests usually try to catch *difficult* parts for audio codecs. MPC is VBR and try to use as much bitrate as necessary. Difficult audio parts use more bitrate than easy audio parts. And so the instant bitrate is higher although average bitrate is close to 128 (for example)
Lefungus is offline   Reply With Quote
Old 08 August 2005, 07:38 pm   #5
pieroxy
Member
 
Join Date: Aug 2005
Posts: 5
Default

I understand that. It shows clearly that MPC is allocating more bits to the difficult parts. This is all well and good but for the fact that allocating more bits in a part of the music comes at one cost: Having less bits for the rest of the song.

Of course, MPC's algorithm that detects these complex parts is allocating more bits there than any other codec. Naturally, MPC sounds better in these parts, but what of the other parts? the parts where LAME would allocate 128kbps and MPC only 110kbps?

I guess the idea is to say that a lower bitrate there would be less audible than on the complex parts, hence justifying the choice of allocating a higher bitrate on the more complex parts. But this is just speculation, and I have yet to see a test which would back these assertions up. This is all I am saying.

By testing only the difficult parts, all these tests do favor MPC by highlighting its strong point. By doing a test in a less difficult portion of a song, one would assert that this strong point doesn't comes at too great of a cost.
pieroxy is offline   Reply With Quote
Old 08 August 2005, 10:07 pm   #6
xmixahlx
Musepack developer
 
xmixahlx's Avatar
 
Join Date: Nov 2004
Location: seattle, washington usa
Posts: 111
Send a message via ICQ to xmixahlx
Default

Quote:
Originally Posted by pieroxy
I understand that. It shows clearly that MPC is allocating more bits to the difficult parts. This is all well and good but for the fact that allocating more bits in a part of the music comes at one cost: Having less bits for the rest of the song.
dude, you're smoking crack. musepack is quality-based NOT bitrate-based.
Quote:
Of course, MPC's algorithm that detects these complex parts is allocating more bits there than any other codec. Naturally, MPC sounds better in these parts, but what of the other parts? the parts where LAME would allocate 128kbps and MPC only 110kbps?
again... crack
Quote:
I guess the idea is to say that a lower bitrate there would be less audible than on the complex parts, hence justifying the choice of allocating a higher bitrate on the more complex parts. But this is just speculation, and I have yet to see a test which would back these assertions up. This is all I am saying.
how do you have such strong opinions based on bull shit? what tests are you waiting for? have you been absent for the last 4 years of musepack development? do you know what you are talking about?
Quote:
By testing only the difficult parts, all these tests do favor MPC by highlighting its strong point. By doing a test in a less difficult portion of a song, one would assert that this strong point doesn't comes at too great of a cost.
musepack's strong point (in relation to transparency) is that it is completely quality-based and has no regard for bitrate (file size, etc.). in addition, "problem samples" are used in listening tests to highlight easily detectable deficiencies (re: possible deficiencies), and a large range of real music is used as a comparison of bitrate management.


later
__________________
-xmixahlx, the one they call "mike"
http://xmixahlx.com -|- http://rarewares.org
xmixahlx is offline   Reply With Quote
Old 09 August 2005, 06:00 am   #7
pieroxy
Member
 
Join Date: Aug 2005
Posts: 5
Default

xmixahlx, Thanks for your interestin the thread. Maybe you should read the beginning of the thread so you would get an idea of what we are talking about. I'm going to try to summarize it for you:

I was asking: 3How come in all the listening tests MPC is around 140-146kbps and MP3 at 128kbps. How is that a fair comparison?
Lefungus answered: bitrates on tiny samples are irrelevant, only average bitrate (on full albums for ex) is relevant.
I countered: Then why doing tests only on the tiny samples that are 140+kbps? How is it fair vs. MP3@128?
Lefungus answered once more: The tests are supposed to test difficult parts, hence the tendency of MPC to allocate more bits there, where it would average 128kbps on a large sample.
Then, I said: Well, but if the average is 128kbps and the hard parts are 146, then other parts "easy parts" must be around 110kbps. How come there is no test on these parts, where MPC could very well sound worse than MP3 since it is using a lower bitrate?

So, basically, you answered there:
dude, you're smoking crack. musepack is quality-based NOT bitrate-based.
1. My smoking habit is relevant only to me
2. Thanks for the info. You will notice that I know that and I was not the one that suggested that MPC would average at 128kbps on a large sample (with the encoding test parameters), so please don't get mad at me!

again... crack
Tell me then, how can MPC average 128kbps on an album, then allocate higher bitrates on hard parts without allocating bitrates lower than 128 on some parts of the music? I am apparently not the only one smoking crack over here.

how do you have such strong opinions based on bull shit?
You will notice that my sentence was starting with "I guess", hence not referring to a 'strong opinion' as you put it, but to a guess. It is my guess that quality based means allocating high bitrates to complex parts and low bitrates to the less complex parts. Hence, with parameters that will give an average of 128kbps (for example) some parts will be over 128kbps and some parts under. But these parts were 'simple parts' so they did not require 128kbps to sound as good as the more complex parts.

So far, I don't think anyone will disagree.

Now, comparing such a codec to a CBR 128kbps MP3 is highly unfair if you compare only the complex parts. Of course MP3 will sound worse: It is allocating less bits per second (I do understand there are other factors coming into play here). A fair comparison would compare complex parts and simple parts, just to prove that the quality based algorithm doesn't degrade the simple parts (encoded < 128kbps) too much.

do you know what you are talking about?
Yes, I do. Comparing a Codec at 146kbps and a codec at 128kbps is just nuts. It has little point, let me try to explain:

I could write a codec (assuming I had the time and skill to do it ;-)) that would detect the complex parts of the music. It would encode these parts of the music at 320kbps. Then, on the other parts, it would just produce a 32kbps crappy piece of music that would sound horrible. By comparing 'problematic samples' as all the 4 tests listed in the front page does, my codec would be on top of every single one of them. The fact that the non complex parts are just horrible wouldn't be covered by the tests.

My codec's strong points would be to allocate maximum number of bits to complex parts. Since these tests all focus on this point, they would miss the crappy part of my codec and they would be highly unfair to the other codecs.
pieroxy is offline   Reply With Quote
Old 09 August 2005, 01:09 pm   #8
ak
Guest
 
Join Date: Sep 2004
Posts: 13
Default

Quote:
Originally Posted by pieroxy
I was asking: 3How come in all the listening tests MPC is around 140-146kbps and MP3 at 128kbps. How is that a fair comparison?
All? I see only one.
Besides you'll have hard time making musepack respect your bitrate plan.

> Then why doing tests only on the tiny samples that are 140+kbps? How is it fair vs. MP3@128?

I suggest coming at ha.org and state that sample selection (and test organisation in general) are plain wrong and unfair. :wink:

> Well, but if the average is 128kbps and the hard parts are 146, then other parts "easy parts" must be around 110kbps. How come there is no test on these parts, where MPC could very well sound worse than MP3 since it is using a lower bitrate?

Noone showed enough interest/effort to make one, perhaps?

From homepage one can conclude: 128k isn't really bitrates where mpc shines, yet it can compete and doesn't loose to any other tested codecs in that range. Few tests are linked, to prove the claims. If there are others tests, showing opposite, plese share.

Really it started as you wasn't happy with wordings on homepage, now there's tests that unfair and lack of tests revealing supposed suckiness at low bitrates. Why not conduct some then, if you feel there's a need?
ak is offline   Reply With Quote
Old 09 August 2005, 06:37 pm   #9
Seed
Musepack Nanny
 
Seed's Avatar
 
Join Date: Jul 2004
Posts: 168
Default

While I enjoy the spirit of the thread, the debate should not derail to personal insults. I'd like a constructive conversation where both sides exchange information and test results. Discussion of Musepack's excellence or lack of it at ~128 kbps could not interest me less, but since we're on the subject:

1. Allocating more bits for a particular short section of a song does not mean that Musepack's encoder will also allocate much more than 128 for the entire song the short sample is from, so the guys are right. It's not indicative of the music it's taken from.

2. If the bitrate for a short sample is significantly higher than what the other encoders used, pieroxy has a point too.

Summary: Let's create a test ourselves. If the tests linked to from the main page don't seem accurate or fair, a new one should be made by those who are not partial to one format or another.
Seed is offline   Reply With Quote
Old 09 August 2005, 06:41 pm   #10
xmixahlx
Musepack developer
 
xmixahlx's Avatar
 
Join Date: Nov 2004
Location: seattle, washington usa
Posts: 111
Send a message via ICQ to xmixahlx
Default

i have read this thread.

you aren't grasping (1) how listening tests work - e.g. roberto's listening tests (2) how pure VBR codecs work (3) how musepack bitrate management works.

(1) ^^ see previous post.

in addition: to compare LAME VBR @128 and musepack VBR @128, a wide range of real music is used to tune the encoder to an acceptable parameter, then this parameter is used during the test.

this is important for you: musepack is able to use greater bits in problem samples more efficiently than other codecs (the situation is similar to ogg vorbis and AAC)

(2) pure vbr codecs use a quality scale and throw more bits at harder to encode music. this does not mean that the music that uses less kbps is inferior compression within the same quality scale. it is more of the opposite, as a higher quality setting allows for the harder-to-encode areas of the source to be encoded at a higher bitrate.

(3) musepack bitrate management utilizes a dynamic scale, and is not limited to e.g. LAME's bitrate management, making it a more effecient encoder

what i really would like to see, is you conduct a listening test highlighting only the SIMPLEST music to encode, like, lets say silence - and we will know for sure the true silence encoding champ!


later
__________________
-xmixahlx, the one they call "mike"
http://xmixahlx.com -|- http://rarewares.org
xmixahlx is offline   Reply With Quote
Old 09 August 2005, 06:46 pm   #11
xmixahlx
Musepack developer
 
xmixahlx's Avatar
 
Join Date: Nov 2004
Location: seattle, washington usa
Posts: 111
Send a message via ICQ to xmixahlx
Default

apologies regarding personal/sarcastic comments
__________________
-xmixahlx, the one they call "mike"
http://xmixahlx.com -|- http://rarewares.org
xmixahlx is offline   Reply With Quote
Old 09 August 2005, 11:20 pm   #12
guruboolez
Member
 
guruboolez's Avatar
 
Join Date: Aug 2004
Location: Strasbourg (France)
Posts: 13
Default

pieroxy is not completely wrong. The latest 128 kbps include samples supposed to be difficult. Bitrate allocation is therefore logicially superior to 128 kbps (it's not a problem, and it doesn't biased the test).
But one question remain: how will perform MPC on musical parts supposed to be less difficult (bitrate allocation < 128 kbps)? Theoretically, it shouldn't be a problem (it was said: mpc is quality based). In in practice, things are nevertheless different. This is not "crack": in the same test giving MPC on first place, there were only two samples for which bitrate drops to an unusual value (<100 kbps): Debussy.wav and ItCouldBeSweet.wav. And for both samples mpc quality was also unusually lower:





I wouldn't call "bullshit" or "crack" pieroxy questioning about quality for MPC on low complex part. It's really pertinent, and there are samples and tests (debussy for example) which proves that MPC obviously fails on musical part supposed to be easy-to-encode. You can also find the same questioning on HA.org, here.

These two samples raise the question about a possible methodological problem with this test. If MPC has problems on "non-difficult" samples [I don't like this concept], and if the test only include "difficult" one, could we still say that samples are representative? Note that it doesn't only apply to MPC, but to all VBR encoders having sudden bitrate+quality diminution at a given VBR setting; LAME MP3 is therefore in the same situation (same problem with Debussy and even higher with ItCouldBeSweet).

From my experience, MPC VBR mode is unstable at mid/low bitrate. Bitrate could suddenly drop and quality will follow the same curve. But with some other samples, the lower bitrate hasn't a big impact on quality. My feeling is that a listening test including samples divided in two equal parts (one for bitrate < 128 kbps and the second one for bitrate >128 kbps) would probably be less favorable to MPC. But how much is something I can't say.
guruboolez is offline   Reply With Quote
Old 10 August 2005, 04:01 am   #13
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

First of all, Musepack is not the only format having a bitrate higher than "128kbps" on test samples presented in the various tests. Other codecs do too, and Musepack definitely doesn't always have the highest "bitrate".
guruboolez: You sure have lots of theories and conclusions. If you feel like making them, do that in a seperate thread, along with something that substantiates them. If you feel like making suggestions on how to make tests, do that in another thread too.
I don't see how Musepack's quality was "unusually lower" with any sample either. That can't be said about other codecs however. And that too doesn't mean much since the overall performance is what matters in the end. People should take note of the numbers more than the places on the graphs. Also, taking those two results out of the rest means very little to anyone if they can't see the rest.

Note: Please refrain from embedding images from pages you have no control of and link to them instead, or better, to the page containing them.
Shy is offline   Reply With Quote
Old 10 August 2005, 04:41 am   #14
xmixahlx
Musepack developer
 
xmixahlx's Avatar
 
Join Date: Nov 2004
Location: seattle, washington usa
Posts: 111
Send a message via ICQ to xmixahlx
Default

guru -

i know, in the past, you have done tests on musepack which highlight deficiencies - can you make this another thread on this board so we can address this without relation to this current thread?

regardless of situations where musepack is inferior with low bitrate samples (or even high bitrate samples for that matter) the approach in testing presented in this thread will not yield anything significant (it has a faulty premise) and actual testing would need to be done to address problems with the presets.

even if the encoder cannot be changed at a low/psymodel level, the presets can be modified to accomodate test results.

(also, i have already apologized for personal/sarcastic comments, please don't reuse them as benign ammunition.)


later
__________________
-xmixahlx, the one they call "mike"
http://xmixahlx.com -|- http://rarewares.org
xmixahlx is offline   Reply With Quote
Old 10 August 2005, 07:00 am   #15
guruboolez
Member
 
guruboolez's Avatar
 
Join Date: Aug 2004
Location: Strasbourg (France)
Posts: 13
Default

Pieroxy asked in this thread:
Quote:
Naturally, MPC sounds better in these parts, but what of the other parts? the parts where LAME would allocate 128kbps and MPC only 110kbps?
Giving an element of answer to a question is really off-topic?

Shy>
Quote:
I don't see how Musepack's quality was "unusually lower" with any sample either.
To help you: take another look on this graph:
MPC is worse than atrac3, WMA, vorbis and AAC. Bitrate droped to ~90 kbps.

Quote:
. And that too doesn't mean much since the overall performance is what matters in the end.
You missed the point. Of course MPC overall results are good, if not excellent. But what asked pieroxy, and before him many people, is what would be the performance with another samples, including more samples with bitrate < 128 kbps.

Quote:
Also, taking those two results out of the rest means very little to anyone if they can't see the rest.
As developer, I suppose that you might be more interested by "exceptions" (aka problem sample) than overall results, no? Users should also be aware about these possible problems occuring with MPC (or any other format), and not only be enlighted about positive listening tests.
Then, "the rest [of the test]" (as you said) is not necessary representative of the kind of music people are listening to. People listening to classical music for example won't consider Debussy.wav as something exceptional. I've tested MPC --radio with classical music only long time ago (here), and MPC performance was:
- average for overall results
- bad with some samples
guruboolez is offline   Reply With Quote
Old 10 August 2005, 10:19 am   #16
pieroxy
Member
 
Join Date: Aug 2005
Posts: 5
Default

Thanks guru. I got a look at the same listening test today and was about to make approximately the same comment that you did. I therefore have nothing more to say.

Shy, his post was right on topic, since it answered - with hard facts - pretty much what I was wondering, even though it took me three posts to formulate it in an understandable way. I have sometimes trouble getting my questions out :roll:

xmixahlx, no hard feelings :wink: I have also have sometimes the displeasing tendency to flame more often than I should. ops:

Thanks to all for the great responsiveness.

Edit: Just so you know, I posted a proposal for a different method to select samples when doing a 'general purpose' listening test on HA.org There.
pieroxy is offline   Reply With Quote
Old 10 August 2005, 03:13 pm   #17
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

First of all, you both seem to fail to understand the ANOVA analysis and how to interpret the graphs. MPC, LAME and WMA are tied on that sample, unless you consider WMA having a ~50% chance being better than the previous two good enough to call it second. Vorbis, iTunes and ATRAC3 is the other group of tied codecs on that sample (which by the way, is very easy to encode and the result is not much different between all the codecs, thus the result is not so reliable either), and the fact that Musepack has the lowest bitrate means little.

pieroxy: I see no "hard facts" of any kind. His post "answered" no "question."

guruboolez: If you feel like having to make such groundbreaking statements, at least make them after having actually proven a point, which you haven't in any way.
Shy is offline   Reply With Quote
Old 21 October 2005, 07:27 pm   #18
burr
Member
 
Join Date: Oct 2005
Posts: 1
Default

The newest tests I've read shows that Vorbis (AOTUV4) at standard (~160-190) bitrates is on par or better than musepack with classical samples, and wonder when there will be another formal test on pop or rock samples, it would be very interesting to see how the two perform against one another on the "louder" genres of music.
burr is offline   Reply With Quote
Old 21 October 2005, 08:48 pm   #19
Seed
Musepack Nanny
 
Seed's Avatar
 
Join Date: Jul 2004
Posts: 168
Default

I'm sure the results would be different with other genres that are more balanced and representative of a real music lover's collection of albums. Whoever believes Vorbis is better, I'm happy for them. I will not conduct tests without co-operating with the users of this community who are considered authorities on the subject. If someone else does and many believe their claims, good for them
Seed is offline   Reply With Quote
Old 07 December 2005, 12:40 pm   #20
user
Member
 
Join Date: Feb 2005
Location: in the great wide open
Posts: 11
Default

Hi all,

maybe you have followed the long pre discussions of the latest Sebastian Mares 128k vbr multformat test at HA.
The longness of that topic shows imo, that the goal of that test was & is finally not very clear.

It was first about formats, which have portable support etc,
but the conductor does not think, that iriver/Rockbox or PPC/PDA's are portable,
then the test should be about "popular" formats, but wma-standard is not included, though wma-professional.

Well, all in all, MPC was excluded because of not being "modern", not being "popular", not being "portable".
(Though it is still popular at HA..., though not by the vorbis guys, who seem to consider mpc still as competitor, rival.)

So,
I suggest, that a fair 128k test is organized, including mpc and wma-standard, lame 3.97b2, ogg, aac.

For mpc i suggest 1.14v and 1.15v to include.
The previous tests were with 1.14b. (Another point against the deselection of mpc at that ha-test, as with 1.15v there is a modernized untested version).
One of the difficult organisation tasks will be, to find out q settings, wic average to same averaged bitrate on a large variety of music/albums.
But i see eg. gurus points, the behaviour of formats should also be tested at samples, which get clearly lower averaged bitrate.

I suggested at HA, that a 2nd consecutive test is organized, including some "comoparabnle anchor" and untested formats, ie. MPC & WMA-standard.

I dunno, if musepack.net guys should organize such a test, or if we should wait, if HA performs it.

Maybe it is even interesting, to make tests easier to carry out, to lower from 128k vbr to (9x - 10x) vbr, because of wma-standard and because of ability to abx.
__________________
users' guides: www.High-Quality.ch.vu
user is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
MPC in 192 kbit/s listening test @SoundExpert Serge Smirnoff General 19 31 December 2010 02:14 pm
Comparing rankings of different listening tests user General 5 25 May 2006 03:12 am
Listening Test Seed General 2 09 March 2005 01:29 pm
Wavpack (lossy-mode) vs mpc - any ABX tests done yet? Rain General 2 26 January 2005 11:40 pm


All times are GMT. The time now is 12:49 pm.


Powered by vBulletin® Version 3.8.11 Beta 2
Copyright ©2000 - 2019, vBulletin Solutions Inc.