Musepack Forums  

Go Back   Musepack Forums > Main > General

Reply
 
Thread Tools Search this Thread Display Modes
Old 06 July 2006, 07:37 am   #1
Serge Smirnoff
Member
 
Join Date: Jul 2006
Posts: 4
Default MPC in 192 kbit/s listening test @SoundExpert

As you probably know MPC 1.15v is a contender in new 192 kbit/s listening test @SoundExpert. Actual bitrate calculations in SE tests are made on basis of 9 sound samples used for testing since 2001. In order to achieve target bitrate I used “- - quality 5.5” (193.8 kbit/s). As there are some doubts that fractional q-parameters are less tuned and rarely used, I would like to hear your comments and suggestions on the point. Discussion at HA is here.
__________________
Keep your audio clear! - www.soundexpert.info
Serge Smirnoff is offline   Reply With Quote
Old 06 July 2006, 09:54 am   #2
shadowking
Member
 
Join Date: Aug 2004
Posts: 4
Default

I am not a musepack dev, but the fractional quality should be totaly safe as it simply adjusts NMT / TMN values. You can see the values by using --quality 5.5 --verbose. The only time something major changes is when you go to a new profile : --quality 5.99 >> 6
shadowking is offline   Reply With Quote
Old 07 July 2006, 12:04 pm   #3
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

Fractional parameters are well tuned. The only non "tuned" parameters are those other than --quality n or --preset name.
Shy is offline   Reply With Quote
Old 07 July 2006, 12:13 pm   #4
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

As for samples to use, I don't suppose you'll get results that are any better than your 128kbps test's results if you choose yet another set of not very problematic samples.
Demanding samples reach a bitrate much higher than 192kbps with any codec using standard as well as low quality settings. Which brings us to the fault in testing codecs for transparency by the avarage bitrate of the encoded audio segment as opposed to the avarage bitrate of the entire musical piece that segment is extracted from.

Naturally, audio segments that are truly demanding and which expose a codec's capabilities, cause a bitrate higher than the usually much lower bitrate in non demanding audio with the same quality setting. An entire encoded song can have a bitrate of 180kbps while a demanding segment from it can require 280kbps.

There are cases where one codec would allocate much higher bitrate to a problematic audio segment with its standard quality setting than another codec with a similar setting and thus achieve much better quality. And sometimes, the codec that has a lower bitrate achieves similar quality to that with the higher bitrate. And sometimes, a codec allocates a much lower bitrate than another codec and still achieves higher quality than the other.

The issue is not with the bitrate, it's with how efficient and "smart" a codec is with its bitrate allocation for varying degrees of audio complexity. The bitrate which needs to be taken into account is the avarage bitrate of an entire musical piece, and not of a problematic segment.

So to summarize, a test limited to a single avarage bitrate of encoded audio segments can't expose transparency efficiency of codecs. Instead, all codecs would get similar ratings, considering that the audio to encode is not problematic enough to actually expose deficiencies, so personally I can't see the point in such a result.
Shy is offline   Reply With Quote
Old 08 July 2006, 07:36 am   #5
Serge Smirnoff
Member
 
Join Date: Jul 2006
Posts: 4
Default

Quote:
Originally Posted by Shy
Fractional parameters are well tuned. The only non "tuned" parameters are those other than --quality n or --preset name.
Thank you very much for clarification on non-integer MPC q-parameters. MPC codec (1.15v) with “--quality 5.5” will be used in SoundExpert (SE) 192 listening test.

Quote:
Originally Posted by Shy
The issue is not with the bitrate, it's with how efficient and "smart" a codec is with its bitrate allocation for varying degrees of audio complexity. The bitrate which needs to be taken into account is the avarage bitrate of an entire musical piece, and not of a problematic segment.
This problem is well-known and SE test setup uses a sort of compromise solution. Nine sound samples (mostly from SQAM disk) are concatenated in one sound file without gaps/pauses. This file is used both for testing and calculation of actual bitrate in all SE tests since 2001. The samples are very different – from speech to electronic and classical music. So there is a room for a codec to show how smart it is when allocating bits across all nine samples. Yes, this solution is not ideal, but AFIC there is no ideal solution for the case – each has its own “pros” and “cons”.

And finally, MPC “--quality 5.5” produces 193.8 kbit/s average bitrate (or file based bit rate – FBR) on SE united test sample described above.
__________________
Keep your audio clear! - www.soundexpert.info
Serge Smirnoff is offline   Reply With Quote
Old 08 July 2006, 07:10 pm   #6
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

If that's the bitrate, it's most likely that this combined sample is not good enough to be a "problem sample" that can actually pose real issues to the codec. I say this from experience which says the vast majority of serious problem samples (as in samples that clearly expose sound deficiencies) give a much higher bitrate, avaraging around 260kbps with "standard" settings of codecs.

There are very few cases where a ~192kbps audio segment would actually cause a difference in the sound which is noticeable enough with Musepack.

The one likely thing this test can show is that MP3 and WMA are not as good as the alternatives (Musepack, Vorbis, AAC), which is hardly a thing that needs further proof.
For showing which of the better alternatives is best, a 192kbps sample is highly unlikely to be of any use.

I of course don't mean to undermine your efforts which I appreciate, but I must say I think the entire idea to limit yourself to a single bitrate for a segment of audio just to have a "192kbps test" is very counterproductive.

A better, yet still not so sensible way to do a modern audio codec test would be to take an entire track, encode it with various codecs and reach a similar bitrate with all of them, and then choose the segment which you hear poses a problem, and use the same encoding parameters to encode the segment and not pay attention to the segment's bitrate, which would most likely have a much higher bitrate than that of the entire track.

Another way would be to take problematic samples and encode them with various codecs' equivalent parameters, and again, not pay attention to the bitrate. The actual sound artifacts are what matters, not the bitrate.

A modern, useful test would be based on the idea that:
All codecs define quality levels for you to use and trust, and it's those quality levels which should be tested, not bit rates.

I'm sorry if what I think is not so pleasant to hear, but I feel it's important to make it clear that such a test's method is not efficient and not suitable for today's modern audio codec world, which has since many years already gone way past limiting the encoding process to a constant or avarage bitrate, and instead, is based on variable bitrate.

The only avarage bitrate that matters in today's codecs is that which can be derived from calculation of the avarage bitrate an encoding parameter gives with thousands of albums. Bitrate approximations that Musepack, Vorbis and AAC tools show you are derived from just that.
Shy is offline   Reply With Quote
Old 08 July 2006, 10:45 pm   #7
Serge Smirnoff
Member
 
Join Date: Jul 2006
Posts: 4
Default

Quote:
Originally Posted by Shy
A modern, useful test would be based on the idea that:
All codecs define quality levels for you to use and trust, and it's those quality levels which should be tested, not bit rates.
You can “close your eyes” to actual SE bitrate calculations (FBR) and just compare different codecs with chosen (for some reason) quality parameters.

Quote:
Originally Posted by Shy
The only avarage bitrate that matters in today's codecs is that which can be derived from calculation of the avarage bitrate an encoding parameter gives with thousands of albums. Bitrate approximations that Musepack, Vorbis and AAC tools show you are derived from just that.
Even in this case final bitrate will depend on type of albums chosen. Thousands of albums with classical music will give lower figures and with hard rock – higher ones.
__________________
Keep your audio clear! - www.soundexpert.info
Serge Smirnoff is offline   Reply With Quote
Old 09 July 2006, 12:13 am   #8
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

Quote:
Even in this case final bitrate will depend on type of albums chosen. Thousands of albums with classical music will give lower figures and with hard rock – higher ones.
First of all, the notion that on avarage, classical music albums have a bitrate lower than that of hard rock albums is wrong. Second, of course basing that bitrate on thousands of albums of one genre is wrong, and that's not how anyone does it.

Quote:
You can “close your eyes” to actual SE bitrate calculations (FBR) and just compare different codecs with chosen (for some reason) quality parameters.
This sentence is not clear to me.
Shy is offline   Reply With Quote
Old 09 July 2006, 10:54 am   #9
Serge Smirnoff
Member
 
Join Date: Jul 2006
Posts: 4
Default

Quote:
Originally Posted by Shy
First of all, the notion that on avarage, classical music albums have a bitrate lower than that of hard rock albums is wrong. Second, of course basing that bitrate on thousands of albums of one genre is wrong, and that's not how anyone does it.
Seems the first contradicts with the second. Anyway, bitrate depends on genre.

Quote:
Originally Posted by Shy
Quote:
You can “close your eyes” to actual SE bitrate calculations (FBR) and just compare different codecs with chosen (for some reason) quality parameters.
This sentence is not clear to me.
You can interpret future results of upcoming SoundExpert listening test as a comparison of different coders with particular quality settings (chosen for some reason) without taking into account actual bitrates they achieve.
__________________
Keep your audio clear! - www.soundexpert.info
Serge Smirnoff is offline   Reply With Quote
Old 09 July 2006, 12:04 pm   #10
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

Quote:
Seems the first contradicts with the second. Anyway, bitrate depends on genre.
It doesn't contradict. Since you set a rule that classical music has low bitrate while hard rock music has high bitrate, I commented that those generalizations are in fact not true. I meant hard rock and classical album bitrates are generally similar, and that's from comparing a large amount of content. The idea that classical music bitrate is generally low is like the idea that old, low quality recordings are generally easy to encode/result in low bitrate, which is definitely wrong as well.

Regarding the comments in the link - thanks for making the good decision to come and post here to verify things properly.
Shy is offline   Reply With Quote
Old 15 January 2007, 06:01 am   #11
vinnie97
Member
 
Join Date: May 2006
Posts: 4
Default

interesting results in this test to say the least: http://www.soundexpert.info/coders192.jsp
vinnie97 is offline   Reply With Quote
Old 16 January 2007, 11:45 am   #12
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

What's interesting is that it's clearly stated that 5.0 and above means you will not hear any sound artifacts, and every codec in that test got a higher than 5.0 mark as I said would be most likely to happen, yet they feel they can still rate those results.
Quote:
above 5.0 – all sound artifacts will be beyond threshold of human perception with corresponding perception margin
A very "expert" test indeed.
Shy is offline   Reply With Quote
Old 20 January 2007, 08:58 am   #13
vinnie97
Member
 
Join Date: May 2006
Posts: 4
Default

The significance of the methodology used in the real world is questionable at best. But it is the only larger scale blind listening test conducted at that bitrate that I know, so I find the results interesting regardless, as that range and above has always been where MPC supposedly reigns in quality.

But to further your point, the test shows that ALL of the modern formats are equally superior at 192 kbps and above, not just MPC. And this is why developers have been aiming for transparency at 128 kbps for the past several years...because a large cross-section of the population can't detect any differences, even at that bitrate.
vinnie97 is offline   Reply With Quote
Old 20 January 2007, 04:27 pm   #14
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

I too find the results interesting, because it's very interesting how such ratings and recommendations are concluded even though the very rules set for how to conclude them completely contradict.

You seem to have missed the entire discussion in this thread, because as I said it's not 192kbps segments that usually exhibit artifacts, but much higher bitrate ones and not the continuous, avarage bitrate ones. The content in that test is easy to encode and doesn't represent any known problematic characteristics.

In case you haven't noticed, the more people testing a codec in that test, the higher score it gets, even if it doesn't mean anything. Unsurprisingly, not as many people tested MPC, but of course the error margin was also much lower than that of AAC.

Feel free to do so if you want to doubt the quality of a codec because someone has nothing better to do than first say it's impossible to hear artifacts if X codec gets a rating of above 5.0, and then rates and even unrecommends a codec that got the same results as the other codecs that got "rated" based on nothing.

It's in fact pretty funny to see how all that's important to such experts is having their little expert tests to wave in front of people who don't know too much, and not quality or any proper methodology that could be used to actually achieve something that isn't a load of nonsense.
Shy is offline   Reply With Quote
Old 21 January 2007, 12:34 pm   #15
vinnie97
Member
 
Join Date: May 2006
Posts: 4
Default

Actually, I read it months ago. I have a short memory. Basically, anything sifted from that test has to be taken with a grain of salt. I still don't agree on the "problem samples" comment since honing in on such samples is not characteristic of an actual listening experience (not to mention they are different for each format to some extent).
vinnie97 is offline   Reply With Quote
Old 21 January 2007, 07:11 pm   #16
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

You may not agree but the fact is that there are many and many kinds of "problem samples" and developers of every codec know the various types and how they affect their codec, and that's how any codec is tuned to avoid artifacts.

If for a change you'd like a useful, real example to see what I mean about high bitrate segments causing artifacts, here's one from a track of my own which is one big "problem sample" in itself. At bitrates much higher than 192kbps, every codec including Musepack reveals artifacts. You decide which handles it better. Though I suggest you pass on encoding it to MP3 and WMA unless you want to be horrified.
Shy is offline   Reply With Quote
Old 17 March 2009, 10:52 am   #17
dev
Member
 
Join Date: Mar 2009
Posts: 20
Default

Quote:
Originally Posted by Shy View Post
You may not agree but the fact is that there are many and many kinds of "problem samples" and developers of every codec know the various types and how they affect their codec, and that's how any codec is tuned to avoid artifacts.

If for a change you'd like a useful, real example to see what I mean about high bitrate segments causing artifacts, here's one from a track of my own which is one big "problem sample" in itself. At bitrates much higher than 192kbps, every codec including Musepack reveals artifacts. You decide which handles it better. Though I suggest you pass on encoding it to MP3 and WMA unless you want to be horrified.
Hi Shy, can You upload that sample again ?
dev is offline   Reply With Quote
Old 18 March 2009, 08:44 pm   #18
Shy
Admin
 
Shy's Avatar
 
Join Date: Jul 2004
Posts: 370
Default

reuploaded here
Shy is offline   Reply With Quote
Old 19 March 2009, 07:21 pm   #19
dev
Member
 
Join Date: Mar 2009
Posts: 20
Default

thx
dev is offline   Reply With Quote
Old 31 December 2010, 02:14 pm   #20
JoshuaChang
Member
 
Join Date: Sep 2007
Posts: 1
Default

Quote:
Originally Posted by Shy View Post
reuploaded here
this sample is very powerful, almost all the codec below 400kbps have noticeable artifacts, including aotuv vorbis and nero aac
well, i've encode it using my optimized sv8 encoder, q8 setting, seems a bit better than the official one:
http://www.uudisc.com/user/joshuachang/file/3796678
JoshuaChang is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Comparing rankings of different listening tests user General 5 25 May 2006 03:12 am
low level clicks and noise davidm General 20 29 June 2005 09:05 am
test release of new gui mpc player for the mac kuniklo MPC for UNIX 4 20 March 2005 07:54 am
Listening Test Seed General 2 09 March 2005 01:29 pm


All times are GMT. The time now is 03:47 pm.


Powered by vBulletin® Version 3.8.11 Beta 2
Copyright ©2000 - 2018, vBulletin Solutions Inc.