I made a small change to make seeking in the buffer faster :
http://trac.musepack.net/trac/changeset/459
I've rapidly checked that it works, but maybe it's buggy or not fast enough for you
You said that seeking is slow the first time : the file doesn't contain an index ? Adding the index may be the best way to fix the issue.
regards,
Nicolas