rescanning shares stalls/fails in some cases

[anonymous]

I am using Arch Linux 64 bit, python 3.1.2, nicotine-plus-svn 1446-1. I have network drive mounted with sshfs, and I can add files to the share and see them in my shares. If I choose my entire collection however the 'building database' stalls and the files are not added to the share (see attachment for commandline output).

[quinox]

sounds like a bug in Mutagen, I'll see I can work around it (try/catch everything is ugly, but perhaps the best way to "solve" it)

[quinox]

Thanks for your report, I've changed N+ so it try/catches the whole mutagen-part with r1448. This will prevent N+ from crashing while scanning.

The source of your problem is an MP3 file that Mutagen cannot process correctly - it fails certain sanity checks. Nicotine+ will now throw a warning about it and skip the file altogether.

There's nothing I can do about the cause of the bug, it's either because your MP3 file is corrupt or because Mutagen doesn't understand it properly. I don't know which one it is; The Mutagen people are hosted at ​http://code.google.com/p/mutagen/issues should you want to talk to them

[anonymous]

Thanks for looking into this. With r1448 'building database' still hangs though, now with a different error.

The m3u tags in my mp3 files contain errors indeed, but that is the reality for mp3 collections in general It also seems correct behavior for mutagen to raise an error if tags cannot be read.

[anonymous]

That's my bad -_-

Try r1449

[anonymous]

about mutagen raising errors: I haven't read their design documents/API, but most of the time Mutagen returns a None object when it encounters something it cannot process. I agree that errors should be expected while scanning, I just didn't expect Mutagen to raise one :)

[anonymous]

I've let the 'building database' running for 30 minutes (see commandline_output3.txt). I see 'mutagen crashed on .. (entire line is too long for the gui). In ~/.nicotine/ the .db files stay the same size, so I assume no actual building is taking place. Aside from the actual bug, I suggest you show feedback if the building/scanning has failed or not.

[anonymous]

About the feedback: Added with r1451.

About your problem: See the new warning :) I suspect something got corrupted in your *.db files

[anonymous]

build r1451

[anonymous]

I deleted all. db files from ~/.nicotine, and with r1451 (see attachment commandline_output4.txt) rescanning/building still fails.

[quinox]

Thanks for hanging in there, we're now dealing with another problem: One of your files returns an unbelievable piece of meta-data.

Python line-of-failure must be off by one since it points to the cofiying of '2', which is clearly between 0 and 4294967295. Either the VBR-flag fails, or the meta-data suggests the song is shorter than 0 seconds or longer than 4294967295 seconds (that's a bit over 136 *years*)

Can you replace pynicotine/shares.py with ​this version? It's rigged to output what meta-data it encounters before it codifies it, so it will show what files makes n+ trip and what meta-data is attached to that file.

[anonymous]

r1448 with modified shares.py

[anonymous]

okay. Output added with modified share.py. But aren't you trusting metadata too much? It can be any data/garbage. (Also see www . well . com/~doctorow/metacrap.htm).

[anonymous]

Your mp3 is corrupt, just delete it, share scanning will proceed as normal.

[anonymous]

Replying to anonymous:

okay. Output added with modified share.py. But aren't you trusting metadata too much? It can be any data/garbage. (Also see www . well . com/~doctorow/metacrap.htm).

We probably are. It's code I didn't write, and I'm trying to establish who's at fault - I rather fix bugs than try/catch them away.

Re comment:11: deleting it will help in this case, but not in general.

Later today I'll modify N+ to try/catch the makeBinaryPacket part so you can scan without having to remove files.

[anonymous]

About the meta data link: That page seems to be talking about something else, since the meta-data we're using is not user-generated, instead it's automatically written/updated by tools, so in theory it's always perfect. When it's not the file is corrupt, which is different from incorrect meta-data.

We should fix it, but it's more of a be liberal with input, strict with output situation than a don't trust the meta-data one :)

[quinox]

The meta data exception is being try/catch'ed with r1452, pleaes see if that enabled you to complete the share scanning

[anonymous]

This fixes it for me thanks! It would be even nicer if you show feedback about corrupt files in the GUI instead of only in the commandline. Something like a warning icon where the 'scanning/building database' text is shown. , if you click on it you see : 'the following files where found corrupted and could not be added to the share:..' and maybe advise how to fix corrupted metadate.

[anonymous]

It shows up in the GUI in the log window, the lower part that is filled with informational messages.

The warning icon sounds like a great idea though, easy to implement as well since we already have a multi-channel log functionality

[anonymous]

Almost on all shares it says here:

Mutagen crashed on '/Users/blabla...': No module named oggflac

I suspect this could be due to spaces in file names? Perhaps, these should be correctly escaped when passing arguments to Mutagen?

[anonymous]

I suspect it has to do with a missing... oggflac module.

Do you have OGG/Flac files and are you missing this module? That sounds more plausible to me than the spaces :)

(All of my files/directories have spaces in them, and I don't get that error)

[anonymous]

thanks for the response, but is there ogg/flac module for python26? i have compiled the latest SVN version of nicotine+ as a native app using the doc/build_mac.txt and it runs just fine with (all needed?) python26 libraries embedded into the resources. The only problem I have is with rescanning the shares and mutagen crashing on all my folders, as I see now.

[anonymous]

Also, does oggflac have anything to do with rescanning mp3s? :)

[quinox]

I guess some of the Mutagen files weren't included then, when Mutagen is started it performs:

from mutagen.oggflac import OggFLAC

So that's where your error is coming from. You're probably missing this file:

./site-packages/mutagen/oggflac.py

All the mutagen files that my system has:

quinox@tikko /usr/lib64/python2.7 $ equery files mutagen
 * Searching for mutagen ...
 * Contents of media-libs/mutagen-1.20:
/usr
/usr/bin
/usr/bin/mid3iconv
/usr/bin/mid3iconv-2.7
/usr/bin/mid3v2
/usr/bin/mid3v2-2.7
/usr/bin/moggsplit
/usr/bin/moggsplit-2.7
/usr/bin/mutagen-inspect
/usr/bin/mutagen-inspect-2.7
/usr/bin/mutagen-pony
/usr/bin/mutagen-pony-2.7
/usr/lib64
/usr/lib64/python2.7
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/mutagen
/usr/lib64/python2.7/site-packages/mutagen-1.20-py2.7.egg-info
/usr/lib64/python2.7/site-packages/mutagen/__init__.py
/usr/lib64/python2.7/site-packages/mutagen/_constants.py
/usr/lib64/python2.7/site-packages/mutagen/_util.py
/usr/lib64/python2.7/site-packages/mutagen/_vorbis.py
/usr/lib64/python2.7/site-packages/mutagen/apev2.py
/usr/lib64/python2.7/site-packages/mutagen/asf.py
/usr/lib64/python2.7/site-packages/mutagen/easyid3.py
/usr/lib64/python2.7/site-packages/mutagen/easymp4.py
/usr/lib64/python2.7/site-packages/mutagen/flac.py
/usr/lib64/python2.7/site-packages/mutagen/id3.py
/usr/lib64/python2.7/site-packages/mutagen/m4a.py
/usr/lib64/python2.7/site-packages/mutagen/monkeysaudio.py
/usr/lib64/python2.7/site-packages/mutagen/mp3.py
/usr/lib64/python2.7/site-packages/mutagen/mp4.py
/usr/lib64/python2.7/site-packages/mutagen/musepack.py
/usr/lib64/python2.7/site-packages/mutagen/ogg.py
/usr/lib64/python2.7/site-packages/mutagen/oggflac.py
/usr/lib64/python2.7/site-packages/mutagen/oggspeex.py
/usr/lib64/python2.7/site-packages/mutagen/oggtheora.py
/usr/lib64/python2.7/site-packages/mutagen/oggvorbis.py
/usr/lib64/python2.7/site-packages/mutagen/optimfrog.py
/usr/lib64/python2.7/site-packages/mutagen/trueaudio.py
/usr/lib64/python2.7/site-packages/mutagen/wavpack.py
/usr/share
/usr/share/doc
/usr/share/doc/mutagen-1.20
/usr/share/doc/mutagen-1.20/API-NOTES.bz2
/usr/share/doc/mutagen-1.20/NEWS.bz2
/usr/share/doc/mutagen-1.20/PKG-INFO.bz2
/usr/share/doc/mutagen-1.20/README.bz2
/usr/share/doc/mutagen-1.20/TODO.bz2
/usr/share/doc/mutagen-1.20/TUTORIAL.bz2
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/mid3iconv.1.bz2
/usr/share/man/man1/mid3v2.1.bz2
/usr/share/man/man1/moggsplit.1.bz2
/usr/share/man/man1/mutagen-inspect.1.bz2
/usr/share/man/man1/mutagen-pony.1.bz2

Every single file from /usr/lib64/python2.7/site-packages/ should be present, the other files probably don't matter