Discussion:
New in 7.9: Fulltext search
Michael Herger
2014-10-23 14:40:38 UTC
Permalink
Something new to play with: full text search (FTS).

Your search no longer is limited to a simple title/name search. With the
FTS you can now easily find track X by artist Y. Just type "hurry love
supremes" if you want to make sure not to get Phil Collins' version.

Other examples:

"mp3" - only find tracks in lossy mp3 (or with mp3 in the name - oh well)

"playlist elvis" - find playlists with tracks by Elvis, or with Elvis in
the name or a track's name

"flac 88.2 mozart" - get that highres classical stuff

The search code tries to weight results: hits in track titles have
higher priority than hits in a track's artist or album name. Eg. "lov"
would rank ABBA's "loving me, loving you" higher than "love me tender"
from the "greatest love songs, ever" album. Playlist names have priority
over track titles included in the playlist etc.

FTS is available from both the web UI's "live search" as well as the
regular search menus. Please give it a try and let me know what you
think about it.


A few caveats:

- FTS most likely will be a bit heavier on your system, with regards to
both the CPU as well as memory. I guess it shouldn't be used without the
highmem preference being set to high or max.

- FTS only works with SQLite. If you decided to use MySQL for whatever
reason, then I'm sorry, this won't work, as it's based around SQLite's
FTS extension

FTS is implemented as a plugin (mostly). If you don't like it, then just
disable the plugin to get the old behaviour back.

--

Michael
Mnyb
2014-10-23 15:31:08 UTC
Permalink
+ excellent idea :)



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
tufty
2014-10-23 17:10:33 UTC
Permalink
Great new feature, thanks!


------------------------------------------------------------------------
tufty's Profile: http://forums.slimdevices.com/member.php?userid=47700
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
pippin
2014-10-23 17:25:28 UTC
Permalink
Michael, you are awesome!



---
learn more about iPeng, the iPhone and iPad remote for the Squeezebox
and
Logitech UE Smart Radio as well as iPeng Party, the free Party-App,
at penguinlovesmusic.com
*New: iPeng 8, the Universal App for iOS 7 and iOS 8*
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-23 18:35:05 UTC
Permalink
> Michael, you are awesome!

...except that it doesn't work with iPeng... Only tested it after I had
it announced here. Web UI should be fine though. I'll take care of it asap.

--

Michael
frank1969
2014-10-23 19:54:05 UTC
Permalink
Great idea but...
... after 40 minutes it still says "Volltext Index erstellen (0 von 5)
Läuft 00:39:45" - and there is only one "dot" shown - so it should
take hours, dates ...
I know, i have a 420k track library, but on a high performance PC.
Any idea how long full text search "usually" should do or should I stop
and restart so it maybe "hanged`up"?


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
mdconnelly
2014-10-23 19:58:50 UTC
Permalink
Awesome concept! Where is it picking up the hirez "88.2" in a query?
Is it actually looking at the files or just if that was stored in a tag?
Just wondering if I could do the same for 24bit depth?



Mike (1 SB3, 1 Duet, 1 Boom, 2 Touch, & heavy use of iPeng on various
iThings)
------------------------------------------------------------------------
mdconnelly's Profile: http://forums.slimdevices.com/member.php?userid=10312
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-23 21:12:28 UTC
Permalink
> Awesome concept! Where is it picking up the hirez "88.2" in a query?

The data is pulled from the files during the regular scan. Depth is in
there too, though "16" or "24" as keywords probably aren't that
meaningful. Plus I've seen a majority of my files not report any depth
at all. This might depend on the file format.

Check a track's info menu, "Read Tags" to see whether it's reported
there. If it is, then it's in the fulltext index, too.

--

Michael
Michael Herger
2014-10-23 20:18:31 UTC
Permalink
"usually" is hard to define. My 20k tracks take a second or two to index on a poor Atom based system. Even less on the Macbook Pro with SSD, of course.

Is there any activity going on? Eg. massive memory usage? CPU load? library.db changes?



> Am 23.10.2014 um 21:54 schrieb frank1969 <***@no-mx.forums.slimdevices.com>:
>
>
> Great idea but...
> ... after 40 minutes it still says "Volltext Index erstellen (0 von 5)
> Läuft 00:39:45" - and there is only one "dot" shown - so it should
> take hours, dates ...
> I know, i have a 420k track library, but on a high performance PC.
> Any idea how long full text search "usually" should do or should I stop
> and restart so it maybe "hanged`up"?
>
>
> ------------------------------------------------------------------------
> frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
> View this thread: http://forums.slimdevices.com/showthread.php?t=102347
>
> _______________________________________________
> beta mailing list
> ***@lists.slimdevices.com
> http://lists.slimdevices.com/mailman/listinfo/beta
frank1969
2014-10-23 20:24:10 UTC
Permalink
Yes, thanks - after 55 minutes it finished full text search (for 420k
mp3-files). I will now test the feature and report if it was only the
initial scan that took so "long" or if it's the same with every
(update-) scan. Great idea, by the way! :-)


mherger wrote:
> "usually" is hard to define. My 20k tracks take a second or two to index
> on a poor Atom based system. Even less on the Macbook Pro with SSD, of
> course.
>
> Is there any activity going on? Eg. massive memory usage? CPU load?
> library.db changes?


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
frank1969
2014-10-23 20:33:20 UTC
Permalink
Ok, first thing I recognized:
New artists are not to be found.

Yesterday I added some tracks from (e.g.) Willy Wagner or Tina (02) - I
didn't own any tracks of them before (as "co"-artist, separated with ";"
as I always do)

If I call the album, I see performer "Willy Wagner", can click on the
name and it shows me I own own album he contributes tracks too - fine,
works as it should!

If I search for "Willy Wagner", it finds - nothing.

If I browse my artist list, it finds - nothing!


This behaviour seems to affect only new added TRACK ARTIST .

I also added an album by an artist I did own nothing from before (Ric
Seroka) so he is mentioned as ("single") track artist and ALBUM artist -
and everyhting is finde.
I can search for his name and I find him in the artist's list...


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
frank1969
2014-10-23 20:42:54 UTC
Permalink
And another thing (I use a second "answer" for this, for it seems to be
another issue?):
It doesn't seem to work ..

I type in "hurry love supremes" - and it just finds - nothing.
If I use the extended search and type in "hurry love" in title and
"supremes" in artsists - it finds the song (even on 6 cd's)

Do I have to change anything on my settings (except for setting the "x"
in PlugIns) to use the feature?

Below the end of my scanner log - I see the "out of memory" error (Its
run with 8 GB RAM and the high memory option...) but this seems to be as
a later point when the FullTextSearch is successfully done...?!


[14-10-23 21:12:56.9772] Slim::Music::Import::runImporter (485) Starting
Slim::Plugin::FullTextSearch::Plugin scan
[14-10-23 22:08:16.6278] Slim::Music::Import::endImporter (617)
Completed Slim::Plugin::FullTextSearch::Plugin Scan in 3319 seconds.
[14-10-23 22:08:16.6286] Slim::Music::Import::runImporter (485) Starting
Slim::Music::VirtualLibraries scan
[14-10-23 22:08:20.1274] Slim::Music::Import::endImporter (617)
Completed Slim::Music::VirtualLibraries Scan in 3 seconds.
[14-10-23 22:08:22.8880] Slim::Music::Artwork::updateStandaloneArtwork
(204) Starting updateStandaloneArtwork for 8807 albums
[14-10-23 22:10:50.7034] Slim::Music::Artwork::__ANON__ (309)
updateStandaloneArtwork finished in 147.80641913414
[14-10-23 22:10:51.7668] Slim::Music::Artwork::precacheAllArtwork (615)
Starting precacheArtwork for 48 albums
[14-10-23 22:11:11.4912] Slim::Music::Artwork::__ANON__ (734)
precacheArtwork finished in 19.7102890014648
[14-10-23 22:11:11.4917] Slim::Music::Import::endImporter (617)
Completed precacheArtwork Scan in 20 seconds.
[14-10-23 22:11:11.6310] Slim::Music::Import::runScanPostProcessing
(409) Starting Database optimization.
[14-10-23 22:11:57.3599] Slim::Utils::SQLiteHelper::vacuum (514)
Warning: DBD::SQLite::db do failed: out of memory at
/<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Slim/Utils/SQLiteHelper.pm
line 514.
[14-10-23 22:12:13.7029] Slim::Music::Import::endImporter (617)
Completed dbOptimize Scan in 62 seconds.


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-23 21:10:27 UTC
Permalink
> [14-10-23 22:11:57.3599] Slim::Utils::SQLiteHelper::vacuum (514)
> Warning: DBD::SQLite::db do failed: out of memory at
> /<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Slim/Utils/SQLiteHelper.pm
> line 514.

Ok, the scan failed. No good. But I'll ignore all your other reports, as
they're most likely just side-effects of this major issue.

Unfortunately I only have Erland's 100k library to test with. I'll let
you know what I get there.

--

Michael
frank1969
2014-10-23 21:23:06 UTC
Permalink
mherger wrote:
> Unfortunately I only have Erland's 100k library to test with. I'll let
> you know what I get there.
>

If there is anything/any details I can provide to help You, please let
me know.

mherger wrote:
> Oh, and you do use the "maximum" memory preference in
> Settings/Advanced/Performance?
>

We're talking about the "High (recommended for Computers with 1GB RAM or
more)" (or someting like that, I'm using the German Version)-option?
Yes, I do...


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-23 21:31:37 UTC
Permalink
>> Oh, and you do use the "maximum" memory preference in
>> Settings/Advanced/Performance?
>
> We're talking about the "High (recommended for Computers with 1GB RAM or
> more)" (or someting like that, I'm using the German Version)-option?
> Yes, I do...

Use "Maximum". Though, your "out of memory" message seems as if that
wouldn't help...

--

Michael
frank1969
2014-10-23 21:38:34 UTC
Permalink
mherger wrote:
>
> Use "Maximum". Though, your "out of memory" message seems as if that
> wouldn't help...
>

Ah, ok, sorry, I didn't recognize this third option, only knew the other
two. I changed to maximum and will do another scan.
If it fails once again I'll re-produce a "successful" db for You. Thanks
in advance for letting me know about this option.


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
frank1969
2014-10-23 21:45:41 UTC
Permalink
Another thing I recognized (and I know see again while it restarts):
r-1414070955 takes MUCH longer to start the server. Usually it takes
about some 20 seconds - know it takes some 10 minutes (I didn't stop but
roundabout...)

server.log says:
[14-10-23 23:37:05.2476] Slim::Schema::forceCommit (2147) Warning:
Trying to commit transactions before DB is initialized!
[14-10-23 23:37:15.3643] main::init (370) Starting Logitech Media Server
(v7.9.0, 1414070955, Thu Oct 23 13:56:44 CUT 2014) perl 5.014001
[14-10-23 23:37:15.8985] Slim::Utils::Misc::msg (1343) Warning:
[23:37:15.8983] DBIx::Class::ResultSet::update_or_create(): Query
returned more than one row. SQL that returns multiple rows is
DEPRECATED for ->find and ->single at
/<C:\PROGRA~2\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Schema.pm line 1876


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
frank1969
2014-10-23 22:01:30 UTC
Permalink
Another thing I recognized (and I know see again while it restarts):
r-1414070955 takes MUCH longer to start the server. Usually it takes
about 15 seconds - know it takes some 10 minutes

So I downgraded to r-1413907472 - it's up in 15 seconds again

upgrade to r-1414070955 once more - 5 minutes ago and it's still
"starting" in progress...

Maybe this helps You to locate the issue?

Sever.log seems to be the same:

[14-10-23 23:53:53.1442] main::init (370) Starting Logitech Media Server
(v7.9.0, 1413907472, Wed Oct 22 04:05:14 CUT 2014) perl 5.014001
[14-10-23 23:53:53.8945] Slim::Utils::Misc::msg (1343) Warning:
[23:53:53.8942] DBIx::Class::ResultSet::update_or_create(): Query
returned more than one row. SQL that returns multiple rows is
DEPRECATED for ->find and ->single at
/<C:\PROGRA~2\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Schema.pm line 1876

[14-10-23 23:55:38.6684] main::init (370) Starting Logitech Media Server
(v7.9.0, 1414070955, Thu Oct 23 13:56:44 CUT 2014) perl 5.014001
[14-10-23 23:55:39.3874] Slim::Utils::Misc::msg (1343) Warning:
[23:55:39.3872] DBIx::Class::ResultSet::update_or_create(): Query
returned more than one row. SQL that returns multiple rows is
DEPRECATED for ->find and ->single at
/<C:\PROGRA~2\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Schema.pm line 1876


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
JJZolx
2014-10-23 22:38:15 UTC
Permalink
FWIW, it took 1:06 to index 42,000 FLAC tracks and a handful of small
playlists on my server during a full library scan. I have few comments
and no lyrics in the metadata. Total scan time went from about 14 to 15
minutes.

At this time (v38612, updated via SVN just a few minutes ago), search
appears to still be broken.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-24 04:53:52 UTC
Permalink
> At this time (v38612, updated via SVN just a few minutes ago), search
> appears to still be broken.

Broken in what way? Anything in the log file?

--

Michael
JJZolx
2014-10-24 14:07:53 UTC
Permalink
mherger wrote:
>
> > At this time (v38612, updated via SVN just a few minutes ago), search
> > appears to still be broken.
>
> Broken in what way? Anything in the log file?

At the moment:
- The drop-down search fails to find anything.
- The quick search and the Search > Artists/Albums/Songs/Playlists form
only finds artists. All other search categories come up empty.
- Advanced search is fine.

I have 'search within words' enabled, but I've tried partial searches,
start-of-title searches, and exact title searches and all of them fail.
The only thing I see in the server log is from the scheduled new &
changed scan that ran last night:


Code:
--------------------

[14-10-24 04:12:13.1788] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db do failed: no such function: CONCAT_CONTRIBUTOR_ROLE [for Statement "
INSERT INTO fulltext (id, type, w10, w5, w3, w1)
SELECT tracks.id, 'track',
-- weight 10
IFNULL(tracks.title, '') || ' ' || IFNULL(tracks.titlesearch, '') || ' ' || IFNULL(tracks.customsearch, '') || ' ' || IFNULL(tracks.musicbrainz_id, ''),
-- weight 5
IFNULL(tracks.year, '') || ' ' || GROUP_CONCAT(albums.title, ' ') || ' ' || GROUP_CONCAT(albums.titlesearch, ' ') || ' ' || GROUP_CONCAT(genres.name, ' ') || ' ' || GROUP_CONCAT(genres.namesearch, ' '),
-- IFNULL(tracks.year, '') || ' ' || GROUP_CONCAT(contributors.name, ' ') || ' ' || GROUP_CONCAT(contributors.namesearch, ' ') || ' ' || GROUP_CONCAT(albums.title, ' ') || ' ' || GROUP_CONCAT(albums.titlesearch, ' ') || ' ' || GROUP_CONCAT(genres.name, ' ') || ' ' || GROUP_CONCAT(genres.namesearch, ' '),
-- weight 3 - contributors create multiple hits, therefore only w3
CONCAT_CONTRIBUTOR_ROLE(tracks.id, GROUP_CONCAT(contributor_track.contributor, ','), 'contributor_track') || ' ' || IFNULL(comments.value, '') || ' ' || IFNULL(tracks.lyrics, '') || ' ' || IFNULL(tracks.content_type, '') || ' ' || CASE WHEN tracks.channels = 1 THEN 'mono' WHEN tracks.channels = 2 THEN 'stereo' END,
-- weight 1
printf('%i', tracks.bitrate) || ' ' || printf('%ikbps', tracks.bitrate / 1000) || ' ' || IFNULL(tracks.samplerate, '') || ' ' || (round(tracks.samplerate, 0) / 1000) || ' ' || IFNULL(tracks.samplesize, '') || ' ' || tracks.url

FROM tracks
LEFT JOIN contributor_track ON contributor_track.track = tracks.id
-- LEFT JOIN contributors ON contributors.id = contributor_track.contributor
LEFT JOIN albums ON albums.id = tracks.album
LEFT JOIN genre_track ON genre_track.track = tracks.id
LEFT JOIN genres ON genres.id = genre_track.genre
LEFT JOIN comments ON comments.track = tracks.id

GROUP BY tracks.id;
"]
[14-10-24 04:12:13.1795] Slim::Schema::Storage::throw_exception (122) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Schema/Storage.pm line 122)
frame 1: Slim::Schema::Storage::throw_exception (C:\Program Files (x86)\Logitech Media Server 7.9\server\CPAN/DBIx/Class/Storage/DBI.pm line 1006)
frame 2: DBIx::Class::Storage::DBI::__ANON__ (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Plugin/FullTextSearch/Plugin.pm line 190)
frame 3: Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Plugin/FullTextSearch/Plugin.pm line 170)
frame 4: Slim::Plugin::FullTextSearch::Plugin::_triggerIndexRebuild (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Utils/Timers.pm line 266)
frame 5: (eval) (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Utils/Timers.pm line 266)
frame 6: Slim::Utils::Timers::__ANON__ (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Networking/IO/Select.pm line 168)
frame 7: (eval) (C:/Program Files (x86)/Logitech Media Server 7.9/server/Slim/Networking/IO/Select.pm line 168)
frame 8: Slim::Networking::IO::Select::loop (C:\Program Files (x86)\Logitech Media Server 7.9\server\slimserver.pl line 715)
frame 9: main::idle (C:\Program Files (x86)\Logitech Media Server 7.9\server\slimserver.pl line 665)
frame 10: main::main (C:\Program Files (x86)\Logitech Media Server 7.9\server\slimserver.pl line 1200)

[14-10-24 04:12:13.1822] Slim::Utils::Timers::__ANON__ (273) Error: Timer Slim::Plugin::FullTextSearch::Plugin::_triggerIndexRebuild failed:

--------------------


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-24 14:31:18 UTC
Permalink
> [14-10-24 04:12:13.1788] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db do failed: no such function: CONCAT_CONTRIBUTOR_ROLE [for Statement "

Huh!?! Did you disable the plugin without restarting or something? Are
you running from the source or the Windows binary? It complains about a
missing function which is defined in the plugin itself... but that code
would not even be called without the function being initialized. Puzzling...

--

Michael
JJZolx
2014-10-24 14:41:23 UTC
Permalink
mherger wrote:
>
> Huh!?! Did you disable the plugin without restarting or something? Are
> you running from the source or the Windows binary?

The plugin has not been disabled. This is run from the source.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-24 15:18:21 UTC
Permalink
> The plugin has not been disabled. This is run from the source.

Ok, I have an idea what could have caused this. Sometimes the database
handles are closed. In this case I need to re-register those custom
functions.

I've applied a change which should do this. Please give it a try.

--

Michael
JJZolx
2014-10-24 16:03:02 UTC
Permalink
Ok. Not quite as broken, but still broken.

Then biggest issue is that search within strings no longer works. I
would typically use searches such as 'nny c' to find 'Johnny Cash' or
'slow' to find Clapton's 'Slowhand' album. But it only finds full
words.

Searching in the quick search of the title bar for 'johnn' finds
nothing, but 'johnny' works.

Searching in the quick search -form- for 'johnn' shows the following,
listing no artist or song names. Clicking 'more' gives a similarly blank
list. Clicking 'All Songs' does show a list of songs (with no artist or
album shown). If a full word such as 'johnny' is used, it does list
artists, albums and songs.

Artists: 1 / 20: More...
All Songs
Albums: 1 / 3: More...
All Songs
Songs: None
Playlists: None


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
JJZolx
2014-10-24 16:17:11 UTC
Permalink
I ran a full scan after the last update. Here's the scanner log.


+-------------------------------------------------------------------+
|Filename: scan.txt |
|Download: http://forums.slimdevices.com/attachment.php?attachmentid=16537|
+-------------------------------------------------------------------+

------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-24 18:52:29 UTC
Permalink
> Then biggest issue is that search within strings no longer works. I
> would typically use searches such as 'nny c' to find 'Johnny Cash' or
> 'slow' to find Clapton's 'Slowhand' album. But it only finds full
> words.

Are you running the latest? I changed that behaviour again (it doesn't respect the pref yet - that's on the todo list)

> Searching in the quick search -form- for 'johnn' shows the following,
> listing no artist or song names. Clicking 'more' gives a similarly blank
> list.

Yeah, I've seen similar things... concentrated too much on performance with a large library today... Will review this.
gharris999
2014-10-24 21:12:15 UTC
Permalink
Ok, searching for "Mozart K.466"

..and expecting to find 2 tracks (from two Mozart different albums) with
the title:

"Piano Concerto No. 20 in d, K.466 (1785) - 1 Allegro"

..instead FTS is returning 75 songs, all but two seemingly unrelated to
the search. I'm a bit flummoxed as to why "Mozart K.466" should return
a track like:

"11 Andante, ma non troppo in Bb" from the album "Piano Bagatelles -
Stephen Kovacevich" by "Beethoven, L"

Are there any debugging options with FTS? I.e. can we log the SQL query
statement?

Can FTS be made optional?


------------------------------------------------------------------------
gharris999's Profile: http://forums.slimdevices.com/member.php?userid=115
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
gharris999
2014-10-24 21:15:51 UTC
Permalink
..or could FTS be made to search for littoral strings when the input is
quoted?


------------------------------------------------------------------------
gharris999's Profile: http://forums.slimdevices.com/member.php?userid=115
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-25 05:27:40 UTC
Permalink
> ..or could FTS be made to search for littoral strings when the input is
> quoted?

I think it should do so. See the "Phrase queries" section under
http://www.sqlite.org/fts3.html#section_3.

But I might actually have a bug breaking this. I'll look into this.

--

Michael
Michael Herger
2014-10-24 21:18:49 UTC
Permalink
> Are there any debugging options with FTS? I.e. can we log the SQL query
> statement?

database.sql? If this doesn't help, then I'll add some.

> Can FTS be made optional?

Just disable the plugin. See the announcement posting ;-).

But before you do so I'd be interested in your library.db. I'd like to
understand why you're getting the seemingly wrong results you see.

--

Michael
gharris999
2014-10-24 21:49:13 UTC
Permalink
mherger wrote:
> ...I'd be interested in your library.db. I'd like to understand why
> you're getting the seemingly wrong results you see.
Ok, my db files (artwork.db, cache.db, imgproxy.db, library.db,
persist.db) are posted at:

http://www.hegardtfoundation.org/slimstuff/lms_db_files.zip

Thanks..


------------------------------------------------------------------------
gharris999's Profile: http://forums.slimdevices.com/member.php?userid=115
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-25 05:32:37 UTC
Permalink
>> ...I'd be interested in your library.db. I'd like to understand why
>> you're getting the seemingly wrong results you see.
> Ok, my db files (artwork.db, cache.db, imgproxy.db, library.db,
> persist.db) are posted at:

Thanks, got it.

--

Michael
JJZolx
2014-10-25 05:55:49 UTC
Permalink
Yes, I'll probably disable it.

I can see its value in searching playlist content.

I'm not sure there is any other text to be searched for in artists apart
from the name, so it probably makes no difference there.

And possibly some value in searching tracks and albums, by looking into
artists, comments, lyrics. Although that can be done fairly well,
possibly better by using the advanced search. Trouble is, it can make
title searches difficult by producing too many results.

Just makes me wonder now why nobody ever implemented a better playlist
search.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-25 07:34:50 UTC
Permalink
> I'm not sure there is any other text to be searched for in artists apart
> from the name, so it probably makes no difference there.

The main advantage for artist searches is that eg. "clapton king" would find "Eric Clapton / B.B. King", while the normal search wouldn't. It's a bit less strict.

> And possibly some value in searching tracks and albums, by looking into
> artists, comments, lyrics. Although that can be done fairly well,
> possibly better by using the advanced search. Trouble is, it can make
> title searches difficult by producing too many results.

As I wrote in the previous posting I think there's room for improvement in the weighting algorithm.

> Just makes me wonder now why nobody ever implemented a better playlist
> search.

Nobody cared enough, I guess. I think of all the SB engineers working on LMS nobody was a playlist user. I certainly am not :-).
JJZolx
2014-10-25 08:08:53 UTC
Permalink
While I still have it enabled, here are some bugs I'm seeing:

If search for "eric c" and for artists it shows the header "Artists: 10
/ 140 More...". I click "More..." and it takes me into a page with 11
artists and displays a pagebar with "1 2" (which would be the correct
number if indeed there were 140 items). The second page is blank.

J.J. Cale & Eric Clapton
Eric Clapton
B.B. King & Eric Clapton
Eric Clapton
John Mayall & Eric Clapton
Eric Benet
Eric Dolphy
Eric Marienthal
Eric Hutchinson
Oliver Nelson & Eric Dolphy
Eric Reed

Among the artists is everyone named "Eric", with Eric Clapton listed
twice. By drilling down, I see that one Eric Clapton appears to be only
for the artist in a trackartist role, the other for albumartist role.

If I search for "eric cl" then I get a header saying "Artists: 6 / 140:
More...", but only 5 results:

J.J. Cale & Eric Clapton
Eric Clapton
B.B. King & Eric Clapton
Eric Clapton
John Mayall & Eric Clapton

Actually, any search I do now shows the totals 140 for Artists, 356 for
Albums, while the song total varies. Could this be a caching issue?


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-28 07:36:28 UTC
Permalink
> If search for "eric c" and for artists it shows the header "Artists: 10
> / 140 More...". I click "More..." and it takes me into a page with 11
> artists and displays a pagebar with "1 2" (which would be the correct
> number if indeed there were 140 items). The second page is blank.

I still haven't found a way to reliably reproduce this. I've seen it,
too. But not always. And not when I tried to see it... Do you have a
100% reproducible way to getting this behaviour right after a server
restart?

> J.J. Cale & Eric Clapton
> Eric Clapton
> B.B. King & Eric Clapton
> Eric Clapton
> John Mayall & Eric Clapton
> Eric Benet
> Eric Dolphy
> Eric Marienthal
> Eric Hutchinson
> Oliver Nelson & Eric Dolphy
> Eric Reed
>
> Among the artists is everyone named "Eric", with Eric Clapton listed
> twice. By drilling down, I see that one Eric Clapton appears to be only
> for the artist in a trackartist role, the other for albumartist role.

Ok, need to review this. One issue is that the search is doing a "eric*
AND c*" - which would indeed match any Eric which has _some_ content
starting with c. I need to get back to this once I'm over the poodle
issues...

> J.J. Cale & Eric Clapton
> Eric Clapton
> B.B. King & Eric Clapton
> Eric Clapton
> John Mayall & Eric Clapton
>
> Actually, any search I do now shows the totals 140 for Artists, 356 for
> Albums, while the song total varies. Could this be a caching issue?

I don't know yet. But yes (again), that's what I saw, too.

--

Michael
JJZolx
2014-10-28 21:24:25 UTC
Permalink
mherger wrote:
>
> > If search for "eric c" and for artists it shows the header "Artists:
> 10
> > / 140 More...". I click "More..." and it takes me into a page with 11
> > artists and displays a pagebar with "1 2" (which would be the correct
> > number if indeed there were 140 items). The second page is blank.
>
> I still haven't found a way to reliably reproduce this. I've seen it,
> too. But not always. And not when I tried to see it... Do you have a
> 100% reproducible way to getting this behaviour right after a server
> restart?

No, it's not consistent. After a server restart, however, it appears
that LMS remembers the totals from the first search that is run and will
use/display those occasionally on subsequent searches. It does this for
artists and albums totals, but not songs.

I haven't verified this, but what it _seems_ like is that if you run an
initials search that produces more than 10 results, then those totals
seem to be shown for every subsequent search.

There is also an off-by-one error in the count when search results are
less than 10. The count is always one greater.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-29 10:10:59 UTC
Permalink
> No, it's not consistent. After a server restart, however, it appears
> that LMS remembers the totals from the first search that is run and will
> use/display those occasionally on subsequent searches. It does this for
> artists and albums totals, but not songs.

Good catch! It was a caching issue indeed. Thanks - fixed.

--

Michael
JJZolx
2014-10-29 10:24:08 UTC
Permalink
mherger wrote:
> > No, it's not consistent. After a server restart, however, it appears
> > that LMS remembers the totals from the first search that is run and
> will
> > use/display those occasionally on subsequent searches. It does this
> for
> > artists and albums totals, but not songs.
>
> Good catch! It was a caching issue indeed. Thanks - fixed.

Ok, that's fixed. Have a look at the off-by-one error. It's not just
totals <10, as I previously suggested. It appears to be true for all
searches.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-29 10:35:36 UTC
Permalink
> Ok, that's fixed. Have a look at the off-by-one error. It's not just
> totals <10, as I previously suggested. It appears to be true for all
> searches.

Missed that one. But I'm not getting counts when there are less than 10
anyway. Do you still see this?

--

Michael
JJZolx
2014-10-29 10:42:04 UTC
Permalink
mherger wrote:
>
> > Ok, that's fixed. Have a look at the off-by-one error. It's not just
> > totals <10, as I previously suggested. It appears to be true for all
> > searches.
>
> Missed that one. But I'm not getting counts when there are less than 10
>
> anyway. Do you still see this?

No. It doesn't display a count when less than 10, although it did before
the last bug was fixed.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-29 21:54:34 UTC
Permalink
now it scans faster or is it just because i decided to run a ful scan
after the latest bug ?


Discovering files/directories: /media/music1 (47640 of 47640)
Complete 00:00:48

Scanning new music files: /media/music1 (43190 of 43190) Complete
00:23:10

Discovering playlists: /media/Playlists (17 of 17) Complete
00:00:00

Scanning new playlists: /media/Playlists (15 of 15) Complete
00:00:59

Smart Mix Profile export (41332 of 41332) Complete 00:02:43

Album cover lookup (81 of 81) Complete 00:00:06

Artist picture lookup (1503 of 1503) Complete 00:07:25

Building full text index (6 of 6) Complete 00:03:20

Create library views (4 of 4) Complete 00:00:02

Pre-caching Artwork (3417 of 3417) Complete 00:08:27

Database Optimize (2 of 2) Complete 00:01:00

The server has finished scanning your media library.
Total Time: 00:48:00 (2014-10-29 Wednesday / 22:26)

But it is slow , you have to wait to see results .?

Can i consider my server a bit weak for this ?



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-29 22:03:08 UTC
Permalink
> now it scans faster or is it just because i decided to run a ful scan
> after the latest bug ?

Indexing playlists was awfully slow. Should be much faster now. If you
have playlists.

> But it is slow , you have to wait to see results .?

Searching is slow, I assume?

> Can i consider my server a bit weak for this ?

Don't remember your specs. But mine is a 3yr old Atom. I doubt it's much
more powerful than yours. What about memory? I think memory is crucial.
If it has to read the index from the disk, then it'll be too slow. If it
can keep it in RAM, then you're fine. Make sure you use the "maximum"
setting for memory (if possible).

--

Michael
Mnyb
2014-10-29 22:44:47 UTC
Permalink
It something like this .

HP PROLIANT N36L MICROSERVER AMD ATHLON II 1.3-2MB 1G

Dual athlon cpu ,but only 1G ram :/ wonder if I can fit more ram to
it....



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
JJZolx
2014-10-30 02:08:49 UTC
Permalink
While you're working on this, is there any chance that song search
results for basic search in the web UI could made so that the individual
pieces - song title, album and artist - are clickable? This is how they
appear in Advance Search, and (IIRC) I believe how it worked at one
time. I think this was another casualty of onebrowser.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-30 07:18:07 UTC
Permalink
> While you're working on this, is there any chance that song search
> results for basic search in the web UI could made so that the individual
> pieces - song title, album and artist - are clickable? This is how they
> appear in Advance Search, and (IIRC) I believe how it worked at one
> time. I think this was another casualty of onebrowser.

Came to realise this a few days ago, too. But it's not high on my list.

--

Michael
Michael Herger
2014-10-30 07:17:25 UTC
Permalink
> Dual athlon cpu ,but only 1G ram :/ wonder if I can fit more ram to
> it....

1GB probably is a bit tight. Did you set the memory setting to maximum?
How much memory is the LMS process using?

--

Michael
Mnyb
2014-10-30 07:28:51 UTC
Permalink
mherger wrote:
> > Dual athlon cpu ,but only 1G ram :/ wonder if I can fit more ram to
> > it....
>
> 1GB probably is a bit tight. Did you set the memory setting to maximum?
>
> How much memory is the LMS process using?
>
> --
>
> Michael

I'll check tonigth is TOP good enough



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-30 07:32:02 UTC
Permalink
mherger wrote:
> > Dual athlon cpu ,but only 1G ram :/ wonder if I can fit more ram to
> > it....
>
> 1GB probably is a bit tight. Did you set the memory setting to maximum?
>
> How much memory is the LMS process using?
>
> --
>
> Michael

Mem setting on MAX , btw i use a very lean OS and its basically LMS so
memory has not been a big concern yet, maybe i'll dump the mip server
thats runs in the background as i stopped using it since your great
plugin .

I saw 4GB sticks for this server at dustin.se wher i bougth it i mail
order some memery ...

Btw great value for this server around 2100 sek



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
jwmelvin
2014-10-30 13:28:57 UTC
Permalink
My scanner is terminating with the fulltext index.

info:

Code:
--------------------
Logitech Media Server Version: 7.9.0 - 1414492657 @ Wed Oct 29 03:59:53 UTC 2014
Hostname: rocco
Server IP Address: 192.168.1.75
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.20.1 - x86_64-linux-gnu-thread-multi
Audio::Scan: 0.95
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 3
--------------------


scanner.log:

Code:
--------------------
[14-10-30 09:25:13.0312] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (212) Starting fulltext index build...
[14-10-30 09:25:13.0314] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (218) Initialize fulltext table...
[14-10-30 09:25:13.0965] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (224) Create fulltext index for tracks...
[14-10-30 09:25:13.0973] Slim::Utils::SQLiteHelper::updateProgress (411) Notify to server: [
"progress:1414675513.03064||importer||plugin_fulltext||1||6||",
]
[14-10-30 09:25:13.1011] Slim::Utils::SQLiteHelper::updateProgress (435) Notify to server OK
[14-10-30 09:25:19.0838] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (252) Create fulltext index for albums...
[14-10-30 09:25:19.0847] Slim::Utils::SQLiteHelper::updateProgress (411) Notify to server: [
"progress:1414675513.03064||importer||plugin_fulltext||2||6||",
]
[14-10-30 09:25:19.0885] Slim::Utils::SQLiteHelper::updateProgress (435) Notify to server OK
[14-10-30 09:25:20.1897] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (275) Create fulltext index for contributors...
[14-10-30 09:25:21.1607] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (293) Create fulltext index for playlists...
--------------------


Happy to provide any further info, of course. I attempted to enable
debugging in the logs. Memory is set to Maximum. System has 8GB, which
appears to be roughly half free during a scan. Library is ~28k songs. I
have 7 m3u playlists and the few opml files in a playlists folder.

Thanks for all your work.

-Jason


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-30 13:55:33 UTC
Permalink
> Happy to provide any further info, of course. I attempted to enable
> debugging in the logs.

What exactly did you set? Please pick the Scanner set from the dropdown
list at the top. Then re-run a scan and provide the _full_ scanner.log file.

--

Michael
jwmelvin
2014-10-30 17:30:15 UTC
Permalink
mherger wrote:
> What exactly did you set? Please pick the Scanner set from the dropdown
>
> list at the top. Then re-run a scan and provide the _full_ scanner.log
> file.

I had set those items relating to scanning. But now I selected the
Scanner set from the dropdown and performed a clear-and-rescan.
The log is available 'here'
(http://www.melvincentral.com/xfer/scanner.log).

Thank you for the help.
-Jason


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-30 19:42:24 UTC
Permalink
> I had set those items relating to scanning. But now I selected the
> Scanner set from the dropdown and performed a clear-and-rescan.

Hmm... did the scan stop there, without error nor other trace? Did the
scanner.exe disappear from the task list? Or did it continue to run,
probably a long time? The playlists are rather short, or even empty...

--

Michael
jwmelvin
2014-10-30 20:04:23 UTC
Permalink
mherger wrote:
> Hmm... did the scan stop there, without error nor other trace? Did the
> scanner.exe disappear from the task list? Or did it continue to run,
> probably a long time? The playlists are rather short, or even empty...

The playlists are not long; among the 7 m3u files, there are 351 lines.


The scanner log I posted is the complete log and has not changed since
the scan terminated.

When the web interface indicates termination--"Media scan terminated
unexpectedly (Scanning Progress)"--the scanner process disappears from
my list of processes using top/htop.

If I move all m3u files out of the playlists directory, it makes no
difference to the termination; it still occurs when the web interface
reports "Building full text index (2 of 6)." If I also move all the
opml files out of the playlists directory, same result.

-Jason


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-31 12:44:54 UTC
Permalink
> The scanner log I posted is the complete log and has not changed since
> the scan terminated.

Could you please re-try after enabling debug logging for database.sql?
Thanks!

--

Michael
Michael Herger
2014-10-31 13:28:26 UTC
Permalink
> Could you please re-try after enabling debug logging for database.sql?
> Thanks!

Oh, no rush: you'll need tomorrow's build to get additional information
out of it.

--

Michael
jwmelvin
2014-11-02 14:21:13 UTC
Permalink
mherger wrote:
> > Could you please re-try after enabling debug logging for
> database.sql?...Oh, no rush: you'll need tomorrow's build to get
> additional information out of it.

I installed the 2014-11-02 build, enabled debug logging for
database.sql, and rescanned. The log is 'here'
(http://www.melvincentral.com/xfer/scanner_2014-11-02.log).

Thanks. -Jason


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-11-04 04:36:10 UTC
Permalink
>>> Could you please re-try after enabling debug logging for
>> database.sql?...Oh, no rush: you'll need tomorrow's build to get
>> additional information out of it.
>
> I installed the 2014-11-02 build, enabled debug logging for
> database.sql, and rescanned. The log is 'here'
> (http://www.melvincentral.com/xfer/scanner_2014-11-02.log).

Thanks. Unfortunately it looks as if you either didn't run the latest
code or (even more likely) did not enable debugging for database.sql. I
can't see a single SQL statement being logged.

--

Michael
jwmelvin
2014-11-04 12:43:23 UTC
Permalink
mherger wrote:
> Unfortunately it looks as if you either didn't run the latest code or
> (even more likely) did not enable debugging for database.sql. I can't
> see a single SQL statement being logged.

I'm not sure what went wrong but it was the 10-31 build. Sorry about
that.

Now:
Code:
--------------------
Logitech Media Server Version: 7.9.0 - 1415020733 @ Tue Nov 4 03:58:50 UTC 2014
Hostname: rocco
Server IP Address: 192.168.1.75
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.20.1 - x86_64-linux-gnu-thread-multi
Audio::Scan: 0.95
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 3
--------------------


With these settings:16587

And the log is 'here'
(http://www.melvincentral.com/xfer/scanner_2014-11-04_1.log).

But that doesn't look to have the right entries either?

With these settings: 16588

'This log' (http://www.melvincentral.com/xfer/scanner_2014-11-04_2.log)
is generated, which seems the same.


+-------------------------------------------------------------------+
|Filename: settings_2014-11-04_2.png |
|Download: http://forums.slimdevices.com/attachment.php?attachmentid=16588|
+-------------------------------------------------------------------+

------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-11-04 13:54:18 UTC
Permalink
> With these settings:16587
>
> And the log is 'here'
> (http://www.melvincentral.com/xfer/scanner_2014-11-04_1.log).
>
> But that doesn't look to have the right entries either?

Oh my... checked the server code again. And it seems the scanner is
started with the scan* debug statements only... I'm sorry for the
hassle. Will change the FTS indexer to use those flags instead of
database.sql.

Could you please try with tomorrow's build again?

--

Michael
jwmelvin
2014-11-05 19:02:29 UTC
Permalink
mherger wrote:
> >Could you please try with tomorrow's build again?

With this build:

Code:
--------------------
Logitech Media Server Version: 7.9.0 - 1415129187 @ Wed Nov 5 03:59:22 UTC 2014
Hostname: rocco
Server IP Address: 192.168.1.75
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.20.1 - x86_64-linux-gnu-thread-multi
Audio::Scan: 0.95
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 3
--------------------


Now the scan completes without terminating! But in any event, 'here'
(http://www.melvincentral.com/xfer/scanner_2014-11-05.log) is the log,
with debug turned on for database.sql and for all four scan entries.


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-11-05 23:40:05 UTC
Permalink
> Now the scan completes without terminating! But in any event, 'here'
> (http://www.melvincentral.com/xfer/scanner_2014-11-05.log) is the log,
> with debug turned on for database.sql and for all four scan entries.

Ouch... your log file shows me there is a bug in the playlist indexer
which wouldn't return anything for the playlist tracks. This might
actually have saved the scanner from crashing. But I'm pretty sure
tomorrow's build would crash it again then... could you please give
tomorrow's build yet another try? Thanks a lot!

--

Michael
jwmelvin
2014-11-06 14:22:52 UTC
Permalink
mherger wrote:
> could you please give tomorrow's build yet another try? Thanks a lot!

Okay, looks like there is some more info now.

Using this build with database.sql and scan debugging during a rescan:

Code:
--------------------
Logitech Media Server Version: 7.9.0 - 1415230120 @ Thu Nov 6 03:56:16 UTC 2014
Hostname: rocco
Server IP Address: 192.168.1.75
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.20.1 - x86_64-linux-gnu-thread-multi
Audio::Scan: 0.95
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 3
--------------------


The log is 'here'
(http://www.melvincentral.com/xfer/scanner_2014-11-06.log).


------------------------------------------------------------------------
jwmelvin's Profile: http://forums.slimdevices.com/member.php?userid=2564
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-11-06 14:51:18 UTC
Permalink
> Okay, looks like there is some more info now.

And that's where it stops? There's no error message, not even a
suspicious last SQL query to be run... Would you mind sharing your
library.db with me? Thanks a lot!

--

Michael
pippin
2014-10-25 08:13:53 UTC
Permalink
As far as I can see an album is returned if it contains a song that has
a keyword on it or an artists that has a keyword on it. Is this correct?
It might not always be immediately obvious since at least in some cases
the keyword then doesn't appear in the description of the album, you
have to open it.
Is this the expected behavior? I'm not saying it should not, just feel
for the case of albums it might require some semantics discussion.
It leads to somewhat unexpected results for me but that _might_ be
because I have "albums" of the SXSW showcases which cover pretty much
every keyword you could think of so these always show up, short of them
the results might actually be meaningful....
Or would one expect only albums with the keyword directly on the album
(lacking additional album tags this probably means "in the album name")
and open albums for a certain track through the track's context menu?
OTOH this is probably what you want when adding criteria like bitrate or
file format since if you want to find the flac version of an album you
do want the album again, not just the tracks...



---
learn more about iPeng, the iPhone and iPad remote for the Squeezebox
and
Logitech UE Smart Radio as well as iPeng Party, the free Party-App,
at penguinlovesmusic.com
*New: iPeng 8, the Universal App for iOS 7 and iOS 8*
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-25 10:17:15 UTC
Permalink
How long should it take to do the initial indexing for full text ? Mine
have done one dot in 19 minutes in a relatively smallish collection ?

A thought I hope not Music Brain track id or album id or artist id tags
are included ,they are very long text strings of gibberish ? :)




Discovering files/directories: /media/music1 (47598 of 47598)
Complete 00:00:44

Scanning new music files: /media/music1 (57 of 57) Complete
00:00:03

Discovering playlists: /media/Playlists (17 of 17) Complete
00:00:00

Smart Mix Profile export (57 of 57) Complete 00:00:00

Album cover lookup (77 of 77) Complete 00:00:16

Artist picture lookup (1503 of 1503) Complete 00:00:57

Building full text index (0 of 5) Running 00:18:55



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-25 10:33:42 UTC
Permalink
False alarm this went down faitly well tok only 35 minutes extra .


Discovering files/directories: /media/music1 (47598 of 47598)
Complete 00:00:44

Scanning new music files: /media/music1 (57 of 57) Complete
00:00:03

Discovering playlists: /media/Playlists (17 of 17) Complete
00:00:00

Smart Mix Profile export (57 of 57) Complete 00:00:00

Album cover lookup (77 of 77) Complete 00:00:16

Artist picture lookup (1503 of 1503) Complete 00:00:57

Building full text index (5 of 5) Complete 00:34:21

Create library views (5 of 5) Complete 00:00:03

Find updated coverart files (449 of 449) Complete 00:00:03

Pre-caching Artwork (6 of 6) Complete 00:00:05

Database Optimize (2 of 2) Complete 00:00:28

The server has finished scanning your media library.
Total Time: 00:37:00 (2014-10-25 Saturday / 12:31)



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Apesbrain
2014-10-25 15:11:08 UTC
Permalink
FYI, running V.7.9.0 - 1414184068 on Win XP/Intel Atom today I had to
disable "Full text search" plugin as it caused the "search as you type"
feature to be unresponsive and regular search to be very slow. Before
disabling I did not see a "Building full text index" step in my full
library rebuild which may be a clue. Not sure I'd want to wait 30 extra
minutes for this in any case.


------------------------------------------------------------------------
Apesbrain's Profile: http://forums.slimdevices.com/member.php?userid=738
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-25 15:38:21 UTC
Permalink
> FYI, running V.7.9.0 - 1414184068 on Win XP/Intel Atom today I had to
> disable "Full text search" plugin as it caused the "search as you type"
> feature to be unresponsive and regular search to be very slow. Before

How much memory do you have, how many tracks, and what are the memory
settings in Settings/Advanced/Performance?

--

Michael
Apesbrain
2014-10-25 16:13:11 UTC
Permalink
mherger wrote:
> >How much memory do you have, how many tracks, and what are the memory
> settings in Settings/Advanced/Performance?
I have 2GB of RAM, about 35k tracks, and settings:

[image: http://i.imgur.com/5EiqFqr.jpg]

Again, it may be due to the crash I noted above. At first I didn't make
the connection but if the scanner crashed before completing the
full-text index I suppose the unusual results would be expected. First
time I've seen a scanner crash in as long as I can remember so seemed
safest to just disable full-text.


------------------------------------------------------------------------
Apesbrain's Profile: http://forums.slimdevices.com/member.php?userid=738
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-28 07:47:30 UTC
Permalink
>>> How much memory do you have, how many tracks, and what are the memory
>> settings in Settings/Advanced/Performance?
> I have 2GB of RAM, about 35k tracks, and settings:

Try using the "Maximum" setting.

What about playlists? Do you have any? How large in average? Playlist
indexing is relatively slow (used to be _very_ slow), as it will check
information of every single track, too.

--

Michael
Apesbrain
2014-10-28 17:28:08 UTC
Permalink
mherger wrote:
> What about playlists? Do you have any? How large in average? Playlist
> indexing is relatively slow (used to be _very_ slow), as it will check
> information of every single track, too.

No playlists.


------------------------------------------------------------------------
Apesbrain's Profile: http://forums.slimdevices.com/member.php?userid=738
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-26 20:51:49 UTC
Permalink
HM the search seems sligthly broken did i miss something . The nice
feature that auto completed searches rigth in the search field is gone
it says "no search results" .

In fact all normal no search search is broken ? I can search from the
advanced search but that search only pile up a lot tracks as results no
sorting in artist and albums and then tracks so its not really useful .

I'm using

Logitech Media Server Version: 7.9.0 - 1414184068 @ Sun Oct 26 03:54:27
UTC 2014
Hostname: nostromo.home.lan
IP: 192.168.1.50
HTTP Port: 9000
OS: Red Hat - EN - utf8
Platform: i686-linux
Perl Version: 5.10.1 - i386-linux-thread-multi
Audio::Scan: 0.95
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 3



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
JJZolx
2014-10-27 04:00:50 UTC
Permalink
Mnyb wrote:
> HM the search seems sligthly broken did i miss something . The nice
> feature that auto completed searches rigth in the search field is gone
> it says "no search results" .

I think this may be a function of responsiveness, as discussed above.
What I note on my system is that I occasionally get "no results", but
more often I'll see it get stuck on results that don't include the last
letter entered.


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-27 04:09:07 UTC
Permalink
JJZolx wrote:
> I think this may be a function of responsiveness, as discussed above.
> What I note on my system is that I occasionally get "no results", but
> more often I'll see it get stuck on results that don't include the last
> letter entered.

I also get nothing if I go to the normal search window . I've seen the
responsiveness thing to in an either nigthly you had to wait a while
before results started to drop in . But not now now all of it is gone .

Btw I can't locate the free search text plugin to disable it ? Where is
it .



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
kidstypike
2014-10-27 08:25:13 UTC
Permalink
Mnyb wrote:
>
>
> Btw I can't locate the free search text plugin to disable it ? Where is
> it .

Usual place: Settings > Plugins ?


+-------------------------------------------------------------------+
|Filename: Pic03.jpg |
|Download: http://forums.slimdevices.com/attachment.php?attachmentid=16548|
+-------------------------------------------------------------------+


kidstypike
1 x SB3 - 1 x Boom - 1 x (Squeezebox) Radio - 2 x Touch - 2 x
piCorePlayer
------------------------------------------------------------------------
kidstypike's Profile: http://forums.slimdevices.com/member.php?userid=10436
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-27 09:14:57 UTC
Permalink
kidstypike wrote:
> Usual place: Settings > Plugins ?

I'll check when i get home , thats where they should be :) did not see
it last nigth .



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Mnyb
2014-10-27 20:40:20 UTC
Permalink
Ok with the latest changes fulltext scan actually crashes on one of my
older albums tried 2 times always crash on this album .
I did scan for new and changed .


Code:
--------------------

[14-10-27 21:30:21.5729] Slim::Music::Import::runImporter (485) Starting Slim::Plugin::FullTextSearch::Plugin scan
[14-10-27 21:30:59.6281] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::st execute failed: SQL logic error or missing database
malformed MATCH expression: [type:track w1:file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac] [for Statement "SELECT w10, w5, w3, w1 FROM tracks,fulltext WHERE tracks.url = 'file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac' AND fulltext MATCH 'type:track w1:file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac' AND fulltext.id = tracks.id"]
[14-10-27 21:30:59.6288] Slim::Schema::Storage::throw_exception (122) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (/usr/lib/perl5/vendor_perl/Slim/Schema/Storage.pm line 122)
frame 1: Slim::Schema::Storage::throw_exception (/usr/share/squeezeboxserver/CPAN/DBIx/Class/Storage/DBI.pm line 1006)
frame 2: DBIx::Class::Storage::DBI::__ANON__ (/usr/lib/perl5/vendor_perl/Slim/Plugin/FullTextSearch/Plugin.pm line 295)
frame 3: Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (/usr/lib/perl5/vendor_perl/Slim/Plugin/FullTextSearch/Plugin.pm line 73)
frame 4: Slim::Plugin::FullTextSearch::Plugin::startScan (/usr/lib/perl5/vendor_perl/Slim/Music/Import.pm line 487)
frame 5: Slim::Music::Import::runImporter (/usr/lib/perl5/vendor_perl/Slim/Music/Import.pm line 385)
frame 6: Slim::Music::Import::runScanPostProcessing (/usr/libexec/squeezeboxserver-scanner line 342)
frame 7: (eval) (/usr/libexec/squeezeboxserver-scanner line 342)
frame 8: main::main (/usr/libexec/squeezeboxserver-scanner line 503)

[14-10-27 21:30:59.6522] main::main (346) Error: Failed when running scan post-process: [Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::st execute failed: SQL logic error or missing database
malformed MATCH expression: [type:track w1:file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac] [for Statement "SELECT w10, w5, w3, w1 FROM tracks,fulltext WHERE tracks.url = 'file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac' AND fulltext MATCH 'type:track w1:file:///media/music1/CD/Taj%20Mahal/Taj%20Mahal%20-%20Phantom%20Blues/(You%27ve%20Got%20To)%20Love%20Her%20With%20Feeling.flac' AND fulltext.id = tracks.id"] at /usr/lib/perl5/vendor_perl/Slim/Schema/Storage.pm line 126
]
[14-10-27 21:30:59.6526] main::main (347) Error: Not updating lastRescanTime!

--------------------



I ripped this album in 2008 and probably got the tags from Amazon with
tag&rename . there is some ugly buycdurl tag ?


Code:
--------------------


ALBUM: Phantom Blues
ARTIST: Taj Mahal
BUYCDURL: http://www.amazon.com/gp/redirect.html%3FASIN=B0000000M4%26tag=softpointer-20%26lcode=xm2%26cID=2025%26ccmID=165953%26location=/o/ASIN/B0000000M4%253FSubscriptionId=0RXJS26C80QSDEB56CR2
COMMENT: Encoded by FLAC v1.1.3a with FLAC Frontend v1.7.1
DATE: 1996
ENSEMBLE: Taj Mahal
GENRE: R&B
LABEL: RCA Victor
TITLE: (You've Got To) Love Her With Feeling
TRACKNUMBER: 13
VENDOR: reference libFLAC 1.1.2 20050205

--------------------



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-27 20:59:25 UTC
Permalink
> Ok with the latest changes fulltext scan actually crashes on one of my
> older albums tried 2 times always crash on this album .
> I did scan for new and changed .

Thanks! That's the crash from the log file you sent me in the other
thread. Most likely it's the braces in the URL. I think it's
interpreting them as part of the condition. I'll have to add some sanity
checking around there.

--

Michael
Michael Herger
2014-10-28 07:45:30 UTC
Permalink
> False alarm this went down faitly well tok only 35 minutes extra .

35 minutes is still a lot more than I would expect. Could you please
enable debuging for scan and check which step takes how long? And don't
forget to update, of course.

--

Michael
Michael Herger
2014-10-28 07:43:58 UTC
Permalink
> As far as I can see an album is returned if it contains a song that has
> a keyword on it or an artists that has a keyword on it. Is this correct?

Yes, that's correct.

> It might not always be immediately obvious since at least in some cases
> the keyword then doesn't appear in the description of the album, you
> have to open it.

Agreed.

> Is this the expected behavior?

Yes, it's expected, as the presentation has not changed over the old
search. I was wondering whether there should be a display of some of the
context of the hits, similar to what search engines do. But that would
require a bit more work, might break existin clients, and would
certainly slow down the search. The current implementation tries to work
on the index only, without requiring the db to read the raw data itself
(which is expected to be much slower).

> OTOH this is probably what you want when adding criteria like bitrate or
> file format since if you want to find the flac version of an album you
> do want the album again, not just the tracks...

There certainly can be some more tweaking to the weighting of the
results as well as the filtering itself. Eg. the use of "key*" is a bit
simplistic and rough when you only have a single character (eg. "a*") or
some very common term ("the*"). In my tests with 100k tracks I would eg.
get about 90k hits for these two examples... Some more logic might try
to deal with such keywords, as there are only few words with such high
hit rate, and access to that data is pretty cheap.

--

Michael
JJZolx
2014-10-25 01:43:38 UTC
Permalink
Ok, back to the original premise...

If I'm doing a basic search and I search for 'james', why would I want
to see all 356 albums in my library that have a track on them by anyone
named James? I _would_ expect it to find the following, but nothing
else:

The Best of James Bond: 30th Anniversary Collection (1993) by Various
Artists
The Best of Joe Walsh and The James Gang (1969-1974) (1974) by Joe Walsh


------------------------------------------------------------------------
JJZolx's Profile: http://forums.slimdevices.com/member.php?userid=10
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-25 05:30:42 UTC
Permalink
> If I'm doing a basic search and I search for 'james', why would I want
> to see all 356 albums in my library that have a track on them by anyone
> named James?

Because you've forgotten who covered that track written by James Taylor.

> I _would_ expect it to find the following, but nothing
> else:
>
> The Best of James Bond: 30th Anniversary Collection (1993) by Various
> Artists
> The Best of Joe Walsh and The James Gang (1969-1974) (1974) by Joe Walsh

Your choice. Turn off FTS.

--

Michael
Michael Herger
2014-10-25 07:30:24 UTC
Permalink
> If I'm doing a basic search and I search for 'james', why would I want
> to see all 356 albums in my library that have a track on them by anyone
> named James? I _would_ expect it to find the following, but nothing
> else:
>
> The Best of James Bond: 30th Anniversary Collection (1993) by Various
> Artists
> The Best of Joe Walsh and The James Gang (1969-1974) (1974) by Joe Walsh

Thinking about this a bit more... finding stuff is one problem. But I think the big remaining issue is the sorting/presentation: on Google I don't care to get 100k results, as long as I can find what I'm looking for within the first 10 results. I'd like to achieve the same with the FTS in LMS. Therefore I'd be interested how this sorting fails for you. Doe the tracks you were expecting as the result not show up at all, or in the wrong position? Eg. would you be happier if the two tracks you mentioned were at the top, so you could ignore the long tail?

There is some weighting going on in the background. Titles have higher priority than artists than lyrics etc. But let's say your search term shows up 10x in the comments of some track, but only once in the title of what you're looking for, then you might currently get the wrong results first. Therefore I'd like to understand what you're getting, and why you're getting those results.
Michael Herger
2014-10-25 05:43:57 UTC
Permalink
> Mozart K.466

Ok, this is resulting in a search for "select * from fulltext where
fulltext match 'Mozart* AND K* AND 466*'". I need to improve the string
token parser. In one case it would hit on eg. the bitrate (466kbps) etc.
Need to be a bit smarter.

--

Michael
Michael Herger
2014-10-24 04:55:12 UTC
Permalink
> Another thing I recognized (and I know see again while it restarts):
> r-1414070955 takes MUCH longer to start the server. Usually it takes
> about 15 seconds - know it takes some 10 minutes
>
> So I downgraded to r-1413907472 - it's up in 15 seconds again
>
> upgrade to r-1414070955 once more - 5 minutes ago and it's still
> "starting" in progress...

LMS would re-build the index if it was absent or outdated. As you've had
serious problems with your scan I wouldn't be surprised if it was
re-building it upon the startup.

--

Michael
Michael Herger
2014-10-23 21:30:53 UTC
Permalink
> Unfortunately I only have Erland's 100k library to test with. I'll let
> you know what I get there.

Ok, with Erland's 100k library plus some k of mine:

[14-10-23 23:27:28.5785]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (177) Starting
fulltext index build...
[14-10-23 23:27:28.5815]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (183) Initialize
fulltext table...
[14-10-23 23:27:28.5831]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (189) Create
fulltext index for tracks...
[14-10-23 23:27:55.6936]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (215) Create
fulltext index for albums...
[14-10-23 23:28:02.4535]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (236) Create
fulltext index for contributors...
[14-10-23 23:28:02.9775]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (252) Create
fulltext index for playlists...
[14-10-23 23:28:02.9881]
Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (283) Fulltext index
build done!

34s for all of it on a 5yrs old MacMini with 3GB of RAM...

Could you please send me a copy of your library.db from a regular,
successful scan (with the FulltextSearch plugin disabled)? It would
allow me to run the queries against the library even without the actual
files on disk.

--

Michael
frank1969
2014-10-28 12:03:40 UTC
Permalink
mherger wrote:
> >
> Could you please send me a copy of your library.db from a regular,
> successful scan (with the FulltextSearch plugin disabled)? It would
> allow me to run the queries against the library even without the actual
>
> files on disk.

Sorry, I've been off for a few days. How can I send the library.db to
You? It's about 700 MB size (BEFORE using FTS - after that it's 1,5 GB
so I turned FTS off).

Yesterday I did some testing with the last nightly before FTS.
The long starting time is definitively caused by FTS.

- Using last nightly before FTS: Starting time: 12 sec.
- Upgrading to most recent nightly: Starting time 55 min.
- disabling FTS plugin - restart: Starting time 12 sec. again
- Enabling FTS plugin: Starting time 55 min.

So it seems it takes longer to start EVERY time (not only the first
"initial" time) - so I keep it off and switched back to an old "smaller"
library.db before FTS - recognizing another error in the search engine,
but I open another thread for it.


------------------------------------------------------------------------
frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-28 14:11:05 UTC
Permalink
> Sorry, I've been off for a few days. How can I send the library.db to
> You? It's about 700 MB size (BEFORE using FTS - after that it's 1,5 GB
> so I turned FTS off).

Do you have some web server available, or dropbox or something? And then
make sure you compress it to a zip file.

> - Upgrading to most recent nightly: Starting time 55 min.

Yeah, that's wrong. But hard to tell _what_ is wrong without having more
information about your collection, the scanner.log file, your settings etc.

> So it seems it takes longer to start EVERY time (not only the first

Most likely it's on every start because it crashes at some point.

--

Michael
Mnyb
2014-10-28 16:40:44 UTC
Permalink
Cant try yet i see your commits but no new build yet 28/10 build is the
same as the one I downloaded 26/10 number 1414184068



--------------------------------------------------------------------
Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x
MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3
sub.
Bedroom/Office: Boom
Kitchen: Touch + powered Fostex PM0.4
Misc use: Radio (with battery)
iPad1 with iPengHD & SqueezePad
(in storage SB3, reciever ,controller )
server HP proliant micro server N36L with ClearOS Linux

http://people.xiph.org/~xiphmont/demo/neil-young.html
------------------------------------------------------------------------
Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-24 10:55:19 UTC
Permalink
> Yes, thanks - after 55 minutes it finished full text search (for 420k
> mp3-files). I will now test the feature and report if it was only the
> initial scan that took so "long" or if it's the same with every
> (update-) scan. Great idea, by the way! :-)

I knew FTS might be heavy on your server. But ahm... it's a massive hog
with a large collection. I just tried with ~400k tracks:

Dateien/Verzeichnisse werden erkannt:
/Users/mh/Documents/SqueezeZeugs/Massive Music Library (431973 von
431973) Beendet 00:02:25
Neue Musikdateien werden durchsucht:
/Users/mh/Documents/SqueezeZeugs/Massive Music Library (398652 von
398652) Beendet 00:52:23
Volltext Index erstellen (5 von 5) Beendet 00:12:19
Musiksammlungs-Ansichten erstellen (4 von 4) Beendet 00:01:25
Nach aktualisierten Plattenhüllen suchen (32304 von 32304) Beendet
00:06:03
Datenbank Optimierung (2 von 2) Beendet 00:08:31
Der Server hat das Durchsuchen Ihrer Medienbibliothek abgeschlossen.
Laufzeit: 01:23:06 (Freitag 24. Oktober 2014 / 12:21)

Building the index was pretty quick (compared to what you reported). 12
minutes is nowhere near the 55 minutes you experienced. I could imagine
that disk IO is becoming an issue again: the resulting library.db weighs
a whopping 1.1GB. With the temporary .wal file it created about 2.5GB at
peak times. The fulltext index accounts for about 500MB of the 1.1GB.

Now scan times are one issue. But unfortunately searches are useless
with this large a library. The web UI would constantly time out, while
the server was eating up to 800MB of memory.

I'll see what I can do about this...

--

Michael
Michael Herger
2014-10-24 13:20:23 UTC
Permalink
> Yes, thanks - after 55 minutes it finished full text search (for 420k
> mp3-files). I will now test the feature and report if it was only the
> initial scan that took so "long" or if it's the same with every
> (update-) scan. Great idea, by the way! :-)

How many stored playlists do you have? And how many tracks are in there
(approximately)? I think indexing playlists is brutally slow right now.
There definitely is some room for improvement in many places...

--

Michael
Michael Herger
2014-10-23 20:29:50 UTC
Permalink
Oh, and you do use the "maximum" memory preference in Settings/Advanced/Performance?

Michael



> Am 23.10.2014 um 21:54 schrieb frank1969 <***@no-mx.forums.slimdevices.com>:
>
>
> Great idea but...
> ... after 40 minutes it still says "Volltext Index erstellen (0 von 5)
> Läuft 00:39:45" - and there is only one "dot" shown - so it should
> take hours, dates ...
> I know, i have a 420k track library, but on a high performance PC.
> Any idea how long full text search "usually" should do or should I stop
> and restart so it maybe "hanged`up"?
>
>
> ------------------------------------------------------------------------
> frank1969's Profile: http://forums.slimdevices.com/member.php?userid=31156
> View this thread: http://forums.slimdevices.com/showthread.php?t=102347
>
> _______________________________________________
> beta mailing list
> ***@lists.slimdevices.com
> http://lists.slimdevices.com/mailman/listinfo/beta
pippin
2014-10-23 20:21:38 UTC
Permalink
mherger wrote:
> > Michael, you are awesome!
>
> ....except that it doesn't work with iPeng... Only tested it after I had
>
> it announced here. Web UI should be fine though. I'll take care of it
> asap.
>
> --
>
> Michael

And it probably won't until you tell me how the CLI command for it works
:)



---
learn more about iPeng, the iPhone and iPad remote for the Squeezebox
and
Logitech UE Smart Radio as well as iPeng Party, the free Party-App,
at penguinlovesmusic.com
*New: iPeng 8, the Universal App for iOS 7 and iOS 8*
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
pippin
2014-10-23 20:55:28 UTC
Permalink
Some more logging_

Code:
--------------------

Slim::Control::Request::execute (1890) Error: While trying to run function coderef [Slim::Control::Queries::searchQuery]: [Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db prepare_cached failed: no such column: me.audio [for Statement "SELECT COUNT(1) FROM (
SELECT me.id, me.title FROM (
SELECT FULLTEXTWEIGHT(matchinfo(fulltext)) w, me.id, me.title FROM fulltext, tracks me WHERE fulltext MATCH 'type:track *ABC*' AND me.id = fulltext.id ORDER BY w
) AS me
WHERE me.audio = ? ) AS t1"] at /Users/jorgschwieder/Daten/Business/Dev/iPhone/Source/Libs/Squeez&UE/slimserver/Slim/Schema/Storage.pm line 126

--------------------



---
learn more about iPeng, the iPhone and iPad remote for the Squeezebox
and
Logitech UE Smart Radio as well as iPeng Party, the free Party-App,
at penguinlovesmusic.com
*New: iPeng 8, the Universal App for iOS 7 and iOS 8*
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
pippin
2014-10-23 20:52:03 UTC
Permalink
mherger wrote:
> > Michael, you are awesome!
>
> ....except that it doesn't work with iPeng... Only tested it after I had
>
> it announced here. Web UI should be fine though. I'll take care of it
> asap.
>
> --
>
> Michael

Oh, the problem is probably on your side.
The normal "search" CLI command now seems to be broken:


Code:
--------------------

error = "request failed with error: Bad dispatch!";

--------------------


on this command


Code:
--------------------

{
cmd = (
search
);
params = {
term = Abc;
};
}

--------------------



---
learn more about iPeng, the iPhone and iPad remote for the Squeezebox
and
Logitech UE Smart Radio as well as iPeng Party, the free Party-App,
at penguinlovesmusic.com
*New: iPeng 8, the Universal App for iOS 7 and iOS 8*
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Michael Herger
2014-10-23 21:08:34 UTC
Permalink
> Oh, the problem is probably on your side.

Exactly.

> The normal "search" CLI command now seems to be broken:

I'm almost done with the fix.

--

Michael
atrocity
2014-10-23 18:26:40 UTC
Permalink
mherger wrote:
> Your search no longer is limited to a simple title/name search. With the
>
> FTS you can now easily find track X by artist Y. Just type "hurry love
> supremes" if you want to make sure not to get Phil Collins' version.

I've wanted this forever, thank you so much!!!


------------------------------------------------------------------------
atrocity's Profile: http://forums.slimdevices.com/member.php?userid=16009
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Eric_2014
2014-10-23 20:37:44 UTC
Permalink
mherger wrote:
> Something new to play with: full text search (FTS).
>
> Your search no longer is limited to a simple title/name search. With the
>
> FTS you can now easily find track X by artist Y. Just type "hurry love
> supremes" if you want to make sure not to get Phil Collins' version.
>
> Other examples:
>
> "mp3" - only find tracks in lossy mp3 (or with mp3 in the name - oh
> well)
>
> "playlist elvis" - find playlists with tracks by Elvis, or with Elvis in
>
> the name or a track's name
>
> "flac 88.2 mozart" - get that highres classical stuff
>
> The search code tries to weight results: hits in track titles have
> higher priority than hits in a track's artist or album name. Eg. "lov"
> would rank ABBA's "loving me, loving you" higher than "love me tender"
> from the "greatest love songs, ever" album. Playlist names have priority
>
> over track titles included in the playlist etc.
>
> FTS is available from both the web UI's "live search" as well as the
> regular search menus. Please give it a try and let me know what you
> think about it.
>
>
> A few caveats:
>
> - FTS most likely will be a bit heavier on your system, with regards to
>
> both the CPU as well as memory. I guess it shouldn't be used without the
>
> highmem preference being set to high or max.
>
> - FTS only works with SQLite. If you decided to use MySQL for whatever
> reason, then I'm sorry, this won't work, as it's based around SQLite's
> FTS extension
>
> FTS is implemented as a plugin (mostly). If you don't like it, then just
>
> disable the plugin to get the old behaviour back.
>
> --
>
> Michael

Sweet!


------------------------------------------------------------------------
Eric_2014's Profile: http://forums.slimdevices.com/member.php?userid=63470
View this thread: http://forums.slimdevices.com/showthread.php?t=102347
Loading...