NetWin{ Products }{ Products }
{ Index } { FAQ } { Configs } { Feeds In } { Feeds Out } { Downloads } { Tellnews }

Full Text Searching - An Extension to Dnewsweb

Warning! To build a really complete index will require free space of about 10-50% the size of your spool area, it will also gobble up significant processor & memory resources. See the file fts.conf to control how much of each item is actually indexed, and limit the list of groups that you send to the indexing service trying to index a full usenet news feed is not a good idea generally.

You can 'index' a subset of groups, possibly just your local news groups which are important to you, this will require very little resources.

What to do:

1) In dnews.conf define the directory to use to build indexes, it must be a disk with lots of free space :-)

fts_dir c:/dnews/fts (NT)

fts_dir /var/spool/dnews/fts (Unix)

2) Create the directory and move or copy fts.conf into it.

(NT)

mkdir c:/dnews/fts

copy /dnews/fts.conf c:/dnews/fts

del /dnews/fts.conf (so you don't mix it up with the real one)

(UNIX)

mkdir /var/spool/dnews/fts

copy fts.conf /var/spool/dnews/fts

3) Install an xmit service to build the indexes:

(NT)

addsvc -del dfts

addsvc -add dfts xmit.exe "dfts ftsfeed -fts -at *5 *"

(UNIX)

(run this as the 'news' user) nohup ./xmit ftsfeed -fts -at "*5" "*" &

(put that in a startup file, use su news -c "..." to run it as user 'news', e.g.

su news -c "nohup ./xmit ftsfeed -fts -at '*5' '*' &"

4) Add an ftsfeed to newsfeeds.conf to send items to the xmit service. Select a list of groups that seem sensible to index, don't index the binary groups :-) (add this to the end of newsfeeds.conf)

site ftsfeed

groups !*,comp.*,rec.*,local.*

5) Install dnewsweb, and in dnewsweb.ini tell it that fts is enabled

search true

6) From Program Manager start the xmit fts service:

In Control Panel, Services, Start the DFTS Service

(UNIX) nohup xmit ftsfeed -fts -at "*5" "*" &

7) Wait for the indexes to start being created. If you have a small system you might want to send your current items into the database. We DO NOT RECOMMEND using this command as it WILL lock up dnews for several minutes:

tellnews refeed ftsfeed (DO THIS AT YOUR OWN RISK)

8) Try the system out:

Extensions added in dnews5.3

In 5.3 the following setting was added (this goes in fts.conf)

    index_time true     (Index the time of each post)

    index_all true         (Treat words as anything other than spaces)

    index_asian true         (Index words with non ascii text in)

    index_numbers true     (Index numbers as well as words and words with numbers in)

If this is defined, then the fts database will include a date stamp, this allows the date based searching options to work.  If you want to add this to an existing database then YOU MUST DELETE your existing FTS database, e.g.

        stop xmit service
        del \fts\*.fts
        start xmit service
        tellnews refeed ftsfeed (optional)

See the template files search.tpl and results.tpl to find the additional form fields you can add to your own tempaltes if you have an existing system.

Notes, or ... What if it doesnt work?

1) New items will not appear immediately as DNEWS does not reload the database very often. You might need to run an expire (tellnews expire) to trigger the initial index to be written, then a few minutes later restart dnews and the search function should come to life. 

2) Check the log file created by xmit, this is the name of the feed you defined in newsfeeds.conf and stored in the workarea, so it's usually on NT, d:\dnews\work\ftsfeed.log  If this file is empty then possibly your xmit parameters are wrong, or you may have failed to start xmit entirely.  A common problem is not using the same site name in newsfeeds.conf as is used on the xmit command line, they must match.

3) If you use password access rules you must add a non password entry so your machine can talk to itself without being asked for a password, e.g.    v.x.y.z:read,post::::* at the end of access.conf

4) Jump start it by creating two empty files in the fts directory, expire.del and expire.do, wait 5 minutes, restart dnews and it should work.


Products Downloads Prices Support Company