![]() ![]() |
||||
![]() |
![]() |
![]() |
![]() |
![]() |
Feeding News Into Your Server
This section describes the settings necessary to get news on to the dnews server in the simplest way possible - from a single provider using either of the two main methods used to transfer news items.
Setting up a basic sucking feed
This is the default configuration and DNEWS comes setup to do this straight out of the box.
In dnews.conf set the nntp_feeder to the site that you are going to suck from, set the myname variable to the name of the machine that you are running dnews on, e.g.
nntp_feeder pull-feed-out.internetmci.com *
myname news.mydomain.com
Setting up a basic IHAVE (traditional) news feed
In access.conf, give your feed site access to send you ihave messages, sometimes you will have to give several names or a wild card to allow all the news servers that may send you items. Also, it is wise to add the ip numbers if possible then you can turn reverse DNS translation off if necessary.
news*.mci.com:ihave:::*
223.232.11.*:ihave:::*
In dnews.conf, comment out the nntp_feeder definition and add a line:
# nntp_feeder ...
cachedall true
In newsfeeds.conf, replace
site $NNTP_FEEDER
groups *
builtin
posting
With
site my.feeder.site
groups *
builtin
And correct the ME feed, the default one does not allow any binary groups in, if you really want a full feed fix this by removing the !*binary*
For an IHAVE feed you must get an active file
You MUST get a copy of the active.dat file from your provider. The file is simply called the 'active' file on INN or CNEWS. You need to FTP the file and copy it to the workarea directory defined in the dnews.conf file and rename it as active.add (Not active.dat). We do not supply an active file with the installation kit because the list of valid news groups is different in every country in the world. The full list is a combination of world wide groups and local groups. So you should get an active file from your own provider if at all possible. If they will not give you one, then there is a reasonable one on our ftp site that you can use.
If you have a Sucking feed then DO NOT do this, DNEWS will fetch the list of groups itself.
ftp> ascii
ftp> cd /usr/local/news
ftp> get active
ftp> quit
(NT) > copy active c:\dnews\spool\active.add
(NT) > net start dnewssvc
> tellnews group_merge
You can use the same procedure again at any time in the future to add newsgroups to your server. NEVER edit the active.dat file by hand and if you do edit it (I told you not to) then do it while dnews is stopped or the changes will be ignored. The active.dat file contains 'group id's which are allocated to each new news group, if you mess these up then the items will be unreadable.
Downstream sites modifying their own entry in newsfeeds.conf
If you have customers who are getting a full or partial 'ihave' news feed from your dnews server they may wish to use gupcgi which is a web utility to allow them to administer the list of groups sent to your server. Please follow this link for more information.
Sucking in UUCP or Satellite articles
DNEWS can read UUCP bag files in standard rnews format, these are often used by satellite systems. To read these files just specify in dnews.conf the directory and file names that dnews should scan. e.g. in dnews.conf
suck_uucp d:\dnews\uucp\*.bag
The rnews file format is a collection of articles in a single file, each one starts
#! rnews nnn
(News article 1)
#! rnews nnn
(News article 2)
Where 'nnn' is the length of each news article in bytes, counting one character for each end of line. In general the file should be in 'unix' format, that is there should be no 'cr' characters, each line should end in a line feed character.
If you are using 'real' uucp and need to compress the batches and send them etc then you will find the two utilities in this tar file useful it includes the shell scripts to compress the batches and send them and to take incoming batches and store them in the correct directory for dnews to process.
ftp://ftp.netwinsite.com/pub/netwinsite/dnews/uucp.tar.gz
Adding a sucking feed for a subset of groups
To add a sucking feed there are two legs to consider, sucking the articles in, and sending the articles out. Here is an example of sucking some local groups bank.* from a second server news2.bank.com, and sending any posts back to that server:
In dnews.conf
nntp_feeder main.upstream.site *,!bank.*
nntp_suck2 news2.bank.com bank.*
update_at2 20 *
In newsfeeds.conf
site me
groups *
site main.upstream.site
type builtin
groups *,!bank.*
posting
site news2.bank.com
type builtin
groups !*,bank.*
posting
Then issue this command to fetch the news group names from the new site.
tellnews getgroups 2
Adding a sucking feed for all groups
Some people want to suck news from two different providers, this will 'fill in the gaps' if items are missing from one of the servers. Here is an example of sucking all the groups from two servers. We don't strongly recommend this option but it will work if you really want to do it.
In dnews.conf
nntp_suck2 news-out.byke.com *
update_at2 20 *
In newsfeeds.conf
site me
groups *
site main.upstream.site
type builtin
groups *
exclude second.upstream.site
There's no need to send back to the second site, the items will arrive there via the usual news distribution.
Sucking from suck.news.com and taking a feed from ihave.news.com
In dnews.conf
nntp_feeder suck.news.com rec.*,news.* (or whatever groups you want to suck)
cachedall false (If you want to suck all groups available on the sucking site, then
you could set this to TRUE, it's optional, false is probably best to start with if you are
not sure)
in newsfeeds.conf
...
site $nntp_feeder
groups rec.*,news.*
builtin
exclude ihave.news.com
site ihave.news.com
groups *
builtin
exclude suck.news.com
in access.conf
ihave.news.com:Ihave:::*
This allows you to configure DNEWS to suck certain groups in 'header only' mode, in this mode dnews only sucks the header of the items and then fetches the body of the message when the item is read, a cache is used so if another user reads the same body it is not requested again.
As new bodies are delivered to the user 'as they arrive' a user who is at the end of a modem may not notice any performance drop.
To implement this for *binaries* you would do this:
Now try reading one of the specified groups.
Note: header sucking does not currently work for dmulti systems, it is assumed that these features are going to be of most use to smaller sites where dmulti is not used, but we will fix this at some point in the future.
An important point to remember is that when useing header sucking dnews is in replicate mode in relation to those news groups, this can have side effects that you might not expect. Also when turning header caching on and off, you have to use the expire_groups.
Related new settings:
Setting | Example | Description |
---|---|---|
header_groups | *binaries*,*warez* | Groups which dnews should suck only headers for. |
header_chan_n | 3 | Number of simultaneous channels to use when fetching bodies, if set too small then long delays will occur when a user reads a group. |
header_body_mb | 100 | How much space dnews can use to cache bodies of header only items. |
header_path | d:/dnews/header | Directory to store cached bodies, defaults to (workarea) |
header_host | 2 | Site to fetch bodies from, nntp_feeder is used by default, use 2 for nntp_suck2 etc... |
header_prefetch | 20 | This specifies how many headers/items to fetch 'on the fly' when an uncached news group is first read, this can be used with a normal sucking feed instead of the 'downloading' message, when a group command is recieved dnews will rush off and fetch this many headers/items before responding to the user, so the user is never faced with an empty news group. |
header_timeout | 30 | How long to try and talk to the upstream site before admitting to the user that item 'xxx' cannot be fetched. |
Use the command tellnews status_hcache to determine how well the cache is working.
Example Output and explanation:
Header requests cached/remote 23/10 5000k/3340k, Size 10/4000 3MB/20MB
23/10 23 items read from cache, 10 items fetched
from upstream 5000k/2340k 5000K read from cache, 2340k Read from upstream 10/4000 10 of the 4000 cache entries are currently used 3MB/20MB 3MB of the cache files are strored, of 20mb permitted. |
Adding an incoming IHAVE feed is very easy, you simply add to the access.conf file an entry to let that server send you items.
second.sending.site:ihave:::*
The only thing you need to consider is if you wish to route items from this server to your other providers, if you don't want to route these items then you must set your exclude parameters carefully so that no messages are routed between the sites e.g.
site main.upstream.site
type builtin
groups *
exclude second.sending.site
site second.sending.site
type builtin
groups *
exclude main.upstream.site
If you do wish to route news between these two servers then you should be using live links to handle the volume, e.g.
site main.upstream.site
type live
groups *
site second.sending.site
type live
groups *
Spam protection system. The following settings can be added to dnews.conf, you can just set 'spam_stop true' or you can fiddle with all the settings and thus tune your system differently to other peoples which may be an advantage as 'spammers' will try and get past some of these rules. (these settings go in dnews.conf)
Setting | Default | Description |
---|---|---|
spam_stop | false | Enables spam protection |
spam_test | false | Stops the spam rules from rejecting items, just logs what it would do. You still need spam_stop true set |
spam_debug | false | Makes the rejectiosn 'noisey' normally the user will not be informed if the post is rejected dnews pretends it accepted the item. But while debugging your spam rules you might want to set this to true so you can check it is working. |
spam_keepn | 5 | Number of rejected items to store in spam_n.tmp files (for reviewing) |
spam_dup_total | 5000 | Number of items to 'remember' in our history of 'duplicate' bodies. Uses about 30 bytes per entry. |
spam_dup_hits | 6 | Number of duplicate bodies (not cross posted) to allow |
spam_dup_cross | 400 | Number of duplicate bodies counting cross posts to allow |
spam_from_hits | 60 | Number of messages from any single user per hour |
spam_from_cross | 1000 | Number of messages counting cross posts from any single user per hour |
spam_from_total | 5000 | Number of users to keep tabs on in our history of duplicate users, uses about 100 bytes per user. |
spam_host | false | Use 'nntp-posting-host' instead of 'from' for the above 3 rules |
In addition to fiddling with all the above settings, you can create a file in the WORKAREA as defined by dnews.conf and add lines to the file, any message containing text matching the lines in the file will be rejected.
The idea is to use things that common spammers place in the body of their messages, this allows you to stop anyone who manages to get past the automatic mechanisms above.
e.g. c:\dnews\spool\filter.dat
sex
1-800-3333-4343
1-800-SEXYPHONE
www.sex.
Make money fast
Here is a sample filter.dat file which has kindly been donated by one of our customers, this will stop about 98% of known spammers at the cost of killing about one in a thousand real news messages. This is a fairly 'strong' filter, you may want to go thru it and remove some of the more 'general' phrases, but if you really hate spam then you might use it as is. Here is a trimmed version which is not quite so strong.
DNEWS uses a 'super' efficient mechanism to test matches in this file so that you can have several hundred lines in the file without performance sufferring, each line must be at least 3 characters long. The matching is not case sensitive and is only in the 'body' and 'subject' of the message.
To prevent wild cross posting use this setting on your 'me' feed in newsfeeds.conf.
site me
groups *
maxcross 10
You can also use the 'accept/reject' rules to reject 'known spammers', click here for details.
Example of full feed from UUNET
-------dnews.conf---------
# nntp_feeder your.feeder.ohio.edu
cachedall true
------newsfeeds.conf cut here ----------
site me
groups *
site nntp-in.uu.net
type builtin
groups *,!local.*
exclude uunet
site YOUR.MAIL.GATEWAY
smtp
----access.conf cut here and change your domain and ip numbers ----
# Don't let the world connect :-)
*:logoff:::*
# Do let your domain and ip range connect **** CHANGE THIS BIT ***
*.your.domain:Read,Post:::*
161.232.32.*:read,post:::*
# Do let your news feed from UUNET send you items
nntp-out1.uu.net:ihave:::*
nntp-out2.uu.net:ihave:::*
nntp-out3.uu.net:ihave:::*
nntp-out4.uu.net:ihave:::*
nntp-out5.uu.net:ihave:::*
nntp-out6.uu.net:ihave:::*
nntp-out7.uu.net:ihave:::*
nntp-out8.uu.net:ihave:::*
nntp-out9.uu.net:ihave:::*
nntp-out10.uu.net:ihave:::*
nntp-out11.uu.net:ihave:::*
nntp-out12.uu.net:ihave:::*
nntp-out13.uu.net:ihave:::*
nntp-out14.uu.net:ihave:::*
nntp-out15.uu.net:ihave:::*
Help! I'm not getting any news
There are several things that can be wrong, here are check lists for IHAVE and SUCKING feeds, go through the appropriate check list to fix your feed. (Info on sucking feeds is a few pages further down)
IHAVE feeds only
Q: Is the group incorrectly set as 'moderated'
telnet your.news.server 119 list active problem.group.name problem.group.name 14306 13982 m
The 'm' at the end means it's moderated, so if you see that then fetch this file ftp://ftp.netwinsite.com/pub/netwinsite/dnews/fixmoderators.sh and run it, this file corrects most of the commonly incorrect groups, however it may add groups you don't want, so first scan the file by hand to see if it contains heirachies you don't carry. This script will work on NT or Unix, it may take some time to run :-)
You should also modify your CONTROL.CONF file to prevent impersonators from causing this problem again, to do this modify all the 'doit' rules to 'log' or install PGP to check the validity of control messages, see http://netwinsite.com/dnews/pgp.htm for details.
Q: Is someone sending you a feed?
A: You must get someone to send a news feed to you, click here for more information.
Q:Is the feed being rejected due to a problem with access.conf?
A:Search your dnews.log file for 'Refusing' or 'Transfer' , if you find this then it is
likely that either you have not put the correct IP address in access.conf or you have used
a name but disabled reverse dns checking so dnews cannot lookup the name. Or if you find
'Transfer' then you haven't given IHAVE access to that host in access.conf, (post access
is not enough).
Also DNEWS may find the 'real' name of an ip address is not the same as the name your provider told you, you can always use the name listed in the 'Refusing' error message to correct your access.conf file, e.g. if you find
Refusing connection from {news3.sprint.net}, ...
Then you would add to the END of your access.conf
news*.sprint.net:ihave:::*
Remember access.conf is sensitive to the order of the lines, the LAST matching rule is applied.
Q: Do you have an active.dat file?
A: An active.dat file is the list of newsgroups, with a full news feed this is between 20,000 and 40,000 groups.
With an IHAVE feed dnews cannot get an active.dat file automatically, it must be FTP'd by hand and placed in the work directory. Look in dnews.log for the following line about 1 page down from the start.
init: Server initialized and running, 25748 groups
If it reads '0 groups' then you need to ftp an active file from your feed provider, copy it to c:\dnews\spool\active.add or wherever your 'workarea' directory is, this is defined in your dnews.conf by the 'workarea' symbol. It is essential that you copy the file to active.add and NOT active.dat, never never never over write or mess with the active.dat file directly, it's part of the database :-)
copy feeders_active_file c:\dnews\spool\active.add
tellnews group_merge
Q: Is your ME feed stopping the items from coming in?
A: In newsfeeds.conf there is a special feed at the top, it is your 'me' feed, which is actually used for controlling what you news server will accept. This may seem confusing but all the rules that can be used to 'send out' a news feed are just as useful to control what is accepted on your machine.
The 'me' feed MUST be the first feed in the file. Here is an example that will work:
site me
groups *
If you specify a list, then make sure you don't have spaces between the items as this will not work, e.g. rec.*,comp.* will work but rec.*, comp.* will not.
However, we recommend that you control your incoming groups to limit them just to those that are relevant and meaningful, here is a list that will allow most of the standard news trees and block out some of the rubbish and foreign groups.
site me
groups rec.*,alt.*,news.*,comp.*,biz.*,misc.*,sci.*,soc.*,talk.*
Some ISP's brag about having 30,000 or 40,000 news groups. If you want to provide your customers with a good service then a smaller list is much better, having a huge list of inane or local newsgroups does not make reading news easier for your customers.
SUCKING feeds only
Q: Do you have any newsgroups defined?
A: Normally with a sucking feed dnews will fetch the list of newsgroups automatically, however if you had something setup wrong initially and have since fixed it you may need to use the command:
tellnews getgroups
You can check that dnews has got the groups by using the command:
tellnews status
...
Groups Xover n nMB, cached nn of nnnnnn, GID ....
If nnnnnn is zero then you try the getgroups command, if it doesn't help look in dnews.log to see what went wrong.
Q: Is the connection to nntp_feeder working?
A: This is the most likely problem. If you have zero groups as described above and
getgroups doesn't fix it then DNEWS cannot talk to your upstream site. Even if you do have
groups dnews may not be able to talk any more. So look in dnews.log for an obvious error,
e.g.:
tellnews update_clear
tellnews cached rec.humor 1
tellnews updateone rec.humor
Then look in dnews.log, here are some common errors you might see:
suck: BAD status code returned {you have no permission to talk}
This means your ISP has not given you 'news reader' access, call your ISP.
suck: Open channel 0 to {news.upstream.com}
nntp: Channel closed or didn't open [0] 13 {news.upstream.com}
suck: Drat channel closed or failed to open properly...
The three lines above, mean that the connection could not be opened, this implies that either the address 'news.upstream.com' is not valid, or their news server is not running at the moment or your internet link is not working at the moment.
You should test the TCPIP link to your upstream site using telnet, it should respond as follows, be sure to specify port 119 in the connection dialogue on NT:
telnet news.upstream.com 119
200 news.upstream.com DNEWS Version 4.3c, S0, posting OK
It should respond as above, if it closes the connection, or doesn't even open the connection then you need to fix before dnews will work.
Q: Is your ME feed stopping the items from coming in?
A: Do the items exist on your upstream site? Configure a news reader to read directly from your upstream site, and compare a problem news group, if they more or less match then dnews is working and the problem is on your upstream server. If dnews is missing articles, then try this incantation:
tellnews getold "*" 300
tellnews update
And in dnews.conf add this line:
suck_maxold 1900
Help! I'm not getting any news in alt.binaries.xxx
Check your 'me' feed in the file 'newsfeeds.conf' this must be the first site defined in the file, by default this feed contains "*,!*binaries*" which will partly block binary news groups you should remove this so the line reads, "groups *" and then use these commands:
tellnews undelete "*"
tellnews matchfeed
tellnews getgroups
tellnews update
Help! I'm not getting any news in x.y.z
Check dnews.in for rejection messages, and examine your 'me' feed in 'newsfeeds.conf' as described above. With an IHAVE feed the most likely cause for this problem is that the upstream site is not sending messages for that group. Talk with your ISP to confirm if they are sending that group to you.
Another common reason is that the group has been renamed to x.y.q.z and your users aren't aware of the move (or the new group hasn't been created for some reason)
Help I'm not getting any news from MCI
Recently MCI installed a new news server, which responds slightly differently to most servers, this confuses DNEWS's sucking mechanism due to a bug in DNEWS. The quick fix is to add this to dnews.conf
suck_stream true
Or upgrade to DNEWS 4.6g which should also solve the problem (4.6g will only be available at the end of April 98)
Products | Downloads | Prices | Support | Company |
---|---|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |