|
|
|
|
|
|
|
| |
|
|||||
| |
|
|
|
|
|
|
| |
||||||
 |
 |
 |
 |
 |
 |
Feeding News Into Your Server
This section describes the settings necessary to get news on to the DNews server from your provider. There are also suggestions on finding a provider and a test sequence to follow for fixing problems with incoming feeds.
Where can I get a News Feed?
Click on this reference for suggestions
Setting up a basic sucking feed
This is the default configuration and DNews comes setup to do this straight out of the box.
In dnews.conf set the nntp_feeder to the site that you are going to suck from, set the
myname variable to the name of the machine that you are running DNews on. Also
specify when the sucking feed should be updated with the update_at setting.
e.g.
nntp_feeder pull-feed-out.internetmci.com *
update_at 20,40 *
The update_at setting is the time(s) at which to start sucking, the first parameter is
the minutes, and the last parameter is the hours, and the '*' matches all times, so 20,40
*, means twice an hour every hour 1:20,1:40,2:20,2:40...
Then tell DNews to get any new groups from the sucking feed.
tellnews getgroups 1
Setting up a basic IHAVE (traditional) news feed
In access.conf, give your feed site access to send you ihave messages, sometimes you will have to give several names or a wild card to allow all the news servers that may send you items. Also, it is wise to add the ip numbers if possible then you can turn reverse DNS translation off if necessary. If possible avoid using a wild card as wild cards are inherently unsafe for your incoming feed and DMULTI will not use them:
news.feed.name.com:ihave:::*
223.232.11.*:ihave:::*
In dnews.conf, comment out the nntp_feeder definition and add a line 'cachedall true'
# nntp_feeder ...
cachedall true
In newsfeeds.conf, replace
site $NNTP_FEEDER
groups *
builtin
posting
With
site my.feeder.site
groups *
onlyposts
builtin
And correct the ME feed, the default one does not allow any binary groups in, if you really want a full feed fix this by removing the !*binary*
For an IHAVE feed you must get an active file
You MUST get a copy of the active.dat file from your provider. The file is simply called the 'active' file on INN or CNEWS. You need to FTP the file and copy it to the workarea directory defined in the dnews.conf file and rename it as active.add (Not active.dat). We do not supply an active file with the installation kit because the list of valid news groups is different in every country in the world. The full list is a combination of world wide groups and local groups. So you should get an active file from your own provider if at all possible. If they will not give you one, then there is a reasonable one on our ftp site that you can use.
If you are lucky this command may be able to fetch the list of groups from your feed site:
tellnews getgroups upstream.feed.site
ftp> ascii
ftp> cd /usr/local/news
ftp> get active
ftp> quit
(NT) > copy active c:\dnews\work\active.add
(NT) > net start dnewssvc
> tellnews group_merge
There is a copy of this file on our ftp site that is always kept up to date with the 'standard' news groups so you may want to use it.
ftp ftp.netwinsite.com Username: anonymous Password: your@email.address ftp> cd pub/dnews ftp> ascii ftp> get active active.add ftp> quit copy active.add \dnews\work tellnews group_merge
You can use the same procedure again at any time in the future to add newsgroups to your server. NEVER edit the active.dat file by hand and if you do edit it (I told you not to) then do it while DNews is stopped or the changes will be ignored. The active.dat file contains 'group id's which are allocated to each new news group, if you mess these up then the items will be unreadable.
Downstream sites modifying their own entry in newsfeeds.conf using GUP
Sucking in UUCP or Satellite articles
DNews can read UUCP bag files in standard rnews format, these are often used by satellite systems, the satellite receiver software may write 'uucp' files to a directory on your system, DNews can then automatically scan and read in items that appear in this directory. To read these files just specify in dnews.conf the directory and file names that DNews should scan, e.g. in dnews.conf
suck_uucp d:\dnews\uucp\*.bag
The rnews file format is a collection of articles in a single file, each one starts
#! rnews nnn
(News article 1)
#! rnews nnn
(News article 2)
Where 'nnn' is the length of each news article in bytes, counting one character for each end of line. In general the file should be in 'unix' format, that is there should be no 'cr' characters, each line should end in a line feed character.
You may want to use this feature as a simple way of sending messages into DNews from your own software, if so you can use a length of '0' in which case DNews will try and guess where one items ends and the next starts based on the #!rnews header.
If you are using 'real' uucp and need to compress the batches and send them, etc., then you will find the two utilities in this tar file useful. It includes the shell scripts to compress the batches and send them, and to take incoming batches and store them in the correct directory for DNews to process.
ftp://ftp.netwinsite.com/pub/dnews/uucp.tar.gz
Adding a sucking feed for a subset of groups
To add a sucking feed there are two legs to consider, sucking the articles in, and sending the articles out. Here is an example of sucking some local groups bank.* from a second server news2.bank.com, and sending any posts back to that server:
In dnews.conf
nntp_feeder main.upstream.site *,!bank.*
nntp_suck2 news2.bank.com bank.*
update_at2 20 *
In newsfeeds.conf
site me
groups *
site main.upstream.site
type builtin
groups *,!bank.*
posting
site news2.bank.com
type builtin
groups !*,bank.*
posting
Then issue this command to fetch the news group names from the new site.
tellnews getgroups 2
Adding a sucking feed for all groups
Some people want to suck news from two different providers, this will 'fill in the gaps' if items are missing from one of the servers. Here is an example of sucking all the groups from two servers. We don't strongly recommend this option but it will work if you really want to do it. If you are doing this you will want to use the new concurrent pulling module, to do this set 'pull true' in dnews.conf see this page for more info on the pull mode settings. The PULL setting requires DNews 5.3 or later.
In dnews.conf
nntp_suck2 news-out.byke.com *
pull true
update_at2 20 *
In newsfeeds.conf
site me
groups *
site main.upstream.site
type builtin
groups *
exclude second.upstream.site
There's no need to send back to the second site, the items will arrive there via the usual news distribution.
Sucking from suck.news.com and taking a feed from ihave.news.com
In dnews.conf
nntp_feeder suck.news.com rec.*,news.* (or whatever groups you want to suck)
cachedall false (If you want to suck all groups available on the sucking site, then
you could set this to TRUE, it's optional, false is probably best to start with if you are
not sure)
in newsfeeds.conf
...
site $nntp_feeder
groups rec.*,news.*
builtin
exclude ihave.news.com
site ihave.news.com
groups *
builtin
exclude suck.news.com
in access.conf
ihave.news.com:Ihave:::*
This allows you to configure DNews to suck certain groups in 'header only' mode. In this mode DNews only sucks the header of the items and then fetches the body of the message when the item is read. A cache is used so if another user reads the same body it is not requested again.
Click here for details on header sucking
Adding an incoming IHAVE feed is very easy, you simply add to the access.conf file an entry to let that server send you items.
second.sending.site:ihave:::*
The only thing you need to consider is if you wish to route items from this server to your other providers. If you don't want to route these items then you must set your exclude parameters carefully so that no messages are routed between the sites e.g. or use the onlyposts setting to stop DNews sending any non local posts to it.
site me
groups *
site main.upstream.site
type builtin
onlyposts
groups *
exclude second.sending.site
site second.sending.site
type builtin
onlyposts
groups *
exclude main.upstream.site
If you do wish to route news between these two servers then you should be using DFeed to handle the volume, e.g.
...
site main.upstream.site
type DFeed
groups *
site second.sending.site
type DFeed
groups *
Spam protection system. Their are many options in DNews for controlling spam, click here for a summary and technical details
Example of full feed from UUNET
-------dnews.conf---------
# nntp_feeder your.feeder.ohio.edu
cachedall true
------newsfeeds.conf cut here ----------
site me
groups *
site nntp-in.uu.net
type builtin
groups *,!local.*
exclude uunet
site YOUR.MAIL.GATEWAY
smtp
----access.conf cut here and change your domain and ip numbers ----
# Don't let the world connect :-)
*:logoff:::*
# Do let your domain and ip range connect **** CHANGE THIS BIT ***
*.your.domain:Read,Post:::*
161.232.32.*:read,post:::*
# Do let your news feed from UUNET send you items
nntp-out1.uu.net:ihave:::*
nntp-out2.uu.net:ihave:::*
nntp-out3.uu.net:ihave:::*
nntp-out4.uu.net:ihave:::*
nntp-out5.uu.net:ihave:::*
nntp-out6.uu.net:ihave:::*
nntp-out7.uu.net:ihave:::*
nntp-out8.uu.net:ihave:::*
nntp-out9.uu.net:ihave:::*
nntp-out10.uu.net:ihave:::*
nntp-out11.uu.net:ihave:::*
nntp-out12.uu.net:ihave:::*
nntp-out13.uu.net:ihave:::*
nntp-out14.uu.net:ihave:::*
nntp-out15.uu.net:ihave:::*
Help! I'm not getting any news
There are several things that can be wrong. Here are check lists for IHAVE and SUCKING feeds - go through the appropriate check list to fix your feed.
IHAVE feeds only - Testing & Fixing
Q: Is someone sending you a feed?
A: You must get someone to send a news feed to you, click here for more information.
Q: Is the feed being rejected due to a problem with access.conf?
A:Search your dnews.log file for 'Refusing' or 'Transfer' , if you find this then it is
likely that either you have not put the correct IP address in access.conf, or you have
used a name but disabled reverse dns checking so DNews cannot lookup the name. Or if you
find 'Transfer' then you haven't given IHAVE access to that host in access.conf (post
access is not enough).
Also DNews may find the 'real' name of an ip address is not the same as the name your provider told you, you can always use the name listed in the 'Refusing' error message to correct your access.conf file, e.g. if you find
Refusing connection from {news3.sprint.net}, ...
Then you would add to the END of your access.conf
news3.sprint.net:ihave:::*
Remember access.conf is sensitive to the order of the lines, the LAST matching rule is applied.
ALSO NOTE: If you use dmulti, then you must restart DNews after correcting access.conf, and you must NOT use wild card name entries, e.g. news*.sprint.net should not be used, but 1.2.3.* would be fine.
Once your access.conf is correct you should see your upstream site connect and see a Chan... entry in the status output with your upstreams providers ip number and possibly name, if you don't see this then it's time to ring up your upstream site and ask what error they get when they connect to your system.
Q: Do you have an active.dat file?
A: An active.dat file is the list of newsgroups, with a full news feed this is between 20,000 and 80,000 groups.
With an IHAVE feed DNews cannot get an active.dat file automatically, it must be FTP'd by hand and placed in the work directory. Look in dnews.log for the following line about 1 page down from the start.
init: Server initialized and running, 25748 groups
If it reads '0 groups' then you need to ftp an active file from your feed provider, copy it to c:\dnews\work\active.add or wherever your 'workarea' directory is, this is defined in your dnews.conf by the 'workarea' symbol. It is essential that you copy the file to active.add and NOT active.dat, never never never over write or mess with the active.dat file directly, it's part of the database :-)
copy feeders_active_file c:\dnews\work\active.add
tellnews group_merge
There is an example active file on ftp.netwinsite.com/pub/dnews so if you can't get one from your provider you can get one from us.
Sometimes DNews can get this file for you, it depends on your upstream providers system, try this command on the off chance it works before doing the above steps: tellnews getgroups upstream.news.server
Q: Is your ME feed stopping the items from coming in?
A: In newsfeeds.conf there is a special feed at the top, it is your 'me' feed, which is actually used for controlling what your news server will accept. This may seem confusing, but all the rules that can be used to 'send out' a news feed are just as useful to control what is accepted on your machine.
The 'me' feed MUST be the first feed in the file. Here is an example that will work:
site me
groups *
If you specify a list, then make sure you don't have spaces between the items as this will not work, e.g. rec.*,comp.* will work but rec.*, comp.* will not.
However, we recommend that you control your incoming groups to limit them just to those that are relevant and meaningful. Here is a list that will allow most of the standard news trees and block out some of the rubbish and foreign groups:
site me
groups rec.*,alt.*,news.*,comp.*,biz.*,misc.*,sci.*,soc.*,talk.*
Some ISP's brag about having 30,000 or 40,000 news groups. If you want to provide your customers with a good service then a smaller list is much better, having a huge list of inane or (other peoples) local newsgroups does not make reading news easier for your customers as they have to wait twice as long to download the list of groups.
Q: Is the group incorrectly set as 'moderated'?
telnet your.news.server 119 list active problem.group.name problem.group.name 14306 13982 m
The 'm' at the end means it's moderated, so if you see that then fetch this file ftp://ftp.netwinsite.com/pub/dnews/fixmoderators.sh and run it. This file corrects most of the commonly incorrect groups, however it may add groups you don't want, so first scan the file by hand to see if it contains heirachies you don't carry. This script will work on NT or Unix, it may take some time to run :-)
You should also modify your CONTROL.CONF file to prevent impersonators from causing this problem again, to do this modify all the 'doit' rules to 'log' or install PGP to check the validity of control messages, see https://netwinsite.com/dnews/pgp.htm for details.
Q: I still can't find the problem, where is the news going?
A: If you do have an incoming feed (this will show up as a chan line in the status with the name of your upstream provider) then use the command: tellnews stats_in to get a summary of the incoming news, it may show all the incoming messages are being rejected, in this case examine dnews.in to find the reasons for the rejections.
As a last resort send dnews.conf, newsfeeds.conf, dnews.log, dnews.in to support@netwinsite.com with a description of your problem and the site you are expecting your feed to come from etc...
SUCKING feeds only - Testing & Fixing
Q: Do you have any newsgroups defined?
A: Normally with a sucking feed DNews will fetch the list of newsgroups automatically, however if you had something setup wrong initially and have since fixed it you may need to use the command:
tellnews getgroups
You can check that DNews has got the groups by using the command:
tellnews status
...
Groups Xover n nMB, cached nn of nnnnnn, GID ....
If nnnnnn is zero then try the getgroups command, if it doesn't help look in dnews.log to see what went wrong, it's likely DNews can't connect to your upstream server, test it manually as described below. It's also possible that site doesn't support 'sucking' feeds in which case you need to to either use another site to suck from, or get a traditional news feed.
Q: Is the connection to nntp_feeder working?
A: This is the most likely problem. If you have zero groups as described above and
getgroups doesn't fix it then DNews cannot talk to your upstream site. Even if you do have
groups DNews may not be able to talk any more. So look in dnews.log for an obvious error,
e.g.:
tellnews update_clear
tellnews cached rec.humor 1
tellnews updateone rec.humor
Then look in dnews.log, here are some common errors you might see:
suck: BAD status code returned {you have no permission to talk}
This means your ISP has not given you 'news reader' access - call your ISP.
suck: Open channel 0 to {news.upstream.com}
nntp: Channel closed or didn't open [0] 13 {news.upstream.com}
suck: Drat channel closed or failed to open properly...
The three lines above mean that the connection could not be opened. This implies that either the address 'news.upstream.com' is not valid, or their news server is not running at the moment, or your internet link is not working at the moment.
You should test the TCPIP link to your upstream site using telnet, it should respond as follows
telnet news.upstream.com 119
200 news.upstream.com DNews Version 4.3c, S0, posting OK
group rec.humor
211 623 175829 176451 rec.humor selected
group rec.humor
next
223 175830 <rvto9hcqh1s36@news.supernews.com> article retrieved - request text separately
It should respond as above, if it closes the connection, or doesn't even open the connection then you need to fix this before DNews will work.
Q: Is your ME feed stopping the items from coming in?
A: In newsfeeds.conf there is a special feed at the top, it is your 'me' feed, which is actually used for controlling what your news server will accept. This may seem confusing, but all the rules that can be used to 'send out' a news feed are just as useful to control what is accepted on your machine.
The 'me' feed MUST be the first feed in the file. Here is an example that will work:
site me
groups *
If you specify a list, then make sure you don't have spaces between the items as this will not work, e.g. rec.*,comp.* will work but rec.*, comp.* will not.
However, we recommend that you control your incoming groups to limit them just to those that are relevant and meaningful. Here is a list that will allow most of the standard news trees and block out some of the rubbish and foreign groups:
site me
groups rec.*,alt.*,news.*,comp.*,biz.*,misc.*,sci.*,soc.*,talk.*
Some ISP's brag about having 30,000 or 40,000 news groups. If you want to provide your customers with a good service then a smaller list is much better, having a huge list of inane or (other peoples) local newsgroups does not make reading news easier for your customers as they have to wait twice as long to download the list of groups.
Q: Is there any news on your upstream site in the group in question?
A: Do the items exist on your upstream site? Configure a news reader to read directly from your upstream site, and compare a problem news group. If they more or less match then DNews is working and the problem is on your upstream server. If DNews is missing articles, then try this incantation:
tellnews getold "*" 300
tellnews update
And in dnews.conf add this line:
suck_maxold 1900
Help! I'm not getting any news in alt.binaries.xxx
Check your 'me' feed in the file 'newsfeeds.conf' - this must be the first site defined in the file. By default this feed contains "*,!*binaries*" which will partly block binary news groups, you should remove this so the line reads, "groups *" and then use these commands:
tellnews undelete "*"
tellnews matchfeed
tellnews getgroups
tellnews update
Help! I'm not getting any news in x.y.z
Check dnews.in for rejection messages, and examine your 'me' feed in 'newsfeeds.conf' as described above. With an IHAVE feed the most likely cause for this problem is that the upstream site is not sending messages for that group. Talk with your ISP to confirm if they are sending that group to you.
Another common reason is that the group has been renamed to x.y.q.z and your users aren't aware of the move (or the new group hasn't been created for some reason).
Help! Newsgroup x.y.z isn't up to date
First check to see if it is cached using tellnews showcached.
D:\>tellnews showcached problem.group
200 port DNews Version 5.1a-beta6, S0, posting OK
483 Temporary license expires after Feb-2000
483 Please read LICENSE.TXT for registration information
problem.group start 3 end 2, upto on feed 0, days 0
483 Temporary license expires after Feb-2000
281 Completed
205 closing connection - goodbye!
If it isn't cached then reading an item from the problem.group should set it to cached. If
this not happening then in dnews.conf set...
cache_onxover true
...and that should get it working. You could also set...
cache_ongroup true
... but this is not recommended because it can often be triggered when a group isn't even
read.
You can manually cache the group using the following command
tellnews cached problem.group 10
This will set the group cached for a period of 10 days.
If it is cached (as the results above indicate) then try the following command
tellnews getold "problem.group" 20000
tellnews updateone problem.group
Also check dnews.in for any rejection messages which may explain the missing articles.
If it still fails, then do this:
tellnews debug suck:chan_read:chan_write: (dnews 5.2 and earlier)
tellnews debug suck:pull: (dnews 5.3 and later)
tellnews loglvl debug
tellnews loglvl rotate
tellnews cached "problem.group" 2
tellnews getold "problem.group" 20000
tellnews updateone problem.group
tellnews updateone problem.group (yes type this line in twice!)
Then check the resulting dnews.log and send it to support@netwinsite.com if you require further help identifying the problem.
Help I'm not getting any news from MCI
Recently MCI installed a new news server, which responds slightly differently to most servers. This confuses DNEWS's sucking mechanism due to a bug in DNews. The quick fix is to add this to dnews.conf
suck_stream true
Or upgrade to DNews 4.6g which should also solve the problem.
Help!, I've got missing parts to my binary items
If you are experiencing this problem do the following,
ftp://ftp.netwinsite.com/pub/dnews/feedtest
You can use this utility with a command like this:
tellnews newgroup test.feedtest y
In dnews.conf increase 'too_old' to 2000, temporarily
tellnews reload
./feedtest -group test.feedtest -to site.name:port -size 40000 -items 500
It will print out figures showing the speed it could send news into your system. By testing your server on the server itself and then again on another server in the same LAN, and again outside your LAN on the WAN, you can establish if the server or your network or the wide area network is the problem.
WARNING: Always be careful to send this test feed to a local TEST news group like test.feedtest !!!!.