The following is a brief guide to explain how Usenet news works and a walk thru the the main configuration files that control DNEWS, we suggest you read thru this in a linear fashion to gain a good basic understanding of News.
The first question you should ask is how does news work, how does a news message get from your computer to every other news server in the world, and versa visa, how does a message get from somewhere in Japan to your machine.
Many people imagine the news servers are connected in a tree like pattern, the message would go up from your machine to the 'top' of the tree and then propagate down to all the other news servers.
This is entirely incorrect. Although simple in concept this would not work well in reality, and because the internet is more of an anarchy than an organized system so the news topology is much more random.
Note that systems are more or less randomly connected together, there is no top. This structure results in three conditions that a server can find itself in.
Imagine a message posted on Goofy. The message is stamped with a 'PATH' header which is added to as it passes thru every machine. If a host name is in the path then the server knows not to send it to that host again.
Here is what happens to the 'path' header as a messages travels from machine to machine:
On Goofy: Path: Goofy On Mickey: Path: Mickey:Goofy On Mars: Path: Mars:Mickey:Goofy On Pluto: Path: Pluto:Mars:Mickey:Goofy On Venus: Path: Venus:Mars:Mickey:Goofy |
It may sound dumb but a news server will use this logic even to decide about sending a message straight back to the host it just got the message from. And becuase machines have multiple names it's possible for this logic to fail, when this happens your news server will try and send every single message back up stream. (A waste of network bandwidth)
Above the message is about to be delivered to 'Earth' from two directions. Here even with the path header working correctly both machines will try and send to 'Earth' (because neither has Earth in its path header). How does Earth know that it already has this message? Enter the history database...
Each message is gauranteed to contain a unique message ID. The news server stores these in a database, and with every incoming message it first checks if it currently has that message ID before it stores the new message.
This database which could be 100MBs, has to be very fast, DNEWS probably has the fastest history database of any news server.
- If article is in history database then reject
- For each group in Newsgroups: header
- If groups exists locally, store item (or link), at next free item number for that group.
- Stored: rec.juggling 3028
- Stored: rec.humor 205 (Link rec.juggling:3028)
- Stored: alt.magic 4266 (Link rec.juggling:3028)
- For each FEED, if FEED is not in PATH add item to FEED file or in the case of live feeds send the item to the specified machine.
200 161.29.2.2 DNEWS Version 2.4d, posting OK
help
mode reader
list
group rec.humor
article,head,body number|<message id>
ihave,post
NNTP works with simple TEXT commands, using telnet you can run NNTP by hand.. On some computers (NT 3.5) you need to type ^J after each command instead of just enter There will be no echo so you have to type carefully :-), here is a list of some of the NNTP commands you might use by hand.
help | Gives a list of valid nntp commands for this server |
mode reader | Tells it you are a news reader, not a machine trying to send a feed in. |
list | Gives a full list of news groups, this could take 20 minutes! |
group x.y | Sets the current group, and shows you the number of articles and the first and last item in each group |
article nnn | Shows you an entire article, you can specify it by item number or by message |
ihave | Offers the server a news message, it responds with please send the message, or no thankyou. |
post | Offers the server a NEW article. |
telnet news.server.name 119 200 161.29.2.2 DNEWS Version 2.4d, posting OK mode reader 200 DNEWS, Posting OK. group rec.humor 211 299 101760 104058 rec.humor selected article 104058 |
Here are some important NNTP commands that you should understand:
ihave <message-ID> check, takethis post xover xhdr |
When a news server wants to send a feed at your news server it would traditionally use the 'ihave' NNTP command. Here is how the traditional IHAVE protocol works:
Send: ihave <messageid>
Waitfor: Wanted or Not Wanted
Send Message (if wanted)
This protocol doesnt allow streaming, as it is necessary to wait until the server responds before
sending the message, or skipping.
The new NNTP commands 'check' and 'takethis' are basically the same as ihave, but the operation
is split in half so that a feed can send 100 checks and then send takethis followed by the articles
for the ones that the server wanted.
The 'XOVER' command was added to allow a directory of articles in one group to be shown
without downloading the header information from every article in the group. Example:
xover 1200-1400
The server responds with 200 lines, each line contains the interesting header fields for one message
separated by tabs.
From: bn946@FreeNet.Carleton.CA (Les Griswold)
Newsgroups:
soc.couples.intercultural,alt.politics.white-power,alt.fan.oj-simpson
Subject: Re: Beaver's pride
Date: 23 Apr 1996 09:06:26 GMT
Message-ID: <4li6ii$fj8@freenet-news.carleton.ca>
NNTP-Posting-Host: freenet3.carleton.ca
This is the message here
which Ive actually just made up.
Traditional news servers have the following problems:
|
|
1-10% of disk space used
1-10% of network bandwidth used
Messages can be stored much longer
Nightly expire run down from 24 hours to 5 minutes
Much easier to setup at both ends.
upto 100% groups * keep 1 default 5 purge 30 maxitems 2000 minitems 10
upto 90% groups * keep 1 default 15 purge 30 maxitems 3000 minitems 100
upto 50% groups * keep 4 default 30 purge 30
upto 10% groups * maxitems 4000 minitems 100
upto 100% groups local.* minitems 10000
For each news group, the whole file is searched to find the LAST rule that applies for the current disk usage, e.g. if we are currently using 89% of available disk space, and this group is called rec.humor, then the
upto 50% groups * keep 4 default 30 purge 30
keep | How many days to keep an item even if it has an expiration date in the past |
default | How many days to keep an item with no expiration date |
purge | Max days to keep an item even if it has an expiration date |
minitems | If less than this many items in a group, then dont delete any items despite what the other rules say |
maxitems | If more than this many items, then delete oldest ones despite what other rules say. |
The access.conf file is very powerful, it can be used to control access to your server in many different ways and as a result it is also very tricky to use. The main source of confusion comes because of two features, first the file is passed twice, once when a user connects, and a second time when/if they supply a password. The second tricky bit is that the 'last' matching rule in the file is used. So the order of rules is very significant.
This file can do all of the following:
Access example
*:logoff:::*
*netwin.co.nz:read,post:chrisp:passwd:*
*netwin.co.nz:read,post:::*,!*sex*
161.29.1.*:read,post:::*,!*sex*
news*.mci.net:ihave:::*
mypc.netwin.co.nz:read,post:::*
*:logoff:::* | Immediately closes a connection from anyone outside our domain |
*netwin.co.nz:read,post:chrisp:passwd:* | Allows me to read and post to any news group (including the naughty ones) as long as I configure my news reader with the username: chrisp and password: passwd |
*netwin.co.nz:read,post::*,!*sex* | Allows users to read and post to any newsgroup except ones with the word sex in the name. |
news*.mci.net:ihave:::* | Allows an ihave feed from an MCI machine, e.g. news12.mci.net |
mypc.netwin.co.nz:read,post:::* | Allows me to read all groups, even the naughty ones,
from my own PC as long as it has a reverse DNS entry. |
Note: netscape cannot be configured to give a user/password from the same domain that allows
connections without user/passwords. This is a netscape bug caused by their having mapped the
web security onto news, and as web has the same limitation they have caused the same limitation to
become a feature of news.
Note: reverse DNS translation is often not properly setup, dont rely on it, always give ip numbers
as well as names. You can give wildcard numbers like 122.33.44.* to allow everyone from a given
subnet. Also consider using the 'nodns true' setting which stops DNEWS even trying to perform the
reverse name translation, this can improve performance.
## <message>:<from>:<newsgroups>:<action>
all:*:*:mail
newgroup:*:*:mail
newgroup:tale@*.uu.net:comp.*,news.*,rec.*,sci.*:doit
newgroup:kre@*mu*au:aus.*,melb.*:doit
rmgroup:*:*:mail
rmgroup:tale@*.uu.net:comp.*,news.*,rec.*,sci.*:doit
rmgroup:kre@*mu*au:aus.*,melb.*:doit
Who really runs NEWS? Who creates news groups, once a new group is approved how is it created on 10,000 news servers all over the world.
The control.conf file is the key to this. It specifies who else in the world is allowed to create newsgroups on your news server. Control messages are just like normal news messages, except they have a special header:
Control: newgroup rec.humor.really.funny
When DNEWS receives a control message it is checked against the control.conf file. The last matching rule is used to decide what to do. The possible actions include:
log | Log the comand to a file control_COMMAND.log you would then edit this file to remove any commands you do not wish to have actioned, and then rename it to '.cmd' and run it as a batch process |
EMail the message to the DNEWS manager | |
drop | Just throw away the message |
doit | Automatically perform the action |
DNEWS manager allows you to perform two major tasks, it lets you configure the above configuration files us a nice windows GUI, and it lets you send tellnews commands to the running news server.
Tellnews commands are used to control and query a running dnews news server, they are actually handled by dnews itself, the tellnews or dnews manager process simply creates an nntp connection to the server and sends thru the tellnews command and then listens for dnews to send back a response. Tellnews commands are used to: