The following is a brief guide to explain how Usenet news works. We suggest you read thru this in a linear fashion to gain a good basic understanding of News.
The first question you should ask is how does news work, how does a news message get from your computer to every other news server in the world, and vice versa, how does a message get from somewhere in Japan to your machine?
Many people imagine the news servers are connected in a tree-like pattern, with the message going from your machine to the 'top' of the tree and then propagate down to all the other news servers.
This is entirely incorrect. Although simple in concept this would not work well in reality, and because the internet is more of an anarchy than an organized system so the news topology is much more random.
Note that systems are more or less randomly connected together, there is no top. This structure results in three conditions that a server can find itself in.
Imagine a message posted on Goofy. The message is stamped with a 'PATH' header which is added to as it passes thru every machine. If a host name is in the path then the server knows not to send it to that host again.
Here is what happens to the 'path' header as a message travels from machine to machine:
|On Goofy: Path: Goofy|
On Mickey: Path: Mickey:Goofy
On Venus: Path: Venus:Mars:Mickey:Goofy
It may sound dumb but a news server will use this logic even to decide about sending a message straight back to the host it just got the message from. And because machines have multiple names it's possible for this logic to fail, when this happens your news server will try and send every single message back up stream. (A waste of network bandwidth)
Above the message is about to be delivered to 'Earth' from two directions. Here even with the path header working correctly both machines will try and send to 'Earth' (because neither has Earth in its path header). How does Earth know when it has received this message once? Enter the history database...
Each message is guaranteed to contain a unique message ID (The unique message ID is created by the news client when the message is generated and reflects the machine on which the message is generated and the time it is generated). The news server stores these unique message ID's in a database, and with every incoming message it first checks if it currently has that message ID before it stores the new message.
This database which could be 100MBs, has to be very fast, DNews probably has the fastest history database of any news server.
- If the article is in the history database then reject it.
- For each group in Newsgroups: header:
- If the group exists locally, store the item (or link it to stored item), at next free item number for that group.
- Stored: rec.juggling 3028
- Stored: rec.humor 205 (Linked to rec.juggling:3028)
- Stored: alt.magic 4266 (Linked to rec.juggling:3028)
- For each server FEED: if the server is not in PATH header add the article to the FEED file for that server or in the case of live feeds send the item to the specified machine.
NNTP works with simple TEXT commands, using telnet you can run NNTP by hand. On some computers (NT 3.5) you need to type ^J after each command instead of just enter There will be no echo so you have to type carefully :-), here is a list of some of the NNTP commands you might use by hand:
200 184.108.40.206 DNews Version 2.4d, posting OK
article,head,body number|<message id>
|help||Gives a list of valid nntp commands for this server|
|mode reader||Tells it you are a news reader, not a machine trying to send a feed in.|
|list||Gives a full list of news groups, this could take 20 minutes!|
|group x.y||Sets the current group, and shows you the number of articles and the first and last item in each group|
|article nnn||Shows you an entire article, you can specify it by item number or by message|
|ihave||Offers the server a news message, it responds with please send the message, or no thank you.|
|post||Offers the server a NEW article.|
|telnet news.server.name 119|
200 220.127.116.11 DNews Version 2.4d, posting OK
200 DNews, Posting OK.
211 299 101760 104058 rec.humor selected
Here are some important NNTP commands that you should understand:
When a news server wants to send a feed at your news server it would traditionally use the 'ihave' NNTP command. Here is how the traditional IHAVE protocol works:
Send: ihave <messageid>
Waitfor: Wanted or Not Wanted
Send Message (if wanted)
This protocol doesnt allow streaming, as it is necessary to wait until the server responds before sending the message,
The new NNTP commands 'check' and 'takethis' are basically the same as ihave, but the operation is split in half so that a feed can send 100 checks and then send takethis followed by the articles for the ones that the server wanted.
The 'XOVER' command was added to allow a directory of articles in one group to be shown without downloading the
header information from every article in the group. Example:
The server responds with 200 lines, each line contains the interesting header fields for one message separated by tabs.
From: bn946@FreeNet.Carleton.CA (Les Griswold)
Subject: Re: Beaver's pride
Date: 23 Apr 1996 09:06:26 GMT
This is the message here
which Ive actually just made up.
Traditional news servers have the following problems:
For many organizations these problems have made it either not practical to run a local news server or difficult to provide a good news service. DNews was developed to overcome these problems: DNews is easy to install, exceptionally stable and designed to run unattended. It was also designed to be very fast and much more efficient than earlier servers.
In addition to being able to take a full news feed several times faster than earlier news servers, DNews introduces the unique dynamic sucking feed option. In many enviroments where less than about 3000 groups are actively read the Dynamic Sucking Feed can provide substantial performance advantages and savings in network bandwidth and diskspace.
Advantages 1-10% of disk space used
1-10% of network bandwidth used
Messages can be stored much longer
Nightly expire run down from 24 hours to 5 minutes
Much easier to setup at both ends.