Guide to USENET News and DNews

The following is a brief guide to explain how Usenet news works. We suggest you read thru this in a linear fashion to gain a good basic understanding of News.

How UseNet News works?

The first question you should ask is how does news work, how does a news message get from your computer to every other news server in the world, and vice versa, how does a message get from somewhere in Japan to your machine?

Many people imagine the news servers are connected in a tree-like pattern, with the message going from your machine to the 'top' of the tree and then propagate down to all the other news servers.

This is entirely incorrect. Although simple in concept this would not work well in reality, and because the internet is more of an anarchy than an organized system so the news topology is much more random.

The real topology of news servers   

Note that systems are more or less randomly connected together, there is no ‘top’. This structure results in three conditions that a server can find itself in.

1. Tails, machines connected to only one server
2. Feeders, servers connected to 2 or more servers
3. Loops, servers which are part of a loop, so the same message could arrive from two directions at once.

Imagine a message posted on “Goofy”. The message is stamped with a 'PATH' header which is added to as it passes thru every machine. If a host name is in the path then the server knows not to send it to that host again.

Here is what happens to the 'path' header as a message travels from machine to machine:

It may sound dumb but a news server will use this logic even to decide about sending a message straight back to the host it just got the message from. And because machines have multiple names it's possible for this logic to fail, when this happens your news server will try and send every single message back up stream. (A waste of network bandwidth)

Above the message is about to be delivered to 'Earth' from two directions. Here even with the path header working correctly both machines will try and send to 'Earth' (because neither has ‘Earth’ in it’s path header). How does Earth know when it has received this message once? Enter the history database...

Each message is guaranteed to contain a unique message ID (The unique message ID is created by the news client when the message is generated and reflects the machine on which the message is generated and the time it is generated). The news server stores these unique message ID's in a database, and with every incoming message it first checks if it currently has that message ID before it stores the new message.

This database which could be 100MB’s, has to be very fast, DNews probably has the fastest history database of any news server.

Pseudo code for processing an incoming message

• If the article is in the history database then reject it.
• For each group in ‘Newsgroups:’ header:
• If the group exists locally, store the item (or link it to stored item), at next free item number for that group.
• Stored: rec.juggling 3028
• Stored: rec.humor 205 (Linked to rec.juggling:3028)
• Stored: alt.magic 4266 (Linked to rec.juggling:3028)
• For each server FEED: if the server is not in PATH header add the article to the FEED file for that server or in the case of live feeds send the item to the specified machine.

NNTP Protocol / commands

Network News Transfer Protocol (NNTP) is a protocol that enables news server and client software to "communicate" with each other to; distribute, inquire, retrieve, and post news articles. NNTP is designed so that news articles are stored in a central database allowing a subscriber to select those items he wishes to read. Indexing, cross-referencing, and expiration of aged messages are also provided.The description of the protocol is embodied in RFC's 977 and 1036 and maintained by the Internet Engineering Task Force (IETF), the standards body that maintains Internet protocols.

NNTP works with simple TEXT commands, using telnet you can run NNTP by hand. On some computers (NT 3.5) you need to type ^J after each command instead of just ‘enter’ There will be no ‘echo’ so you have to type carefully :-), here is a list of some of the NNTP commands you might use by hand:

200 161.29.2.2 DNews Version 2.4d, posting OK
help
mode reader
list
group rec.humor
article,head,body number|<message id>
ihave,post 

NNTP example

NNTP Protocol & extensions

Here are some important NNTP commands that you should understand:

When a news server wants to send a feed at your news server it would traditionally use the 'ihave' NNTP command. Here is how the traditional IHAVE protocol works:

Send: ihave <messageid>
Waitfor: Wanted or Not Wanted
Send Message (if wanted)

This protocol doesn’t allow streaming, as it is necessary to wait until the server responds before sending the message, or skipping.

The new NNTP commands 'check' and 'takethis' are basically the same as ihave, but the operation is split in half so that a feed can send 100 check’s and then send ‘takethis’ followed by the articles for the ones that the server wanted.

The 'XOVER' command was added to allow a directory of articles in one group to be shown without downloading the header information from every article in the group. Example:
xover 1200-1400
The server responds with 200 lines, each line contains the interesting header fields for one message separated by tabs.

A raw NNTP article

From: bn946@FreeNet.Carleton.CA (Les Griswold)
Newsgroups: soc.couples.intercultural,alt.politics.white-power,alt.fan.oj-simpson
Subject: Re: Beaver's pride
Date: 23 Apr 1996 09:06:26 GMT
Message-ID: <4li6ii$fj8@freenet-news.carleton.ca>
NNTP-Posting-Host: freenet3.carleton.ca

This is the message here
which I’ve actually just made up.

Problems

Traditional news servers have the following problems:

• With a full feed a 64kbps internet connection cannot deal with the volume. At 400MB per day you need 4Gig just to hold 10 days of news.
• Large numbers of file open and append operations/second to store each item in a separate file.
• On the average site, only 200 news groups are ever read, or approx. 1-10% of the full news volume.
• INN is comparatively difficult and time consuming to install, set up and maintain, and requires a certain level of unix expertise.
• News isn’t smooth, some days 800MB’s arrive, and so the disk is filled up and the system fails.
• Each news reader requires 1-2MB of RAM, (maybe more if nnrpd has not been linked using sharable images) the INN process itself uses 32MB of ram at least. The result on limited resources is heavy swapping and slow response.  

For many organizations these problems have made it either not practical to run a local news server or difficult to provide a good news service. DNews was developed to overcome these problems: DNews is easy to install, exceptionally stable and designed to run unattended. It was also designed to be very fast and much more efficient than earlier servers.

In addition to being able to take a full news feed several times faster than earlier news servers, DNews introduces the unique dynamic sucking feed option. In many enviroments where less than about 3000 groups are actively read the Dynamic Sucking Feed can provide substantial performance advantages and savings in network bandwidth and diskspace.

DNews Dynamic Sucking Feed Option

DNews dynamic sucking results

Advantages

1-10% of disk space used 1-10% of network bandwidth used Messages can be stored much longer Nightly expire run down from 24 hours to 5 minutes Much easier to setup at both ends.