Guide to USENET news and DNEWS

The following is a brief guide to explain how Usenet news works and a walk thru the the main configuration files that control DNEWS, we suggest you read thru this in a linear fashion to gain a good basic understanding of News.

How UseNet News works?

The first question you should ask is how does news work, how does a news message get from your computer to every other news server in the world, and versa visa, how does a message get from somewhere in Japan to your machine.

Many people imagine the news servers are connected in a tree like pattern, the message would go up from your machine to the 'top' of the tree and then propagate down to all the other news servers.

This is entirely incorrect. Although simple in concept this would not work well in reality, and because the internet is more of an anarchy than an organized system so the news topology is much more random.

The real topology of news servers   

Note that systems are more or less randomly connected together, there is no ‘top’. This structure results in three conditions that a server can find itself in.

  1. Tails, machines connected to only one server
  2. Feeders, servers connected to 2 or more servers
  3. Loops, servers which are part of a loop, so the same message could arrive from two directions at once.

Imagine a message posted on “Goofy”. The message is stamped with a 'PATH' header which is added to as it passes thru every machine. If a host name is in the path then the server knows not to send it to that host again.

Here is what happens to the 'path' header as a messages travels from machine to machine:

On Goofy: Path: Goofy
On Mickey: Path: Mickey:Goofy
On Mars: Path: Mars:Mickey:Goofy
On Pluto: Path: Pluto:Mars:Mickey:Goofy

On Venus: Path: Venus:Mars:Mickey:Goofy
On Sun: Path: Sun:Venus:Mars:Mickey:Goofy

It may sound dumb but a news server will use this logic even to decide about sending a message straight back to the host it just got the message from. And becuase machines have multiple names it's possible for this logic to fail, when this happens your news server will try and send every single message back up stream. (A waste of network bandwidth)

Above the message is about to be delivered to 'Earth' from two directions. Here even with the path header working correctly both machines will try and send to 'Earth' (because neither has ‘Earth’ in it’s path header). How does Earth know that it already has this message? Enter the history database...

Each message is gauranteed to contain a unique message ID. The news server stores these in a database, and with every incoming message it first checks if it currently has that message ID before it stores the new message.

This database which could be 100MB’s, has to be very fast, DNEWS probably has the fastest history database of any news server.

Pseudo code for processing an incoming message

NNTP Protocol / commands

200 161.29.2.2 DNEWS Version 2.4d, posting OK
help
mode reader
list
group rec.humor
article,head,body number|<message id>
ihave,post
 

NNTP works with simple TEXT commands, using telnet you can run NNTP by hand.. On some computers (NT 3.5) you need to type ^J after each command instead of just ‘enter’ There will be no ‘echo’ so you have to type carefully :-), here is a list of some of the NNTP commands you might use by hand.

helpGives a list of valid nntp commands for this server
mode readerTells it you are a news reader, not a machine trying to send a feed in.
listGives a full list of news groups, this could take 20 minutes!
group x.ySets the current group, and shows you the number of articles and the first and last item in each group
article nnnShows you an entire article, you can specify it by item number or by message
ihaveOffers the server a news message, it responds with please send the message, or no thankyou.
postOffers the server a NEW article.

NNTP example

telnet news.server.name 119
200 161.29.2.2 DNEWS Version 2.4d, posting OK
mode reader
200 DNEWS, Posting OK.
group rec.humor
211 299 101760 104058 rec.humor selected
article 104058

NNTP Protocol & extensions

Here are some important NNTP commands that you should understand:

ihave <message-ID>
check, takethis
post
xover
xhdr

When a news server wants to send a feed at your news server it would traditionally use the 'ihave' NNTP command. Here is how the traditional IHAVE protocol works:

Send: ihave <messageid>
Waitfor: Wanted or Not Wanted
Send Message (if wanted)

This protocol doesn’t allow streaming, as it is necessary to wait until the server responds before sending the message, or skipping.

The new NNTP commands 'check' and 'takethis' are basically the same as ihave, but the operation is split in half so that a feed can send 100 check’s and then send ‘takethis’ followed by the articles for the ones that the server wanted.

The 'XOVER' command was added to allow a directory of articles in one group to be shown without downloading the header information from every article in the group. Example:
xover 1200-1400
The server responds with 200 lines, each line contains the interesting header fields for one message separated by tabs.

A raw NNTP article

From: bn946@FreeNet.Carleton.CA (Les Griswold)
Newsgroups: soc.couples.intercultural,alt.politics.white-power,alt.fan.oj-simpson
Subject: Re: Beaver's pride
Date: 23 Apr 1996 09:06:26 GMT
Message-ID: <4li6ii$fj8@freenet-news.carleton.ca>
NNTP-Posting-Host: freenet3.carleton.ca

This is the message here
which I’ve actually just made up.

Problems

Traditional news servers have the following problems:

  • 20,000 News Groups
  • 128Kbps 24 hours / day
  • 146000 articles / day
  • INN, installation
  • INN, management
  • Disk Use
  • Memory Use

DNEWS Sucking

  • Only groups which users read are sucked
  • All groups are available for the user to choose from without having to ask
  • Standard NNTP ‘reader’ protocol is used, so a feed doesn’t have to be setup
  • Low impact, history lookups avoided, newnews not used

DNEWS sucking results

1-10% of disk space used
1-10% of network bandwidth used
Messages can be stored much longer
Nightly expire run down from 24 hours to 5 minutes
Much easier to setup at both ends.

DNEWS Configuration Files, an outline

Expire.conf - Intelligent Expire settings

Expire example

upto 100% groups * keep 1 default 5 purge 30 maxitems 2000 minitems 10
upto 90% groups * keep 1 default 15 purge 30 maxitems 3000 minitems 100
upto 50% groups * keep 4 default 30 purge 30
upto 10% groups * maxitems 4000 minitems 100
upto 100% groups local.* minitems 10000

For each news group, the whole file is searched to find the LAST rule that applies for the current disk usage, e.g. if we are currently using 89% of available disk space, and this group is called rec.humor, then the

upto 50% groups * keep 4 default 30 purge 30

 

keepHow many days to keep an item even if it has an expiration date in the past
defaultHow many days to keep an item with no expiration date
purgeMax days to keep an item even if it has an expiration date
minitemsIf less than this many items in a group, then don’t delete any items despite what the other rules say
maxitemsIf more than this many items, then delete oldest ones despite what other rules say.

 

Access.conf - Powerful and tricky!

The access.conf file is very powerful, it can be used to control access to your server in many different ways and as a result it is also very tricky to use. The main source of confusion comes because of two features, first the file is passed twice, once when a user connects, and a second time when/if they supply a password. The second tricky bit is that the 'last' matching rule in the file is used. So the order of rules is very significant.

This file can do all of the following:

Access example

*:logoff:::*
*netwin.co.nz:read,post:chrisp:passwd:*
*netwin.co.nz:read,post:::*,!*sex*
161.29.1.*:read,post:::*,!*sex*
news*.mci.net:ihave:::*
mypc.netwin.co.nz:read,post:::*

 

*:logoff:::*Immediately closes a connection from anyone outside our domain
*netwin.co.nz:read,post:chrisp:passwd:*Allows me to read and post to any news group (including the naughty ones) as long as I configure my news reader with the username: chrisp and password: passwd
*netwin.co.nz:read,post::*,!*sex*Allows users to read and post to any newsgroup except ones with the word ‘sex’ in the name.
news*.mci.net:ihave:::*Allows an ihave feed from an MCI machine, e.g. news12.mci.net
mypc.netwin.co.nz:read,post:::*Allows me to read all groups, even the naughty ones, from my own PC as long as it has a reverse DNS entry.

Note: netscape cannot be configured to give a user/password from the same domain that allows connections without user/passwords. This is a netscape bug caused by their having mapped the web security onto news, and as web has the same limitation they have caused the same limitation to become a feature of news.

Note: reverse DNS translation is often not properly setup, don’t rely on it, always give ip numbers as well as names. You can give wildcard numbers like 122.33.44.* to allow everyone from a given subnet. Also consider using the 'nodns true' setting which stops DNEWS even trying to perform the reverse name translation, this can improve performance.

Control.conf - How newsgroups get created

## <message>:<from>:<newsgroups>:<action>
all:*:*:mail
newgroup:*:*:mail
newgroup:tale@*.uu.net:comp.*,news.*,rec.*,sci.*:doit
newgroup:kre@*mu*au:aus.*,melb.*:doit
rmgroup:*:*:mail
rmgroup:tale@*.uu.net:comp.*,news.*,rec.*,sci.*:doit
rmgroup:kre@*mu*au:aus.*,melb.*:doit
 

Who really runs NEWS? Who creates news groups, once a new group is approved how is it created on 10,000 news servers all over the world.

The control.conf file is the key to this. It specifies who else in the world is allowed to create newsgroups on your news server. Control messages are just like normal news messages, except they have a special header:

Control: newgroup rec.humor.really.funny

When DNEWS receives a control message it is checked against the control.conf file. The last matching rule is used to decide what to do. The possible actions include:

 

logLog the comand to a file control_COMMAND.log you would then edit this file to remove any commands you do not wish to have actioned, and then rename it to '.cmd' and run it as a batch process
mail EMail the message to the DNEWS manager
dropJust throw away the message
doit Automatically perform the action

DNEWS Manager

DNEWS manager allows you to perform two major tasks, it lets you configure the above configuration files us a nice windows GUI, and it lets you send tellnews commands to the running news server.

TELLNEWS commands

Tellnews commands are used to control and query a running dnews news server, they are actually handled by dnews itself, the tellnews or dnews manager process simply creates an nntp connection to the server and sends thru the tellnews command and then listens for dnews to send back a response. Tellnews commands are used to: