Multi Part file completion system (dnews 5.4)

There is a problem with multi part news posts, and the problem is even on a well connected system with multiple feeds there will always be a percentage of items that are not complete, this may be due to any of the following reasons:

• And overloaded server near the source server
• A spam filter that drops parts of the message early on in it's distribution
• The original user not posting all the parts to a message
• A robot or individual cancelling any part of the message.
• Partially expired posts, where some of the parts have expired, the rest are then useless.

These partial posts may make up 50% of multi part messages, as a result users find them 'very' annoying as they spend ages downloading useless parts or just trying to find complete posts.  Also these partial posts waste storage and bandwidth.

Obviously if you have partial posts the first thing you should do is check that you do have a complete news feed etc see https://netwinsite.com/dnews/feedsin.htm for suggestions to fix problems with missing binary parts, however once you have fixed any problems with your feed and the remaining partial posts are real partial posts you may want to implement this feature:

1) First UPGRADE to 5.7b1 or later

2) In dnews.conf add, we recommend AT LEAST 2 part directories, on two different scsi disks, this allows dnews to spread the load ain increase the disk write speed, you can have up to 4 part_dirN settings

    part_dir j:\part
    part_dir2 k:\part2
    part_buckets 200

3) Ideally you want a bigger buffer, about 20gig is probably suitable, part_buckets 200, the max allowed is 2000 or 200gig, if you use the same drives you use for spool areas (sensible enough) then decrease your related diskspace settings by the same amount. Each part bucket is 100mb, so 1gig = 10 buckets.

All incoming multi part posts are stored in this buffer until all their parts have arrived, at that point the post is sent in it's entirety into DNews, this has the following advantages:

• Partial posts are never seen by users
• Partial posts are never sent onto other news servers
• All the parts of an item will always be in order and next to each other in the news group
• The parts will 'almost always' expire together so the entire item dissappears at once.

To monitor the system there is a line in tellnews status which shows how many items are still in the cache and how fast it is rotating, if it rotates too fast then you may miss some complete posts.