|
|
|
|
|
|
|
| |
|
|||||
| |
|
|
|
|
|
|
| |
||||||
 |
 |
 |
 |
 |
 |
expire.conf DNews 5.x Expire Rules - Piles - Cyclic File System
Note: Before converting your expire rules from older formats read this entire page carefully.
With version 5.x of DNews you can now use a cyclic database. With this system as new items arrive if the allocated space has been used then the oldest items are simply deleted and replaced with the new incoming items, in this way the system can never 'run out of space' and you always have as much news available to your users as is possible.
Example pile rules in expire.conf
pile 1 use 95% groups * remember 3
The above rule is all you need, however to maximize the use of your resources you may want to tell DNews to use a certain amount of space for binaries, and also to keep important local groups for much longer, e.g.
pile 1 use 20% groups * pile 2 use 60% groups *binaries*,*pictures*,*warez* pile 3 days 90 groups local.*,netwin.*,rec.humor.funny remember 3
Advantages over older expire systems:
Disadvantages:
We believe the advantages far far out weigh the disadvantages so these new expire rules are now the default for new installations of DNEWS. Version 5.x of DNews does obey the old expire rules so you don't have to convert to these new rules but we recommend it.
Normally DNews stores items in buckets, e.g. db_23.itm, each bucket is about 10MB, when you switch to the new system DNews adds on the PILE number, up to 30 piles can be defined normally four or five should be enough. The bucket files are then called db_23_nn.itm where 'nn' is replaced by the pile number.
You can specify for each pile, a wildcard list of groups, and a percentage of the entire spool area that can be used for that pile, if the usage ever grows above this percentage then the oldest bucket will simply be deleted. There is still a nightly expire process as DNews still needs to modify the indexes but this is very fast.
To convert your existing database to the new format you should do the following:
Note: During the rebuild_piles your server won't accept incoming newsfeeds and will display some odd messages in the log file which can usually be ignored, wait until the rebuild_piles finishes before panicking :-)
There is a FAST conversion method that avoids the rebuild_piles step, but it only works
if you are happy with your existing news all being placed in a single pile, (pile 1) and
then expiring as required by that rule, so you wouldn't want to do this if you had
local groups that must be kept for more than a few days.
Warning: This fast conversion method will delete old items in local groups, so
you may not want to do this, if you have long term local groups, then use the slow
conversion method above!!!!
Here's what you do:
Done, it should then be working, and tellnews showpiles should show all current data is
in pile 1, and incoming news will get placed in the correct piles. And news should be
readable, if you are running dmulti you must restart DNews so the slaves notice whats gone
on.
Then over the next 2-3 days reduce the size of pile 1 and increase the other pile sizes
respectively. Or if you are not too worried just set the pile sizes the way you want now
and you may loose 50% of your current news as pile 1 won't be big enough for all the
existing news.
As the pile a news item is in is chosen when the item arrives, you cannot simply change the expire rules and expect currently stored items to expire faster or slower, and more importantly you cannot change the pile numbers without disasterous affects (e.g. swapping pile 1 and pile 2 around).
You can safely change the 'use %' or 'days' for any existing pile, and you can safely add piles.
You can add or remove group names or wild cards to a pile, but it will only affect newly arriving news.
If you really want to affect existing piles, then the trick is to run the tellnews command 'rebuild_piles' after modifying the expire.conf values and doing a reload. However this can take upto about 1 hour per gig of news to complete.
As buckets now have the PILE number in the name you can backup that part of the DNews database, for example with the following expire.conf sample below you could backup the local.* newsgroups with a command like this:
copy (spool_directory)\db_*_3.itm \backup
The new format of rules in expire.conf is:
pile n [use xx%] [days nn] groups wildcard_list
e.g.
pile 1 use 20% groups * pile 2 use 1% groups None pile 3 use 1% groups None pile 4 use 60% groups *binaries*,*pictures*,*warez* pile 5 days 30 groups local.* pile 6 use 1% groups control* remember 3
The LAST matching PILE is used, so specific rules should be at the END of the expire.conf file.
The piles 2,3 which have groups 'None' allow you to add piles in the future without renumbering existing piles!!
You MUST change local.* to match the names of your local groups if you want them stored for more than a few days.
| Setting | Value (example) | Description |
|---|---|---|
| pile | 1 | This must start each rule in expire.conf, each pile should have it's own unique number. Don't leave gaps in the numbers, and don't renumber the piles unless you don't care when the existing news is expired! |
| use | 10% | Specifies what % of the spool space should be used for this pile |
| days | 30 | Specifies the number of days to store items in this pile, this is only approximate as DNews only deletes entire 'buckets', also DNews will NEVER delete the last bucket in a pile so a small local group could end up being stored for 90 days or more simply because it never used more than one bucket. We don't recommend days based expire rules as they are not reliable due to the fact that entire bucket files are deleted and in the case of small groups this could mean days or weeks of items are deleted in a single step. |
| groups | *binaries*,*pictures* | Specifies the wild card list of groups to store in this pile |
| sort | If set this will tell DNews to sort the contents of these buckets to improve reader speed, you cannot use this setting if you send out live or DFeed news feeds. In ideal circumstances this setting can improve reader speed by a factor of 5, but on heavily cross posted groups it may have little effect. I would generally turn this on for specific popular 'non binary' piles, the smaller the pile the better it will work. The setting does not have a value like 'true' or 'false' it is considered to be on if the keyword is present. NOTE: prior to dnews5.3 adding this setting wil stop DNews from working as DNews shuts down if it finds invalid expire rules and this was only added at version 5.3 | |
| remeber | 3 | This isn't part of a pile rule, this goes on a line by itself in expire.conf and specifies how long to keep expired history message id's in the history file, this prevents duplicate messages bouncing around usenet a setting of 3 is usually sufficient. |
| unread |
(Added in DNews 5.4 beta8) If present, this makes this pile the pile for all groups that are not read, the groups setting for this pile should be the word 'none', this allows you to expire items in groups that no one reads fairly rapidly, e.g. pile 6 use 5% groups none unread Please note you must set cachedall false in dnews.conf for this setting to work!!! |
Using version 5.0, do these steps:
pile 1 use 5% groups * pile 2 use 1% groups none pile 3 use 1% groups none pile 4 use 10% groups comp.* pile 5 use 10% groups rec.* pile 6 use 10% groups alt.* pile 7 use 60% groups *binaries*,*pictures*,*warez* pile 8 days 90 groups local.*,important.local.groups.*,netwin.* pile 9 use 1% groups control* remember 3
NOTE: The piles 1,2 which have groups 'None' allow you to add piles in the future
without renumbering existing piles!!
NOTE: The 'sort' setting is only valid in dnews5.3 and later!!!
Here it's important to keep the non binary piles relatively small or the number of items within a group will get too large.
pile 1 use 5% days 30 groups * pile 2 use 1% groups none pile 3 use 1% groups none pile 4 use 5% days 30 groups comp.* pile 5 use 5% days 30 groups rec.* pile 6 use 5% days 30 groups alt.* pile 7 use 70% groups *binaries*,*pictures*,*warez* pile 8 days 30 groups local.* pile 9 use 1% groups control* remember 3
NOTE: The piles 1,2 which have groups 'None' allow you to add piles in the future
without renumbering existing piles!!
NOTE: The 'sort' setting is only valid in dnews5.3 and later!!!
