New Built-in Filtering System DMail 2.9

Design Goals:

How to configure rules:

New tellsmtp commands:

tellsmtp mfilter_test d:\test.msg d:\test.rul

Give FULL PATHS for both of the files in this command.

(Last parameter is only used in 2.9y and later, prior to that the existing mfilter.rul file would be used with the test message)


Syntax Of mfilter.rul File

There are 6 valid statements in a rule file:

Assignment
Action
if (Conditional_Expression) [and (Conditional_Expression)...] Action
else
end if
call built_in_function()

Assignment

$variable_name = "quoted string" [+ "quoted string" [+ $variable ...]]
$variable_name = function()

Action

accept "reason" | bounce "reason" | drop "reason" | forward "user@domain" | then | setflag("flagname") | clearflag("flagname")

Conditional Expression (if, else, end if)

Any pre-defined function, e.g. isbinary()
isin("subject","free pictures")
Numeric comparisons, e.g. lines()>100
Simple NOT operator, e.g. if (!isbinary()) reject "Only binaries allowed here mate!"
Calculations are NOT permitted, e.g. lines()+10 would fail

Recipients block for processing individual recipients

A single mail message may have many recipients, and in many cases the actions of your spam filter should vary depending on the recipients (you might for example want all messages to your account to get through even if the same messsage would be blocked if sent to any other user.

The recipient block (recipients...end recipients) is processed once for each recipient of the message.

Inside the 'recipients' block there is a dummy variable defined 'recipient' which is the specific recipient in question.

All the action's (accept, bounce, drop) refer to the recipient only, not to the entire message, so when one of those actions that normally terminates message processing is encountered (accept, bounce, drop etc) instead the action is applied only to that recipient and the recipient block is restarted with the next recipient defined.

recipients
if (isin("recipient","manager@this.domain")) accept "Always accept for me so spammers can talk to me"
if (isin("recipient","sales@your.domain")) then
if (isin("subject","order")) then
# Make a Duplicate of sale order
call forward_cc("sales_copy@your.domain")
end if
end if
end recipients


Miscellaneous

Line Continuation

Lines can be continued by ending the line in a '\' character

Quoting Strings

All strings and header names should be within double quotes, sometimes you may get away without doing this, but we don't guarantee this will work in future. e.g. use: exists("Supersedes") not exists(Supersedes); quotes can be escaped in the usual way, e.g. "This \"Word\" has quotes around it"

Assignments

Assignments are processed at compile time, variables DO NOT exist at run time. Do not think of this as a programming language, but rather as a list of rules that are processed with each incoming message. Real run-time variables only exist in the form of the ifflag("xxx") function and the setflag("xxx") action.

For example, the following is NOT VALID, as the assignment is processed before the rules are run. The rejection would always read "big message"

$fred = "small message"
if (lines()>100) then
   $fred = "big message" (this will not work as expected)
end if
reject $fred

Actions & Commands

Actions

accept"reason" (Terminates processing)
bounce "reason" (Terminates processing)
reject "reason" (same as bounce)
forward "reason" (Terminates processing)
print "reason" (Prints debugging line to log file)
setflag("flagname") "reason"
clearflag("flagname") "reason"

Functions that have actions but must bust be preceeded by the 'call' action as they are really functions and must be on a line of their own (not on the end of an if statement)

call forward_cc("new@email.address)
call replace("header_name","wildcard_match_pattern","replacement_pattern")
call report("manger@email.address","subject of message")

Builtin Functions

allmod()
exists("header")
head_len("header")
isbase64()
isbinary()
isencodedhtml()
isencodedtext()
isencodedurl()
isflag("flag-name")
ishtml()
isimage()
isin("header","string-not-case-sensitive")
lines()
match("header","wildcard")
matchall("header","wildcardlist")
matchone("header","wildcardlist")
rexp("header","regular-expression")
size()
spamdetect(n,"reason")


Function Descriptions

allmod("header")

This returns true if all the newsgroups in the specified header are moderated.

exists("header")

This is true if the header exists in the message and is non zero in length, e.g. if (exists("supersedes")) then reject "We don't like supersedes headers"

head_len("header")

Returns the length of the named header, e.g. if (head_len("date")>60) bounce "Naughty message"

isbase64()

This is true if the message appears to contain base64 binary encoded data.

isbinary()

This is true if the message has binary data, either base64 encoding or uuencoded data.

isencodedhtml()

This is true if the message appears to contain mime or uuencoded HTML instead of plain text data.

isencodedtext()

This is true if the message appears to contain mime or uuencoded text data.  This will always be true if isencodedhtml() returns true.

isencodedurl()

This is true if the message appears to contain a uuencoded URL reference.

isflag("flag-name")

Used to check whether a flag variable has been defined as true. This can be done with the setflag("flag-name") action, e.g.

if (size()>100000) setflag("bigitem")
if (isimage()) setflag("bigitem")
if (isflag("bigitem")) reject "It was a big item or had a picture in it"

ishtml()

This is true if the message appears to contain HTML instead of plain text data.

isimage()

This is true if the message appears to contain a picture (either mime or uuencoded)

isin("header","string-not-case-sensitive")

This is a simple 'content' searching function, if the named header contains the string (a non case sensitive match is used) e.g.

if (isin("Subject","Free")) reject "Probably a spammer selling something"

This would reject a message containing a subject of "Get your Free pictures here", it would also reject a message containing a subject of "Is there any real freedom in the world?" so it's probably not a good rule :-)

lines()

This returns the number of lines in the message.

match("header","wildcard")

This function applies a simple wild card matching algorithym, as is typically used to match file names, e.g. match("From","*@netwin.co.nz*") would match against a message from that domain.

matchall("header","wildcardlist")

Used for matching a single wild card, against a header which contains a list of values, like Newsgroups:, Path:, etc..., The match is TRUE only if all entries in the list match, e.g. if (matchall("Newsgroups","news.filters.*")) accept "It is only in the filters list so we will accept it"

matchone("header","wildcardlist")

Identical to the above function, but returns 'TRUE' if any match occurs.

rexp("header","regular-expression")

This function searches the named header for a regular expression, the matching is not case sensitive, use rexp_case() for a case sensitive version.

size()

Returns the size in bytes of the current message, can be used with > and < operators.

spamdetect(n,"reason")

This function can be used to mark a message as possible spam, the 'n' is a number between 1-100 and each time this function is called for a message the total is increased, then finally a header is added to the message;

X-SpamDetect: low: reason
X-SpamDetect: medium: reason
X-SpamDetect: high: reason

low 1-40, med=41-80, high = 81-100

The idea is that users can then set their mail clients to filter messages based on this pseudo header.


Actions

accept "reason"

Accepts the current article reporting the "reason" specified in the log files.

clearflag("flag-name")

Used to set the specified flag variable to the false state.

forward "remote@address.com"

Forwards the message to the specified address, and terminates processing.

call forward_cc("new@email.address")

Sends the current message to this new email address in addition to any existing desitination users.

reject "reason" (or bounce "reason")

Rejects the current article reporting the "reason" specified in the log files and to the user

call replace("header_name","wildcard_match_pattern","replacement_pattern")

If the named header matches the 'wildcard_match_pattern' then the replacement pattern is applied, e.g.

replace("from","*@*.domain.name","BOB_$1@$2.other.name")

Subject: "joe@this.domain.name"

Would be translated to:

Subject: "BOB_joe@this.other.name"

call report("manger@email.address","subject of message")

Sends an email, including the top part of the offending message, to the specified person, with the specified subject. This is intended when you want to be alerted to something, but don't want to simply forward the message itself, which may be 'confusing' as it would look like the message had been sent to the manager directly.

setflag("flag-name")

Used to set the specified flag variable to the true state.


Regular Expression Syntax - In Brief

\s = white space
\S = not white space
\d = digit
\D = not digit
\b = word boundary
\B = not word boundary
\x00 = Hex character

. (period) represents any one character.
[] (brackets) contain a set of characters from which a match can be made. It corresponds to one character in the search string.
\ (backslash) is an escape character which means that the next character will not have a special meaning.
* (asterisk) is a multiplier. It will match zero or more ofthe previous character. (Note that it is not a wildcard character as in file names.)
? (question mark) is a multiplier. It will match zero or one of the previous character. (Note that it is not a wildcard character as in file names.)
+ (plus) is a multiplier. It will match one or more of the previous character.
{} (squiggly brackets) contain a number which specifies an exact number of the previous character. Or range {2,3}
[^] (brackets containing caret and other characters) means any characters except the character(s) after the caret symbol
in the brackets.
^ (caret) is the start of the line.
$ (dollar) is the end of the line.
(note: these two are not implemented, use \b instead   "\<"   "\>"  (beginning and end of word)

[:alpha:] represents any alphabetic letter.
[:digit:] represents any single-digit number.
[:blank:] represents a space or tab.

Lookahead operator
Free(?!dom|bsd) matches freesex but not freedom or freebsd

OR operator
| (pipe) is OR. It requires that the joined expressions have parentheses around them.

Examples:

e.a matches eta, eda, e1a, but not Eta
[eE].a matches eta and Eta
E.*a matches Eudora, Etcetera, Ea
ho+p matches hop, hoop, hoooop, but not hp
etc\. matches etc. but not etc


Example rule file:

$sex = "fuck|xxx|sex"
$free = "free(?!dom|bsd|nix|serve)"
$pics = "pi[cx]"
$free_pictures = $free + $pics
$bad_guys = + "|freepictures|jus.?.?\.doi.?.?\.to|great\.site|webbinaries" \
          + "|yad.?.?.?\.ion.?.?\.org|freehidden|joy.?.?\.to.?.?\.al|from.?behind" \
          + "|love(youhon|ergirl|chatting|stofuck)|forever\.yours|\@ju.?.?\.sex|town.\girl|beachbums" \i
# Do some processing which is specific to individual recipients
recipients
	if (isin("recipient","manager@this.domain")) accept "Always accept for me so spammers can talk to me"
	if (isin("recipient","sales@your.domain")) then
		if (isin("subject","order")) then
			# Make a Duplicate of sale order
			call forward_cc("sales_copy@your.domain")
		end if
	end if
end recipients
# Check for some known spammers and naughty subjects
if (rexp(subject,$free_pictures)) bounce "No emails about free pictures"
if (rexp(from,$bad_guys)) bounce "No emails from black listed people thanks"

# Strip local node names from from addresses:
call replace("From","*@*.parts.co.nz","$1@parts.co.nz")
accept "Great, we liked the message"

Example using spamdetect feature (Version 3.0 or later)

# This is a useable example, but please read through it and 
# make your own decisions on what to classify as spam before using.
$sex = "fuck|xxx|sex"
$free = "free(?!dom|bsd|nix|serve)"
$pics = "pi[cx]"
$free_pictures = $free + $pics
$celebsex = "fuck|xxx|sex|celeb"

$bad_guys = "\bxxx|xxx\b|freepictures|jus.?.?\.doi.?.?\.to|great\.site|webbinaries" \
          + "|yad.?.?.?\.ion.?.?\.org|freehidden|joy.?.?\.to.?.?\.al|from.?behind" \
          + "|love(youhon|ergirl|chatting|stofuck)|forever\.yours|\@ju.?.?\.sex|town.\girl|beachbums" 


#Reject bulk-mailer signitures		
if(exists("Message-ID")) then
	if (!isin("Message-ID","@")) reject "Suspicious Message-id"
end if
if(isin("To","undisclosed")) then
		call spamdetect(20,"Undisclosed mail list")
end if
if(isin("Received","-0600 (EST)")) reject "Suspicious time zone"


# Do some processing which is specific to individual recipients
recipients
        if (isin("recipient","manager@this.domain")) accept "Always accept for me so spammers can talk to me"
        if (isin("recipient","sales@your.domain")) then
                if (isin("subject","order")) then
                        # Make a Duplicate of sale order
                        call forward_cc("sales_copy@your.domain")
                end if
        end if
end recipients

# Check for some known spammers and naughty subjects
if (rexp(from,$bad_guys)) bounce "No emails from black listed people thanks"

# Strip local node names from from addresses:
call replace("From","*@*.parts.co.nz","$1@parts.co.nz")


#Spamdetect: searching mail subject for impact words typically used by spammers.
$bad_nouns = "\bclit|\bdicks?\b|\bslut|\bcunt|penis" 

$suspect_nouns = "xxx|adult|teen|nympho|\bnude|\bporn|hardcore|panties|wanger|virgin|schoolgirl|\bgirl" \
	     + "|amateur|\btits?\b|\bgays?\b|\basse?s?\b|pussy|asian|\blesb|crack|celeb|boobs|nipple" \
	     + "|\bdorm|\bpics\b|\bbabe"

$bad_verbs = "f[\*u][\*c]k|sex|bondage|\bwank|masturbat|incest"

$suspect_verbs = "fetish|spank|stroke|whack|strip|cum|blow|pee"

$adjec_adverb = "hard|\bhot\b|\bbig|huge|nasty|xtreme|\blive\b|horny|freaky|naked|tight"


#Rating sexually related spam.
if(rexp("subject",$bad_nouns)) then
	call spamdetect(40,"Sexual ref")
	setflag("SexRef")
end if
if(rexp("subject",$bad_verbs)) then
	call spamdetect(40,"Sexual ref")
	setflag("SexRef")
end if
if(rexp("subject",$suspect_nouns)) then
	call spamdetect(15,"Suspect ref")
	setflag("SexRef")
end if
if(rexp("subject", $suspect_verbs)) then
	call spamdetect(15,"Suspect ref")
end if
if(rexp("subject",$adjec_adverb)) then
	if (isflag("SexRef")) then
		call spamdetect(30,"")
	end if
end if

#Rating unsolicited junk mail.
$selling_words = "pheromone|morgage|sample|viagra|aphrodis|\bdeal|visa|mastercard|$\d+" \
               + "|\d+%|\bcost|cash|money|credit|freebie|income|dollar|minute"

$selling_modifiers = "cheap|limited|nationwide|\bfree\b|discount|special|unlimited|exclusive|credit" \
                   + "|unsecured|save|guarantee" 

if(rexp("subject",$selling_modifiers)) then
	if(rexp("subject", $selling_words)) then
		call spamdetect(75,"Junk Mail")
	else
		call spamdetect(40,"Possible Junk Mail")
	end if
else
	if(rexp("subject", $selling_words)) then
		call spamdetect(25,"Possible Junk Mail")
	end if
end if

#Suspect sources ie usernnn@domain
if(rexp("from","\d+@")) then
		call spamdetect(20,"Suspect source")
end if

#Check for multiple spaces
if(rexp("subject","[^ ] {2,7}[^ ]")) then
		call spamdetect(20,"Multiple spaces")
end if
if(isin("subject","        ")) then
		call spamdetect(40,"Excess spaces")
end if

#Check for all caps
if(head_len("subject")>6) then
	if(!rexp_case("subject","[abcdefghijklmnopqrstuvwxyz]")) then
		if(rexp_case("subject","[ABCDEFGHIJKLMNOPQRSTUVWXYZ]{3}")) then
			call spamdetect(40,"All capitals")
		end if
	end if 
end if

#Check for hotmail.com
if(isin("to","@hotmail.com")) then
		call spamdetect(40,"Addressed to hotmail.com")
end if

if(isin("from","riskymail")) then
		call spamdetect(40,"Porn spam")
end if

if(isin("from","free")) then
	if(isin("from",$celebsex)) then
		call spamdetect(40,"Dodgy looking sender")
	end if
end if