Disk Use And Files

An incoming message for a user is appended to that users unique 'drop file' by DSMTP. DSMTP will create the file if necessary. Then when the user connects to DPOP to read their mail, DPOP 'bursts' open that drop file and adds the messages that it finds to its own indexed 'bin' files. Then DPOP feeds the mail messages to the email client as it requests them.

DPOP can be made to 'drop' its bin files back to the drop file format for compatibility with other email servers, but this can affect performance.

See the list of DMail files at the bottom of this page.



Drop Files

Drop files are the link between SMTP and POP servers.

DMail creates and uses the Unix 'standard' drop files as used by Sendmail etc.

The drop file is a text file containing all the user's email messages one after another, including all of the message headers. Each message is separated in the file by a blank line followed by a line, starting 'From '. DMSTP inserts the From line at the start of each message (DPOP will also insert such a line, when it is asked to drop its bin files, back to the drop file format).

E.g. Example drop file

From pntest@161.29.2.46 Sat Dec 26 09:37:48 1998
Return-Path:
Received: from tosh ([161.29.2.46]) by tosh ; Sat, 26 Dec 1998 09:37:48 +1200
Comments: Authenticated sender is
From: pntest@161.29.2.46
To: Postmaster@161.29.2.46
Date: Sat, 26 Dec 1998 09:37:47 +0000
Subject: bob
Message-ID: <91461826801@tosh>

message body

the end.

From tam6@[161.29.2.46] Tue Jan 05 16:41:58 1999
Return-Path:
Received: from tosh ([161.29.2.46]) by tosh ; Tue, 05 Jan 1999 16:41:57 +1200
Comments: Authenticated sender is
From: "Tam Willacy"
To: tam@iba.com
Date: Tue, 5 Jan 1999 16:41:34 +0000
Subject: thanks
Message-ID: <91550771801@tosh>

Thanks for buying DMail :-) :-)

**************************************************
Tam Willacy,Product Development
NetWin Ltd.
support: support-dmail@netwinsite.com
email: tam@netwinsite.com
**************************************************

Note: DPOP does not leave messages in the drop file format, so once a user has checked their mail box, their drop file will be left empty. See the section below on bin files for more information.



The Drop File Filename

The name for each user's drop file (or mailbox) is based on the username that they log in with.

So if bob is a valid user then his drop file is called simply, 'bob' (note that there is no DOS file extension).

There is just one other option for the name of the drop file.

Users on virtual domains by default have a prefix added to their drop file filename so that users mail does not get mixed up even if they happen to be stored in the same directory. The prefix is one of the things set by the vdomain line in dmail.conf

The prefix is attached to the user's name with a special character which by default is an underscore, '_'. In general NO USER on your system is allowed this character in their username. This character can be set to something else with the setting, vdomain_separator

As an example bob from domainx.com might end up with a drop file name of,
domx_bob

See the section below on the drop_prefix setting for more on this.

Often with virtual domains there are fancy schemes where users have to log in with strange usernames - e.g. in suffix based virtual domains they have to log in with a suffix to show that the user belongs to a particular domain (bob@domainx.com or bob/domainx.com). NOTE in DPOP these never carry through to the drop file name.

There are only the two options for the name of the actual drop file.

So even though bob might log in with bob@domainx.com because he belongs to the domain, domainx.com, if DPOP recognises that domainx.com is a valid virtual domain suffix then his drop file will be called simply, 'domx_bob' or just 'bob'.



drop_prefix

The setting drop_prefix specifies if the prefix for a virtual domain is put on the drop file name. The prefix allows you to tell which drop file belongs to which domain. It also allows you to place mail from different domains in the same directory. The prefix stops users with the same name from having the same drop filename, e.g dom1_bob and dom2_bob.

By default it is true, as it is a good safety measure to have drop files named differently.



Bin Files

This is the format that DPOP stores user's mail in while it still resides on the POP server. This is generally only until the user reads (retrieves) their email, with their email client.

The user's bin files are created in a directory of the same base name and location as the user's drop file, but with the .bin extension added to its name.

E.g. a user Lucy whose drop file is,
c:\dmail\in\domain1\lucy
would have the bin file directory,
c:\dmail\in\domain1\lucy.bin

Within that directory DPOP creates message data files,
bin00000.dat
bin00001.dat
. . .
and a couple of index files, bin.idx msg.idx

. . .

The dropping of bin files back to drop files by DPOP can be initiated manually e.g.
tellpop drop username
or
tellpop drop_all

or it can be done automatically after a user disconnects from reading their mail, by setting the dmail.conf setting,
drop_users <wild card list>



Queue Files

All messages that arrive at DSMTP's door get put in the work_path directory as a queue file. Then other sections of DSMTP deliver them either locally or non-locally or to robots etc.

These files are in pairs - q_xxxx.itm being the message and q_xxxx.idx being an index of the envelope etc. You can open these up with a text editor. The .idx file is particularly useful as it contains information on the number of times a message has been tried etc.

Note: there is one pair of queue files per message in the queue but each message may have a number of final destinations. The idx file shows a 'fwd' line for each destination address (recipient line) with information on when it was last tried and if it was successful or not. Each queue file remains until all recipient lines for that message have been dealt with.

The way that DSMTP deals with these messages is complex. But basically it loads up 1000 of the recipient lines from the queue files into internal memory and then sets about delivering to each of those destinations. It does things like scanning the internal queue reordering it, moving messages on and off it, looking for destinations that are at the same domain to send down a channel it already has open, etc.

The tellsmtp status command lists amongst other things the number of queue files with its 'pending' counter. It also lists those messages which are in its internal queue (see max_queue). The numbers down the left hand column equate to the queue file number.

If DSMTP cannot deal with the current load then the number of queue files can back up. This is not a bad thing - it is the way SMTP servers work! When things quieten down a bit those messages will get delivered. What is a bad thing is if it backs up so much that messages never get out of the queue or over a period of days the queue continues to grow, i.e. it does not level out.

Typical ballpark queue sizes are,
100 for less than 1000 users
1000 for 1,000-500,000 users.
If your queue is often over these guidelines then you should check that that is ok for your system - what counts is throughput, i.e. are your messages getting through in an amount of time that you are happy with.

Note things like large mailing lists can greatly affect queue sizes.

Being hit by a spammer is a common reason for your queue to shoot through the roof! Using utilities like grep or find in the work_path directory you can often track down messages with the same sender or subject line. See the Spam section for details on how to stop the spam.

Talk to DMail Support if you are worried about a queue problem.

See also this FAQ, Can I delete queue files from the queue?.



Drop Paths

The path for your mail drop files is set for your main domain with the dmail.conf setting,
drop_path.

For each virtual domain the last value in the vdomain setting sets the drop path, e.g.
vdomain dom1 1.2.3.4 domainone.com \dmail\in\domone
sets the drop path for the virtual domain domainone.com to \dmail\in\domone.

If you have a number of synonyms for your main domain, i.e. a number of host_domain settings, then you can specify that certain domains put their mail in specific directories with alt_drop_path settings. NOTE that you cannot use alt_drop_path when using external authentication or for virtual domains.

The drop path can also be extended by using directory hashing> as detailed in the next section.



Hashing

In email systems with a large number of users you can easily end up with an unusably large mail directory. Worse is that performance can be severely affected by 'large' directories, particularly on UNIX based platforms.

To get around this DMail offers two forms of directory hashing. You turn these on with the hash_spool setting. Simply, hash_spool 1 adds one extra level of directories named with single letters of the alphabet, spreading mail drop files across 26 directories. Hash_spool option 2 adds two levels and so 26 squared directories are created.

All hashing directories are ADDED to the 'base' mail location set by the drop_path or vdomain settings.

Note: DSMTP creates these directories as mail arrives to be put in them. DPOP will also create them if it needs to for any reason.

Hashing is not done on virtual domain prefixes. See the reference section for more details on hash_spool



Quotas and Message Size Limits

Relevant settings: max_msgsize, drop_max, user_quota

max_msgsize limits the size of any mail message that passes through DSMTP, whether for delivery to a local user or being sent out.

drop_max limits the size of a user's drop file. If DSMTP is trying to deliver a message to a local user it checks to see that the user's drop file isn't going to be bigger than this limit after it has added the message it wants to add. If it is then it bounces the message and notifies the sender that the recipient was not able to receive the message.

Note that when running DPOP a user's drop file is often 0 bytes in size. This is because when the user connects to check their mail, DPOP clears the drop file. DPOP sorts the messages from the drop_file into its own BIN files.

So to limit the users' disk usage it is advisable to set up a quota system.

Turning on the Quota System

To do this, in dmail.conf you need to add the setting,
user_quota true
then for each user who you wish to have a quota limit, you need to go to the directory where their 'drop file' is, and create a file called,
username_inf
where username is the user's username as it appears for their drop file (note that it may a have virtual domain prefix on it).

In the username_inf text file you need to add a line like,
quota 40000
(to set a quota of 40 kbytes)

DPOP maintains a line,
used xxxx
in the same file, so that when DSMTP receives a message for that user it will check to see that the used amount plus the size of the new message will not exceed the quota limit.

In version 2.4i and above you can set a default quota for all users by simply giving a default quota in kbytes in the user_quota setting, instead of the word 'true', e.g.
user_quota 40
sets a default quota of 40 kybtes for all users that do not have a quota line in their username_inf file.

Remember that in the _inf file the quota setting is in BYTES, but in dmail.conf the user_quota setting is in KBYTES.

NB: The Web Gateway products, CWMail and DMailWeb etc., have their own quota systems which limit the amount of space a user uses on the web server for storage of read mail etc. The DMail quotas only affect space each user has in messages waiting to be read and messages 'left on the POP server'.



Path Settings

. . . drop_path, alt_drop_path, work_path, dlist_path, log_path, dwatch_path



Server Farming, NFS

Click here for general discussion and diagrams

Relevant setting: lock_id nnn

The servers in DMail are quite happy for you to put the users' drop and bin files on a shared network disk.

So you can also run multiple DSMTP and/or DPOP servers in order to share server load or for redundancy. You simply set the relevant path settings to a shared network drive, and then set a unique lock_id on each machine.

Basically the servers work in the following way:

Users connecting to read mail can only connect on one POP server at a time, so when a user logs in DPOP locks the user's drop file to read it, closes it and then talks to the user. If it finds the drop file already locked then it won't let the user connect.

Mail can come in for the same user on all servers in the farm at the same time, so if DSMTP finds that it cannot access the user's drop file, then it retries to access it 3 times at 1 second intervals.

On Windows platforms, you need to use UNC style names (e.g. \\serverA\shared_drive\dmail) to specify paths for the settings above between the server machines for the mail storage area. You may also need to use a UNC name for the authentication module, e.g. our NWAuth module requires this. You can map network drives instead of using UNC names but UNC names have the advantage that if the server reboots then the path specified with the UNC name will be accessible, whereas a mapped network drive is not accessible until someone logs on and the mapping is created. Whether you use UNC names or mapped network drives you MUST enable sharing on the folders that the servers are to share AND ensure that the correct permission for that shared resource is given - see the note below.

NB: the DMail servers are spawned by the DWatch service on NT, which will be running as the user 'system', so that is the user that the servers and the authentication module will also be running as. You cannot give access permission to a mapped network drive for the 'system' account. So you must change the NT DWatch service so that it runs as a specific user (in Control Panel, Services, Startup) and then give read/write permission for that user on the shared directories.

On UNIX based platforms: DMail and NFS drives

DMail has been designed to work with Network File System (NFS) drives. They can be used to allow DSMTP or DPOP servers to share directories such as the mail spool area ( drop_path) and drop_path

NFS can cause a lot of problems for mail servers because of its lack of file locking and delayed file updates. When you use the following setting DMail will use additional locking mechanisms to avoid problems with NFS

Set in dmail.conf lock_id to a unique string for each server that shares the mail spool on the NFS drive.

Please read the following list of notes if you are using or intending to use an NFS drive with DMail.

Notes on DMail and NFS:

  • Use at least version 2.8i: We have recently improved our NFS file locking code. Version 2.8i has improvements for DSMTP and improvements for DPOP will follow in in 2.8j
  • NWAuth: NWAuth can be used by multiple DSMTP and DPOP servers at once on an NFS drive. However it does no locking at present so should not be run in a situation where multiple copies will add a user at once when it is being run on an NFS drive. This will be fixed shortly. Contact DMail Support for further information.




Log Files

log_path . . .



Log Files: Logging of information and error messages

Follow this link for help on Deciphering Log Files.
'ded' files

Various informational messages are logged during the normal operation of the DSMTP, DPOP and DList servers. These events logged include such things as server startup, creation of work files, rejection of connection request etc. These messages appear in separate log files for each application; dpop.log for DPOP, dsmtp.log for DSMTP and dlist.log for DList

When the log file exceeds a preset maximum size it is renamed and a new file is created. For example dpop.log becomes dpop1.log This is referred to as log file rotation. For DPOP four old log files are kept - dpop4.log is the oldest and dpop.log is the current log file.

All DPOP versions after 2.9f check for log file deletion every minute. If the log file has been deleted, it creates a new log file, and places a message at the top of the new log file stating that the old file had been deleted.

The path for the log file is normally the same as the work directory but can also be set individually in the configuration file using the log_path setting. For Example:

log_path \data\logs

In addition, error, warning and debug messages can optionally be stored in the log file. The level of information logged is set by the log_level setting in the configuration file. The available settings are shown in the table below: 
 

error only logs error messages
info logs informational messages and error messages
debug logs debugging messages, informational messages and error messages

The default setting is info. The log_level setting may also be changed temporarily using the Tellpop or Tellsmtp log_level commands:

e.g. tellpop log_level info 

The logging level will revert when the system is restarted or when the configuration file is reloaded.

When one of the servers dies, DWatch should notice and restart the server (see the dwatch section for more details on when it does not restart the server). When it does this, by default it will RENAME the current log file for that server, e.g. dpop.log, to a 'ded' file, e.g. d_1dpop.ded. These ded files are named as, d_xserver.ded, where x is incremented on each death (so 1 is the first death dwatch caught), and 'server' is the name of the server, i.e. DPOP, DSMTP or DList.



Statistics Files: Gathering connection statistics

DPOP and DSMTP can provide various statistical information. In particular DPOP can optionally provide a per connection log of client usage statistics for use in charging clients etc.

These are the log/statistics files that are produced in addition to the standard log files.

  • Tellpop status command - see tellpop commands
  • Tellsmtp status command - see tellsmtp commands
  • daily dpopmmmdd.stats files for DPOP (dpopjan01.stats):

    A new statistics file is produced each day with a filename dpopmmmdd.stats which is intended for billing purposes.

    For example for the first of January the file name will be dpopjan01.stats

    The statistics file is a printable text file containing one line per connection with the six fields separated by spaces:

    • start time of connection in seconds
    • end time of connection in seconds
    • username
    • from host name
    • bytes transferred
    • number of messages transferred

    To disable DPOP's per connection log modify the configuration file to include a blank path for the stats_path:

    Example:

    stats_path my/stats/path
    becomes
    stats_path
    To disable logging of connection statistics for charging

  • a tellpop command to collate DPOP stats for a monthly period:

    The command: tellpop stats can be used to summarize the information from a collection of .stats files. It will produce a one line summary for each user giving number of connections, number of messages, average time between connections etc. See Tellpop commands section for more details.

  • daily .sta files for DSMTP (dm0101.sta):

    This file contains the output of a tellsmtp status command and is created at the end of each day.

  • daily dmddmm.log files for DSMTP (dm0101.log)

    This log file contains a simple log of incoming/outgoing messages and whether they were delivered. It builds up throughout the day and gets rotated at the end of the day

  • A stats dump file,dmddmm.dmp (dm2712.dmp, 2.7h and above):

    This file is appended to throughout the day if you add the setting,
    dump_stats true
    to the dmail.conf log file.

    This file is intended to help support to work out what is happening on your server over the day, e.g. to pick up on DNS connnection problems and periods when queue files get backed up.


File Permissions

. . . mostly as MAIL user and group

drop files will be set to 600 on Unix platforms if not already

DPOP: check_gid =

check_disable true = be happy with file permissions and don't change them.

DSMTP: chmod_drop false = don't change drop file permissions, but still create on Unix platforms with 600 file permissions. So doesn't affect things like a forwarding alias to /dev/null/ (note forwarding to /dev/null can also be achieved by making an alias to user@nul, upon which DSMTP will make it the message disappear into the void :-)



List of DMail files

This list is not complete but files that we are commonly asked about will be added here as links to the section describing them.

ded files (typically in \dmail\dwatch or /usr/local/dmail/dwatch)
d_1dpop.ded d_2dpop.ded d_5dlist.ded
when dwatch has to restart a server it archives the current log file, e.g. dpop.log, by renaming it to a 'ded' file, e.g. d_5dpop.ded is the log file on the fifth death of DPOP.
exit_files (typically in \dmail\dwatch or /usr/local/dmail/dwatch)
dpop.exit dlist.exit dsmtp.exit
any file named this in the dwatch directory will cause the named server to shut itself down.
join.tpl - DList (typically in \dmail\dlist\list_name or /usr/local/dmail/dlist/list_name)
join.tpl - can go in DList directory(applies to all lists) or in individual
This file has a Welcome message for new subscribers to either a particular list or all lists depending on its location. It is a template of the message - DList fills in list name etc for you.
q_x.idx and q_x.itm - DSMTP (typically in \dmail\work or /usr/local/dmail/work)
q_xxx.itm - queued message xxx's data, q_xxx.idx - DSMTP's index for the same message
This file has a Welcome message for