DMail Guide to scaling over multiple redundant systems
This page attempts to give a few brief notes ideas and suggestions on scaling DMail for
large systems, typically where more than a million registered users would exist.
Some general points to think about
- Firstly consider not using multiple systems, you may have over estimated your
requirements, or under estimated what a single machine can cope with, it may be wise to
start with a single machine and then scale up after you have seen your load profile and
know exactly which bits need to be 'scaled up'
- If you can avoid using or providing IMAPD services, then your system will be a lot more
efficient. (Simply put IMAP protocol is a resource hog and will require
significantly large systems, and often users don't like IMAP they prefer POP or Web access
so consider if you really need to offer this service)
- When calculating your required disk use or other resource utilization be very careful
multiplying what you think is an average user by 1 million. People generally get
this calculation wrong, and you don't want to spend half a million on disk arrays that
aren't going to be used.
- For best performance and reliability with NFS drives you should configure DMail to use
'maildir' drop files instead of standard drop files, this is easy to do UNLESS you have
some other application that needs to read drop files directly.
- Complexity decreases reliability, so avoid it when you can. As an extreme example a RAID
array is more complex than a normal disk and will often result in more down time even
though simple disks are prone to failure and RAID 5 arrays in theory are not.
An extreme design with 'n' servers for each part.
The extreme system can be designed like this, with two NFS systems, one for SMTP/POP
and one for WebMail storage. And 'n' servers handling each of the following tasks:
- 1...n SMTP servers - SMTP Incoming/Outgoing email
- 1...n POP servers - POP clients reading email
- 1...n Web servers - WebMail clients
A special router is used that invisibly connects users who ask for 'pop.xxx.com' to one
of the 'n' real pop servers, and similarly for the web servers.
However this is an extreme example, while you should configure your 'naming' scheme to
support the above layout you will probably want to start with a simpler design, typically
running 2 or 3 systems as follows
- Two servers each running POP and SMTP
- One..two server running WebMail interface (this could easily run on the pop/smtp servers
too if you want a simple '2' machine design)
- One NFS array with two areas one for mail and one for web mail folders.
The minimum/cheapest multi system design would be as follows
- Two servers, each running, SMTP, POP, WebMail
- One of the two servers would also include a raid 5 disk array.
- As stated earlier this systems simplicity in some ways increases it's reliability so
don't discount it purely because it's cheap.
- Additional systems can be easily added to spread load further.
Specific configuration suggestions for DMail / WebMail
- In DMAIL.CONF use #include to include 'most' of the configuration from the 'shared' nfs
array, in this way changing settings is much simpler. (Dwatch settings must be in
the actual dmail.conf before the include settings)
- Place the web cgi etc directories on the shared NFS drive again to simplify
- Place the authentication module and it's ini file in a shared directory.
- In dmail.conf define for each system the following settings:
- lock_id N (different for each system)
- drop_path /shared/drop
- bin_path /shared/bins
- hash_spool 3
- For WebMail also specify hashing mode 3.
- NetAuth (if used) must be run on a web server with direct access to the mail server
- Only start/run 'dlist' on ONE of the systems.
If you have any specific questions don't hesitate to contact
firstname.lastname@example.org or email@example.com