DMail Guide to scaling over multiple redundant systems

This page attempts to give a few brief notes ideas and suggestions on scaling DMail for large systems, typically where more than a million registered users would exist.

Some general points to think about

• Firstly consider not using multiple systems, you may have over estimated your requirements, or under estimated what a single machine can cope with, it may be wise to start with a single machine and then scale up after you have seen your load profile and know exactly which bits need to be 'scaled up'
• If you can avoid using or providing IMAPD services, then your system will be a lot more efficient.  (Simply put IMAP protocol is a resource hog and will require significantly large systems, and often users don't like IMAP they prefer POP or Web access so consider if you really need to offer this service)
• When calculating your required disk use or other resource utilization be very careful multiplying what you think is an average user by 1 million.  People generally get this calculation wrong, and you don't want to spend half a million on disk arrays that aren't going to be used.
• For best performance and reliability with NFS drives you should configure DMail to use 'maildir' drop files instead of standard drop files, this is easy to do UNLESS you have some other application that needs to read drop files directly.
• Complexity decreases reliability, so avoid it when you can. As an extreme example a RAID array is more complex than a normal disk and will often result in more down time even though simple disks are prone to failure and RAID 5 arrays in theory are not.

An extreme design with 'n' servers for each part.

The extreme system can be designed like this, with two NFS systems, one for SMTP/POP and one for WebMail storage. And 'n' servers handling each of the following tasks:

• 1...n SMTP servers - SMTP Incoming/Outgoing email
• 1...n POP servers - POP clients reading email
• 1...n Web servers - WebMail clients

A special router is used that invisibly connects users who ask for 'pop.xxx.com' to one of the 'n' real pop servers, and similarly for the web servers.

However this is an extreme example, while you should configure your 'naming' scheme to support the above layout you will probably want to start with a simpler design, typically running 2 or 3 systems as follows

• Two servers each running POP and SMTP
• One..two server running WebMail interface (this could easily run on the pop/smtp servers too if you want a simple '2' machine design)
• One NFS array with two areas one for mail and one for web mail folders.

The minimum/cheapest multi system design would be as follows

• Two servers, each running, SMTP, POP, WebMail
• One of the two servers would also include a raid 5 disk array.
• As stated earlier this systems simplicity in some ways increases it's reliability so don't discount it purely because it's cheap.
• Additional systems can be easily added to spread load further.

Specific configuration suggestions for DMail / WebMail

• In DMAIL.CONF use #include to include 'most' of the configuration from the 'shared' nfs array, in this way changing settings is much simpler.  (Dwatch settings must be in the actual dmail.conf before the include settings)
• Place the web cgi etc directories on the shared NFS drive again to simplify configuration.
• Place the authentication module and it's ini file in a shared directory.
• In dmail.conf define for each system the following settings:
  • lock_id N (different for each system)
  • drop_path /shared/drop
  • bin_path /shared/bins
  • hash_spool 3
• For WebMail also specify hashing mode 3.
• NetAuth (if used) must be run on a web server with direct access to the mail server systems.
• Only start/run 'dlist' on ONE of the systems.

If you have any specific questions don't hesitate to contact dmail-support@netwinsite.com or webmail-support@netwinsite.com