DMail Frequently Asked Questions: Converting To DMail (from other email systems)

Questions:

Transferring Users:

  1. How can I transfer mail accounts (users) from my current email server?
  2. How can I convert from Unix_user (/etc/passwd, shadow passwords etc) to NWAuth.
  3. We have our own special username/password routines. Can these be used with DPOP?
  4. Columns, A utility developed for extracting data from a file to command lines in a batch file for adding users to NWAuth

Transferring mail data - drop files

  1. What mail box (drop file) format does DMail use?
  2. Where drop_files (mail boxes) get put (hashing) and what they are called.
  3. A list of what we know about converting from other email servers.
  4. DPOP Compatibility Settings

General

  1. Change Over Time Suggestions
  2. I run an automated mailer, e.g. CGI or command line mailing...
  3. We would like to try DPOP but are paranoid about upsetting umpty thousand users. How can we ease into it?
  4. I have half a dozen users who leave mail on the server and need to read email direct from Unix drop files.
  5. Is the source for DPOP, DSMTP, DList available so that we can tailor it to our needs?
  6. A typical response to someone converting from Sendmail
  7. (MISC. FAQ) Does DMail support CDONTS?


Answers: Transferring Users

  1. How can I transfer mail accounts (users) from my current email server?

    The best way to answer this is to give you some details on options for DMail and hopefully if you are able to tell DMail support about your current system then they can make relevant suggestions.

    It is worth noting first off that if the users are simply members of the operating system user database then you do not need to do anything with them - simply install DMail and it will find the users by default.

    DMail has two basic authentication options,

    a) use the operating system password list
    b) use an external authentication module

    There is one configuration file, dmail.conf, setting that sets this,
    authent_method

    For a this will either be,
    authent_method nt_user
    or
    authent_method unix_user
    depending on whether you are on a windows or Unix based platform.

    For b you set,
    authent_method external
    and
    authent_process path_to_program
    where path_to_program is the authentication program to run.

    Your options are:

    1. We provide an example authentication module, called NWAuth, which is fully functional and is very efficient with large numbers of users.
    2. You can also write your own to link to any type of user database (or modify one of ours).
    3. Our example module for linking into an LDAP server, LDAPAuth.
    4. Our example module for linking into DNews's users.dat file, DNAuth.
    5. A customer has provided us with the source to talk to a mySQL server, which DMail support can pass on to you to use or modify.
    6. There is a link on the following page to an ODBC authentication module provided by another customer,
      https://netwinsite.com/dmail/utils.htm

    So one of the above might be an option, but it does depend on how the user's details are stored. Our NWAuth module can also be run from the command line, e.g.
    set user password info="details"
    so it may be possible to write a script to run that for all of the users out of your current user database or from a user list.

    See the following section in the manual for more details: External Authentication Modules List

  2. How can I convert from Unix_user (/etc/passwd, shadow passwords etc) to NWAuth.

    NWAuth uses the system crypt function on UNIX based platforms so you can probably simply copy the first two columns from your /etc/passwd file into nwauth.txt. NWAuth should get password comparisons correct. You might want to use the utility cols to pull out the first two columns in /etc/passwd.

    If you are using shadow passwords or yellow pages etc (note that you should be using libc6 versions of DMail) it may not be so simple.

    Please let us know if you find anything out in this area so that we can add to this FAQ.

  3. We have our own special username/password routines. Can these be used with DPOP/DSMTP?

    Yes, DSMTP and DPOP can be configured to use an external authentication process for checking username/passwords.

  4. Columns, A utility developed for extracting data from a file to command lines in a batch file for adding users to NWAuth

    You can download Columns from the DMail Utilities Download page.

    This utility scans a file for data and writes it out to file in a specified format. An example use is to create a batch file to add a list of users to NWAuth, E.g. turn this 2 line file,

    bob;pass1;blah blah
    james:password;blah blah

    into,

    NWAuth -set bob pass1
    NWAuth -set james password

    Where, the 'set' command line option for external authentications adds the specified user with the given password. So running the last file as a batch file would add the 2 users to the NWAuth user database.


  5. Answers: Transferring Data - Drop Files

  6. What mail box (drop file) format does DMail use?

    DMail uses the Sendmail 'standard' format for user drop files (mail boxes). This is where all messages for each user are appended onto the end of one file. The messages are separated by a blank line and a line starting with 'From ...' (the syntax of the From line can need to be specific).

    There is an added complication that when a user connects to DPOP to collect their mail, DPOP removes the mail from that file and converts it to its own bin file format. See various sections in the Disk Administration section for more details.

    NB: if your users run some sort of command line program that directly accesses the drop_file then you need to use, the tellpop drop command to make DPOP convert its bin files back to a drop file for that user. You can set this to be done automatically on pop logout - see drop_users.

  7. Where drop_files (mail boxes) get put (hashing) and what they are called.

    Drop files are created in the directory specified by the drop_path setting in the dmail.conf config file for the main domain or any aliases of it (as set with the host_domain settings). If the user is on a virtual domain the vdomain setting's last parameter sets the drop file path for that domain.

    In addition to this the setting hash_spool sets whether you want any directory hashing done (extra levels of directories added in so that there are not too many files in any one directory). A hash_spool setting of 0 means that no hashing should occur. If you have hash_spool set to something else then you need to be aware of this when you copy drop files across from other systems.

    NB: we have a utility FixHash which allows you to move drop files between different hashing methods.

    The DMail servers generally name drop files with the same name as the username part of the email address. E.g. if bob has the email address,
    bob@netwinsite.com
    then his drop file name would be simply,
    bob

    If bob was on a virtual domain then by default he would have a prefix added to his drop file name (taken from the first parameter of the vdomain setting),e.g.
    domainx_bob
    This is a safety precaution to ensure that two users with the same username on different domains never get their mail merged together!

    NB: FixHash can add prefixes onto the start of drop files for you if you wish, or you can set, drop_prefix false in order to turn drop file prefixes off.

    For more details on all of this see the Disk Administration section of this manual as well as other FAQs on this page.

  8. A list of what we know about converting from other email servers.

    Below are details provided by customers who have converted from other systems to DMail. Please let us know how you get on if your current system is not on this list, or you can add to any of the entries. NB: Don't be surprised if your current system is not on this list, it is a very new list. We have had customers change from most common email servers to DMail, so you are probably not the first :-)

    NB: One 'gotcha' is differences in drop file hashing - see Where drop_files (mail boxes) get put. on this page.

     

    • Sendmail/CuciPOP:
      The drop file format is the same so simply copying over drop files works. You can use the /etc/passwd file (including shadow password files) as is, by setting authent_method to Unix_user, or you can use our Columns utility to move your user database to any of the external authentication modules, e.g. NWAuth, LDAP, SQL etc.
    • Elm, Pine clients:
      Uses sendmail drop file format so should be no problem. Need to convert DPOP's bin files back to drop file format before using these, see drop_users.
  9. DPOP Compatibility Settings

    See,
    msg_separator
    drop_old
    drop_kill


  10. Answers: General

  11. Change Over Time Suggestions

    Here are some suggestions for handling the point at which you change over to using DMail ...

    • use the bind_in setting

      You can make the DSMTP and DPOP servers use the bind_in setting to bind to one specific ip address. This allows you to run them alongside your existing mail server (given you bind that server to another specific ip address).

      This can be a good way to do final testing of your DMail setup, but do be aware of things like the drop_file directories (set with the drop_path setting) conflicting with the old server.

  12. I run an automated mailer, e.g. CGI or command line mailing...

    If your current email system has provision for sending emails generated by robots or CGIs or some sort of command line mailer, then DMail will in most cases be able to work with or replace that part of your system.

    For UNIX users, when you install DMail it replaces Sendmail with what we term the 'Sendmail Stub' which replaces the command line emailing part of Sendmail, so that you can still email from the command line, e.g.
    mail bob@domainx.com

    Many people run CGIs or a command line mailer which sends emails automatically via sendmail's command line mail command - this sendmail stub allows these to keep working when you change to DMail. (A common problem is that the replacement of sendmail does not happen on install. Our sendmail stub is only about 40k in size and dmsetup tries to put it in /usr/sbin. So check the size of that file and that any links for 'mail' or whatever point to that stub. Our stub creates a file, dsmtp_path/sendmail.log which logs if it has been called and with what arguments - so check that file to see if your CGI or mailer is actually running the stub).

    On Windows platforms, most mailers of CGIs if well written will talk to the SMTP part of the email system on port 25, using the SMTP protocol. Given you have a mailer such as this there should not be any problem in continuing to use it with DMail.

    Some email systems (in particular on Windows platforms) have built in support for emailing from web pages or talking to CGIs and command line mailers. This in our opinion is 'bad form' and getting away from such proprietry behaviour is an advantage of moving to DMail :-) To help in moving to DMail for such systems we are creating a CGIMail CGI to run on your web server which when fed a web page will take fields off a form on that page and create an email out of them and feed that over TCPIP to any SMTP server.

    In addition to the above we are adding the ability to put a message file in a directory, which DSMTP then picks up and sends as a way of interfacing a mailer with DMail.it, using information in the file, writing CGIMail to email form information from a web page.

    Contact DMail Support for more information on any of the above.

  13. We would like to try DPOP but are paranoid about upsetting umpty thousand users. How can we ease into it?

    Email is a vital service so even if the current popper you are using is slow it is still a scary step to move to another one. You can't afford to upset users. So how do you ease into it. There are a number of strategies which can be helpful here.

    • If you have the luxury of a spare machine obviously installing DPOP on that first will help. It at least allows you to check out the various options you might want to use and get used to how they work. The DMSetup wizard will help you to remove it from the test machine after your testing is complete. The de install option tries to err on the conservative side. It tells you where the files are you might want to delete. It will only remove something that is definitely part of DPOP and not any other popper.
    • If you have not got a spare machine or you have tried that and are now more comfortable but still cautious: The next easy step is to install DPOP on the main server BUT get it running on a different port. This way you can leave your original popper running. For example you might set DPOP up on port 1100 instead of 110. To do this follow the normal installation procedure but say no to the question: "Shall I comment out current POP3 entries in inetd.conf". Then edit dmail.conf file and change pop_port line as shown below:
      pop_port 110
      pop_port 1100
      You can then get individual users to try switching to DPOP use by changing the setting in their email reading software to read on another port. This is straightforward in Pegasus mail, more difficult on some other email clients. For Eudora on Windows 95 just edit the Services file in the windows directory to change POP3 port. You can even allow someone to connect both ways although if they are going to do this AND leave unread or undeleted mail on the server you must put a line in dmail.conf to tell DPOP to change there bin files back into a drop file at the end of each session. This should only be done if they NEED to read there mail from Unix command line or some other non DPOP connection. It will slow processing down. If Bob,Bill and Bert are Unix gurus who read there mail from the Unix command line and using a POP3 client you might add one of the following lines to dmail.conf:
      drop_users B*
      drop_users Bob,Bill,Bert
      Once you have run DPOP in this mode for a while you can switch back to the real POP3 port by changing the pop_port line in dmail.conf and then issuing the Tellpop reload command.
    • Alternatively you can take the plunge and install DPOP directly on your main server in some off peak time. Test it with a few test accounts and if there are any problems that look difficult revert to the previous popper. To do that all you need to do is put the lines in inetd.conf back how they were and get inet to reload. The DMSetup wizard can do this for you. If the accounts you have tested have undeleted or unread mail left on the server these must be converted back to drop files. This must be done before stopping DPOP by using either:
      tellpop drop_all
      to do all accounts that have used DPOP or
      tellpop drop Bert
      tellpop drop Bill
      etc. to deal with user accounts one at a time.
  14. Drop users:

    You have a few users who check their mail using a normal POP client but leave the mail on the server and want to be able to access the drop files directly, with pine for example. But DPOP converts the drop files to its own format for more efficient manipulation, so once the mail has been checked there is nothing left in the drop files, so the users cant see their mail. This is easily remedied by adding a line to your dmail.conf configuration file. It should look like this:

    drop_users ralph,bill,*smith

    This would force DPOP to leave all the email messages for ralph, bill and anyone with a usercode finishing with the word smith in drop files. Be careful not to put spaces in the list and avoid making it too general as there is a performance hit in keeping messages in drop files, that's why DPOP avoids it in the first place. This setting is only needed for users who check their mail with a POP3 connection AND leave it on the server AND want to read it with software that directly reads the drop file.

  15. Is the source available so that we can tailor it to our needs?

    No, but this should not be necessary as most aspects of DSMTP DList and DPOP can be easily configured. They can also use an external password checking routine, an external routine to indicate where drop files are and how the path is hashed. DPOP can also generate statistics which can be used by an external routine for generating charging information. If there is some other aspect which you need to be able to tailor please let us know.

  16. A typical response to someone converting from Sendmail

    >Currently we store and maintain all our users in Mysql.
    > We generate the
    > passwd, shadow, access, aliase and virtusertable from the database for
    > sendmail. Here is the key elements:
    >
    > * For each user,
    >
    > - user name is unique.
    > - password is Linux style, not Mysql style.
    > - all users are maintained by existing programs. Thus the user
    > maintenance feature of DMail is not needed.
    > - with these setting, our users have these simple set up in their email
    > programs:
    >
    > # outgoing email server: mail.bob.com
    > # incoming email server: mail.bob.com
    > # incoming email account: ...their.login.name...
    > # incoming email password: ...their.login.password...
    >
    > * For all users,
    >
    > - aliases to equate different names to the same user
    > - aliases to provide email forwarding. We do not use .forward files as
    > there is no real home directory for the users in the mail server. -
    > aliases to provide lists or multiple forwardings
    >
    > * For virtual domains, using virtusertable,
    >
    > - maps domain email addresses to unique user names.
    > - define default email address or user for each domain.
    >
    > As an on-going business, we just cannot stop the current email
    > arrangement and ask all users to change their settings. How can DMail be
    > programmed to simulate the above settings?
    >
    > We maintain our own database and do not mind if I have to create or
    > modify tables to suit the DMail structure. Please give us some
    > suggestion and do not hesitate to ask if there is any query.

    You have 2 options.

    1. you can continue to use the same system password file,passwd, shadow, access, aliase and virtusertable created from the database. You should find that there is not too much to change in converting to DMail.

    2. you can run our new SQLAuth authentication module so that the DMail servers talk directly to the MySQL database.

    I suggest that you start with 1 as that will be quickest and then once you are happy with that you can look to simplify your setup and take advantage of things like our proper virtual domains (rather than the virtusertable) by moving to option 2.

    So for option 1, here are the things that I can think of for you to consider, in addition to what is in the Moving to DMail FAQ (i.e. this page)...

    1.RE: system password file, i.e. passwd and shadow passwords

    If you install DMail on a linux box it will default to,
    authent_method unix_user
    in /etc/dmail.conf.

    This means that DSMTP and DPOP will use the standard system user calls so you should not have any problems staying with those files as your mail user database.

    (Note: Because your usernames are unique and I presume they are not the full user's email address, you do not need to use the authent_domain or preserve_domain settings, which you will see talked about throughout the manual).

    2. RE: access

    I don't know which file you are talking about. I presume that it is for restricting who can access the server - maybe to stop you from being an open relay and other uses.

    Can you send me an example of this file and tell me more about its function.

    3. RE: aliases and virtusertable

    DSMTP uses the sendmail style alias files (99% of features are covered), so again you should not have any problems with these. They are specified for you main domain (as set by the first host_domain setting in dmail.conf),
    alias_file <filename>
    and for specific domains with,
    alias_file_domain domain <filename>
    where domain can be a wildcard, e.g.,
    alias_file_domain *bob.com /etc/aliases

    We have also recently added support for sendmail's virtusertable. Previously you had to convert all entries within that table to a mixture of our dmail.conf settings,
    forward
    forward_cc
    fallback_address
    and aliases within alias files.

    With beta version 2.8d, you should be able to use the setting,
    virtual_user_pre <filename>
    and it should work without modification.

    NB: virtual_user_pre is a new beta feature so be aware that it is possible there are problems with it. Please let me know straight away if you come across any such problems.

    Version 2.8d is available from the beta directory of our site.

    4. general settings:

    You indicate that your machine name is,
    mail.bob.com
    I presume that the email domain is actually, bob.com. In which case in /etc/dmail.conf you should set,
    host_domain bob.com
    as your FIRST host_domain setting (which tells DSMTP that it is a local domain) and as your second host_domain setting enter,
    host_domain mail.bob.com
    or
    host_domain *bob.com
    If you have other subdomains that are synonyms of the domain bob.com

    I recommend that you take advantage of the free trial period to set up a test box using your configuration. If you come across any problems please send us your dmail.conf file and either the dsmtp.log or dpop.log file showing the problem from the log_path directory. NB: it is best to send us the logs created when the logging level is set to debug. To set this edit log_level to read,
    log_level debug
    and then reload both DSMTP and DPOP with the commands,
    tellsmtp reload
    tellpop reload
    (as required by most dmail.conf settings).