POPFetch - The DSMTP 'side salad'

Under Construction :-)

Note: POPFetch is a new DMail add-on that is still in beta form - it should not be used unless you are instructed to do so by DMail support!


Introduction

PopFetch is a "side salad" for your SMTP server. It runs beside your SMTP server and works with it in order to provide email for an intranet without a constant internet connection.

On Windows platforms it can automatically dialup to your ISP (RAS dial). This provides economical use of your internet connection.

PopFetch is a perfect solution for email at your remote offices.

So what does it do ?

PopFetch periodically connects to a POP server and collects all mail for a specific email account.  It then feeds that mail to a local SMTP server so that it is ready for collection by email clients on the local network.

When PopFetch feeds the messages to the SMTP server it addresses them to the same recipient

PopFetch can be set to change the domain name of the recipient address(es) before it passes them on to the SMTP server, so that the SMTP server delivers them to local users, e.g. if PopFetch collects messages from a POP server at netwin.co.nz ,

say for john@netwin.co.nz,

and PopFetch is set to re-address for, say, branch_office.netwin.co.nz, then it will re-address the message to john as,

john@branch_office.netwin.co.nz

 

In conjunction with Netwin's DSMTP, SMTP server . . .

PopFetch works in conjunction with DSMTP (Netwin's SMTP server) so that when it starts a dial up networking connection, DSMTP can deliver all mail waiting to be sent to non-local addresses.

What happens when a message cannot be delivered?

POPFetch always copies the current message that it is retrieving to file (msg.dat). It then tries to send the message immediately on to the SMTP server. If for any reason this delivery fails, POPFetch stops and closes all TCPIP connections and waits for the next dial up time, at which point it will try to send the unsuccessful message again.

If the message cannot be delivered because the information stored in msg.dat is corrupt, or the message cannot be sent to any users - even the default address to say that a user cannot be reached, then POPFetch will write the message to a 'duds' file, duds.dat. There is a safety net setting, max_duds with a default of 10, such that if POPFetch detects that it has delivered the max_duds number of messages in a row to duds.dat then it will shut itself down (stop the popfetch service on NT).

Notes:

  • If a message fails to be delivered to the SMTP server, POPFetch stops and closes all TCPIP connections and waits for the next dial up time, at which point it will try to send the unsuccessful message again.
  • Every time the ras_timer period is reached, popfetch will try to retrieve mail for all accounts specified by a pf_user line.
  • If a message cannot be delivered, for example if the recipient does not exist or the SMTP server rejects the recipient address, then the fallback address will receive the message instead. POPFetch adds some informational lines to the message body stating which recipients have received the message and which have not.
  • log_path \dmail\log (\getall.log)
  • work_dir defaults to no path, then log_path def to work dir
  • Ras timer sets period of dial rather than interval, by default (isinterval=FALSE)
  • POPFetch knows that the next line of an SMTP response is part of the same comment when the fourth character of the response is a dash, '-'.
  • reads headers that go over multiple lines. The only header where this matters for POPFetch is the header set by pf_header
  • if the pf_header setting cannot be found in the message to be sent out, POPFetch looks for the 'X-Rcpt-To: ' header, followed by the 'To: ' header, then if even that header cannot be found it will send the message to the default address, if that does not exist then the message will be appended to the duds file, duds.dat
  • -DFLT_WORK_PATH does not exist - i.e. use exe directory
  • minimum ras_timer setting of 5 minutes if doing RAS dial, i.e. if you have a ras_entry or ras_number setting, otherwise there is no mimimum setting.
  • The ras dial up is considered to have succeeded if another program is controlling the dial session - there is a pf_wait setting to set a 'try-again' time to counter the fact that the other process might still be doing the dialup. This wait period is only used if POPFetch cannot reach the servers when it wakes up.
  • If a message fails to be delivered to multiple recipients, POPFetch will only send one message to the pf_dflt_address, with a paragraph added to the top of the message specifying which recipients did not receive the message.


Features:

  • Can connect to multiple POP servers and/or multiple accounts.
  • Can be set to do RAS dial up to the internet on Windows platforms.
  • Can alter the domain in the destination address.
  • Can be set to read destination address from any header,
    e.g. To:, X-Recip-To: etc.
  • Can be run as service on Windows NT
  • specifiable caretaker address for failed messages to be delivered to.
  • Built in "safety net" shutdown in case of misconfiguration
  • Detect when a RAS dial connection is already open, including a configurable 40 second delay if the POP connection fails due to another program still being in the dialling process.

Also in conjunciton with DSMTP, Netwin's SMTP server, POPFetch can . . .

  • Make use of DSMTP's X-Recip-To: header, created from the RCPT TO: line of the SMTP envelope, to get accurate message delivery.
  • Use DSMTP's RAS dial ini settings (set in dmail.conf)
  • DSMTP automatically sends the Extended SMTP, ETRN command, when the dial up connection is made, to initiate the remote server to send any mail waiting for the local domains, including virtual domains.

 

POPFetch Settings

POPFetch reads DMail's configuration file, dmail.conf, to find it's settings. E.g. \winnt\system32\dmail.conf on Windows platforms and /etc/dmail.conf on Unix based platforms.

On all platforms, no matter whether you want RAS dialup or not, POPFetch uses the setting ras_timer x to decide when it should next check for new mail. So this is the first setting you should set. Settings specifically for POPFetch all begin 'pf_' in dmail.conf.

On windows NT it can also use the other DSMTP RAS Dial settings, to do RAS Dialup before each check so that it can reach the POP server from which it is to download mail.

The settings relevant to POPFetch are:

POPFetch will create a log file, popfetch.log, in the directory and at the level set by the DMail settings, log_path and log_level. See Other DMail Settings That POPFetch Uses below.

RAS Settings:

POPFetch specific settings:

Other DMail settings that POPFetch uses:


Details of POPFetch specific settings:

Example settings in dmail.conf might be:

pf_user fred passwd 1.2.3.4
pf_user fred2 passwd 4.3.2.1
pf_domain local.domain
pf_max_duds 15
pf_dflt_address fallback@local.domain
pf_retry_delay 60
ras_timer 5
ras_number 1,,12345678
ras_entry
ras_username our_account
ras_password account_password
ras_domain
ras_smtpip 1..2.3.4
ras_timeout 10

Files used by POPFetch

Note: POPFetch attempts to use the file paths as used by the rest of DMail. If it cannot then POPfetch will simply work in the directory that the executable is running from.

So most of the files below you will find in DMail's work path as set by the work_path setting in dmail.conf.

msg.dat - temporary storage of a message.

dmail.conf - where settings are stored (DMail's configuration file).

duds.dat - all messages that fail to be delivered anywhere are appended to this file.

popfetch.log - the popfetch log file, found in the DMail log_path directory.


Downloading POPFetch

Starting with the 2.8 versions of DMail, you will find popfetch included in the DMail distribution set. So download the current one from this page and look in the temporary unpack directory, dmtemp.

NB: you do not need to use the 2.8 version of DMail, you can simply copy the popfetch executable out of the unpack directory.

If you can not find it there, then please contact DMail Support, as on some platforms we do not yet build it by default (you should find it on Windows and Linux platforms).


Installation NT

  1. Download POPFetch and copy it to your DMail main directory, typically,
    c:\dmail
  2. Edit the dmail.conf file,
    c:\winnt\system32\dmail.conf
    with notepad or another text editor, (e.g. enter the following at a dos command prompt,
    notepad c:\winnt\system32\dmail.conf
  3. In dmail.conf change the setting,
    log_level
    to read,
    log_level debug
    and add the following settings for POPFetch (NB: you need to work out the values for these settings which are correct for your setup),
    pf_user fred passwd 1.2.3.4
    pf_domain local.domain
    pf_max_duds 15
    pf_dflt_address fallback@local.domain
    pf_retry_delay 60
    ras_timer 5
    If you want the POPFetch to do RAS dial-up to the internet for you, then also set,

    ras_number 1,,12345678
    ras_entry
    ras_username our_account
    ras_password account_password
    ras_domain
    ras_smtpip 1..2.3.4
    ras_timeout 10
  4. Test POPFetch manually by running it in a dos command box with the -test option, e.g.,
    c:
    cd \dmail
    popfetch -test
    Because you have, log_level debug, set in dmail.conf POPFetch will log at debug level to the DMail log_path directory, typically,
    c:\dmail\popfetch.log
  5. Once you are happy with it's operation, you should set POPFetch to run as a service just like the DWatch process which runs the DMail servers.

    Starting with version 3 of DMail you will find an executable, addsvc.exe, in the c:\dmail directory, i.e. it is in the distribution set and should have been copied across when you installed DMail. If you can not find that executable, download it from here and copy it to your c:\dmail directory.

    Then enter the following in a dos box to add POPFetch as a service,

    c:
    cd \dmail
    addsvc -add popfetch c:\dmail\popfetch\popfetch.exe "POPFetch - message retrieval utility for DMail"
    You can then go to 'Start,Control Panel,Services' to check the service that you have added.

    (To remove the service use the command,

    addsvc -del popfetch
    )


Installation UNIX

  1. Download POPFetch and copy it to your DMail main directory, typically,
    /usr/local/dmail
  2. Edit the /etc/dmail.conf file, with vi or another text editor, (e.g. enter the following at a shell prompt,
    vi /etc/dmail.conf
  3. In dmail.conf change the setting,
    log_level
    to read,
    log_level debug
    and add the following settings for POPFetch (NB: you need to work out the values for these settings which are correct for your setup),
    pf_user fred passwd 1.2.3.4
    pf_domain local.domain
    pf_max_duds 15
    pf_dflt_address fallback@local.domain
    pf_retry_delay 60
    ras_timer 5
  4. Test POPFetch manually by running it at a command prompt with the -test option, e.g.,
    cd /usr/local/dmail
    ./popfetch -test
    Because you have, log_level debug, set in dmail.conf, POPFetch will log at debug level to the DMail log_path directory, typically,
    /usr/local/dmail/log/popfetch.log
  5. Once you are happy with it's operation, you should set POPFetch to be started by a startup script just like the DWatch process which runs the DMail servers.

    See the section, Startup Scripts for general information on finding your DMail startup script(s). These typically run the script,

    /usr/local/dmail/dwatch/dw_start.sh
    which typically looks like this...
    #!/bin/sh
    cd /usr/local/dmail/dwatch
    ulimit -c 20000
    /usr/local/dmail/dwatch/dwatch
    You can add a line to the bottom of that script so that it looks like...
    #!/bin/sh
    cd /usr/local/dmail/dwatch
    ulimit -c 20000
    /usr/local/dmail/dwatch/dwatch
    /usr/local/dmail/popfetch &
    You can then start popfetch with the command,
    /usr/local/dmail/dwatch/dw_start.sh
    It is safe to run this script when dwatch is already running because DWatch will see a copy of itself is already running and so not start another one.