Building Mails

One of TurboMail’s main goals is to make build emails easy. Therefore we create a Message class which hides all the tedious work like encoding the message header lines and building a MIME compliant body.

class Message([author[, to[, subject[, **kwargs]]]])

All arguments are optional. For the semantics behind these parameters, please read the ‘Properties’ section below. Through the use of kwargs you can set all properties documented below directly in the constructor (a TypeError will be raised if kwargs contains a key which is not a valid property).

Properties:

• author – Email address or addresses of the author(s)
• bcc – Blind carbon-copy address or addresses.
• cc – Carbon-copy address or addresses.
• date – Date if message creation either a date string as per RFC 2822, a datetime.datetime instance or the number of seconds since 1970 as float (optional, uses the current local time by default)
• disposition – Request disposition notification be sent (“email was read”) to this address.
• encoding – Content encoding specific to this message.
• headers – A list of additional messages headers (either as dictionary or as list of tupels).
• nr_retries - How often should TurboMail retry to deliver this message if the delivery failed? (default: 3)
• organization – The descriptive Organization header.
• plain – Plain text content. Will be automatically generated if a rich text part was specified.
• priority – The X-Priority header, in the range of 1-5.
• reply_to – Email address to which replies should be sent (Reply-To header)
• rich – The rich-text (“HTML”) part
• sender – email address of the agent which sends the mail (required if you specified more than one author)
• smtp_from – The SMTP envelope address, if different from the sender (this option is only useful for the SMTPTransport).
• subject – A textual description or summary of the content of the message.
• to – Email address or addresses which should receive the message (To header)

All email addresses can be given either as a string (TurboMail will convert unicode domain names with idna encoding if possible) or as a tuple (‘Full Name’, 'name@example.com‘). Additionally, you can store a list of addresses to the bcc, cc, and to address lists, with individual list elements as either strings or tuples, as above.

The plain- and rich-text parts of the message may be defined as any callable which returns plain- or rich-text. The callable is executed at the time the message is built - usually just prior to the first delivery attempt.

The encoding can be overridden on a per-message basis (you have to pay attention that all characters in your plain or rich contents may be encoded with the chosen encoding). If you enable the UTF-8-QP extension the message body will be encoded with UTF-8 Quoted Printable (Python’s email module uses base64 by default for UTF-8 content).

In order to simplify the use of the Message class even further, you can set default values for every property in your configuration. These configuration values are used in Message’s __init__ (constructor) when you don’t set a value explicitely. The configuration key is the name of the property, prefixed with ‘mail.message’ e.g. ‘mail.message.bcc’. Using that mechanism you can send all messages in copy to a certain address (assuming you don’t overwrite the bcc property after the message instantiation).

Methods:

attach(file[, name])

Attach a given file (either on-disk by passing a path, or in-memory by passing a File descendant) to the message. The name argument is required if passing an in-memory File descendant.

embed(file, name)

As per attach, but limited to image files for embedding in the rich-text (HTML) part of the message. Name is the desired CID of the embedded image.

send()

Send the message via the currently configured manager. In order to send a message, you need to define at least an author, a recipient (either in ‘to’, ‘cc’ or ‘bcc’) and the message content (via ‘plain’ or ‘rich’).

Plain Text Messages

Easy things first: You want to send an email with just some information in it (e.g. unhandled exception in your application). This kind of mail doesn’t have to look fancy, so we’ll just use plain text:

from turbomail import Message
message = Message("from@example.com", "to@example.com", "Hello World")
message.plain = "TurboMail is really easy to use."
message.send()

You see, building a plain text email is very easy: Just put your message content in the ‘plain’ attribute and send the mail. Done. In real world scenarios you probably would use some kind of template engine to generate the text body but that’s your choice.

Messages with Attachments

Sometimes you need send some files along your message content (e.g. a PDF document or some zip file). There two alternative ways of doing it with TurboMail. This is convenient because sometimes you generate a file on-the-fly (like a PDF report with reportlab/z3c.rml) and don’t want to write it to the hard disk (slow, you have to handle disk full conditions etc):

from turbomail import Message

message = Message("from@example.com", "to@example.com", "Sales are skyrocketing!")
message.plain = "Have a look at the sales statistics attached."

filename = '/home/fs/latest_sales.pdf'
# The file will be named 'latest_sales.pdf' in the message
message.attach(filename)

# Alternative way to attach a file (please note that you have to provide a
# name for the file to attach):
fp = file(filename)
# The file will be named 'sales_report.pdf' in the message
message.attach(fp, 'sales_report.pdf')

message.send()

HTML Messages

Originally emails contained only plain text which does not provide much formatting possibilities especially compared to all those fancy looking websites that came up in the late 90ties. Since the advent of the MIME standard you can use HTML in your messages to produce a so called HTML e-mail.

To

html_part = """<html><header/>
<body>
  <h1>Look,</h1>

  this is an HTML message. Really <b>w00t</b>.
</body>
</html>
"""

from turbomail import Message
msg = Message("foo@example.com", "bar@example.com", "Writing a HTML message")
msg.rich = html_part
msg.plain = """This is the boring alternative part for people who don't see
the HTML part."""
msg.send()

You can build an HTML email just by assigning a string which contains the HTML markup to msg.rich. You should always provide msg.plain too just in case someone does not want to read your HTML. If you don’t provide a plain text TurboMail will derive a plain text part from your HTML by using a very simple algorithm [1].

In Mozilla Thunderbird 2 the resulting message looks like this:

The HTML string above is static. In most applications you probably use a template engine like Genshi to produce the final HTML.

Be warned: Keep your HTML simple. E-mail clients generally don’t support the full subset of HTML/CSS. If you are frustrated because Internet Explorer does not like your web page wait until you saw your HTML message rendered by Outlook express before you start cursing... Another difficulty is that there are more email clients with significant market share than browsers (you should count different web mail providers as different mail clients because they all cut different parts of your message to protect their webmail users).

[1]	If you need more information about TurboMail’s built-in HTML to text converter please see Clever plain text generation for your rich text messages.

HTML Messages with embedded images

If you want to have really good looking emails you can add images. Unfortunately it is somewhat complicated because you can’t just attach an image and reference it from your HTML like in a web page. Instead you have to use the embed function which will tell the recipient’s mail client that the image is meant to be shown inside the HTML and is not an ordinary attachment.

After you added the image to the message, you can reference the image in your html mail by using ‘cid: + filename (without path) as the link target.:

<img src=”cid:my_image.jpg” />

The other alternative which keeps the mails also very small is to reference your images with absolute urls to your server. So the recipient has to fetch the images every time he wants to read your message. But this method may not be reliable as some mail clients won’t load images from remote servers (or require special user interaction to do so like in Mozilla Thunderbird) [2].

[2]

Spammers and other evil people used remote images (called web bugs) to track people so they knew who read their emails. Therefore quite few (especially the computer-savvy) people don’t like linked images and many popular mail clients refuse to load files from remote URLs, displaying your mail without any images.

Pre-built Messages

Sometimes you don’t want to create a Message with TurboMail - maybe the message is already on your hard drive or it will be generated by some other mechanism (be it legacy code or just a highly specialized mail-generation library). However you want to benefit from the facilities TurboMail provides (e.g. testability of mail sending, threaded mail sending, ...). This is when the WrappedMessage comes in:

from turbomail import WrappedMessage

# We're using a hard-coded string here - of course you can get this string
# from another library, this is just to make this example as easy as possible.
generated_message = '...'

message = WrappedMessage('from@example.com', 'to@example.com',
                         message=generated_message)
message.send()

The WrappedMessage just wraps a message and adds the minimum information which is necessary for delivery. The contents of your message will not be modified in any way.

Please note that you have to specify the ‘sender’ and the recipient(s) explicitly when building a WrappedMessage - although the message you’re about to pass may contain this information already. However, when it comes to delivery mechanisms like SMTP this information can not be extracted from the message content without guessing. You can find an in-depth explanation of the underlying mechanism in your chapter RFC (2)822 - From, Sender, Disposition, what?.

How TurboMail handles Internationalization, Character Sets, and Encoding

When RFC 822 was written (1982) emails were restricted to ASCII characters only. International characters like German umlauts (“ü”, “ß”), and latin accents (“é”, “ò”) were not allowed. As time went by the original format was extended in a backward compatible way so you could write plain text messages which contain non-ASCII characters.

The default encoding for TurboMail is still ASCII (the same holds true for almost all mail servers). If you want to send mails which contain non-ASCII characters, you have to set the configuration option ‘mail.message.encoding’ appropriately (see TurboMail Configuration). The broadest character set available is UTF-8. Unfortunately Python will encode the body part using base64 if you choose “UTF-8”. This means that a human can not read the message by looking at the plain text which may help debugging problems. Furthermore it will increase the message’s size by approximately 1/3.

To get around these problems, TurboMail offers a simple extension called utf8qp that overrides the default base-64 encoding for UTF-8 text with quoted printable encoding. Python’s email module will use UTF-8 Quoted Printable for messages that use ‘utf8’ or ‘utf-8’ encodings, which is much more human readable than base64 and won’t increase the message size as much as base64. Please note: you will still need to explicitly define your default encoding (or the encoding on a per-message basis) as ‘utf-8’. The extension will not do this for you automatically.

Over the last few years internationalized domain names have become available, allowing the use of umlauts and other extended characters in the domain part of an e-mail address, left of the @ character, e.g. foo@zääz.de. In the future we will probably see internationalized top-level domains. Most email servers will require that you convert internationalized domain names to punycode so that it only contains ASCII characters. With TurboMail you don’t have to care about the conversion as TurboMail will do it for you. However if you choose to use WrappedMessage, you must convert IDNs in your email header (not the SMTP envelope) yourself [3].

[3]	Technically you may not need to do this but often this is necessary so that the readers mail client will display the address correctly.

How to get your Messages through Spam Filters

No, this section is not about how to become a “Spam King”. Due to the amount of spam everyone receives on a daily basis, most of your recipients will use spam filters to filter spam (or rely on their providers like Google or Yahoo to do so). Assuming that your application only sends out legitimate mail, how do you avoid that your mail is accidentally marked as spam?

“Spamminess” is nothing which can defined just by one score value. Every spam filter will use its own rules and thresholds when to classify a message as spam. Most things that influence the spam score are beyond TurboMails’ control: message content, having enough innocent text, sending mails from a static IP address with DNS correctly set up etc. It is impossible to assemble a message that will pass all your recipients spam filters!

However, TurboMail users reported that Google Mail classified their messages as spam if they used UTF-8 characters in their message body. The problem was solved by using the UTF-8 Quoted Printable “encoding” [4]. The previous section How TurboMail handles Internationalization told you already how to do this.

Furthermore it may help if you add the full name of the recipient as sender and not just the email address (assuming you know the name):

# instead of:
msg = Message("foo@example.com", "bar@example.com", ...)
# better write:
msg = Message(("Foo", "foo@example.com"), ("Bar", "bar@example.com"), ...)

If you build HTML messages with TurboMail’s Message class, you should care about the plain text part, too. Message tries to derive a plain text part from your rich-text (“HTML”) part automatically if you don’t set any plain text explicitly. But the algorithm for generating the plain text part is not very clever so some parts of your markup may be included in the plain text part. Some users reported that spam filters classified their message as spam due to these left-overs. We try to improve the text filters if we become aware of these issues but you may think about providing a plain text part explicitly or just disabling the plain text generation. Another alternative is to plug in a better html2text converter.

[4]

If you use non-ASCII characters in a message, Python’s email module may resort to base64 as encoding for your body. In the early days of mass spam spammers used this trick to get their advertising to the recipients by encoding their message bodies with base64. Most scanners at that time did not preprocess messages and just looked for some keywords but base64 will make an unreadable character stream out of your plain text. Something like ‘TurboMail’ becomes ‘VHVyYm9NYWls\n’ when encoded with base64. The users mail client was able to decode base64 without any problems so that may be the reason why most spam filters still give base64 a bad rating.

Sending Mails

When you instantiated a message and added all the information (e.g. message body), you have to tell TurboMail to send the message explicitly. The easiest way of sending a message is calling the send() method of a Message:

from turbomail import Message
msg = Message('foo@example.com', 'bar@example.com', 'Subject')
msg.plain = "Foo Bar"
msg.send()   # Message will be sent through the configured manager/transport.

When the send method is called, an email.Message will be assembled and handed over to the manager you configured. Whether the message will have to wait in a queue until there are free resources to deliver it to the intended recipient or if the message is given to the configured transport immediately, depends on your configuration.

Please note that every time you call the send method, an email is sent.

In TurboMail 2 you had to call turbomail.enqueue(msg) to send a message. This method is still available for backwards compatibility but its usage is deprecated.

If an error occurs while sending the mail, TurboMail will retry several times to deliver the message. If that is not successful, the message will be dropped. If you use the ImmediateManager, you’ll get an exception (smtplib.SMTPError or socket.error). If you use the asynchronous DemandManager, the failure will be logged. We’re not satisfied with that behavior but we were not able to fix this in time for TurboMail 3.0. For 3.1 we’ll build a more robust and flexible system with hooks so you can do your own error handling routines.

TurboMail Configuration

TurboMail’s behaviour (e.g. what default encoding should be used) is highly configurable. If you use TurboMail in a supported framework which has a TurboMail adapter TurboMail will use the general configuration system of your framework. For TurboGears 1 this means you can put configurations for TurboMail in app.cfg and/or {dev, prod}.cfg and TurboMail will pick them up automatically. The configuration is set when TurboMail starts up and should not be changed afterwards.

Configuration is based on key-value pairs where the key is always a string. What type is required for the value depends on the key. Unless otherwise noted, the values should be specified as strings, too.

Later in this manual we use the ConfigObj syntax when we refer to specific settings. For example, “mail.on = True” means that TurboMail’s configuration contains a key “mail.on” with the boolean value True. Which configuration files you have to edit to configure (and if you have to use a special syntax for some values) depends on your framework. We have more information about the supported frameworks in TurboMail Adapters.

Throughout this documentation we use the ConfigObj syntax:

mail.on             = True        # boolean value True for the option 'mail.on'
mail.manager        = 'immediate' # string value 'immediate'
mail.demand.threads = 5           # int value 5

This section deals with general configuration options which are not specific to a single manager or transport. You find specific options in chapter Detailed TurboMail.

Configuration options

The single most important option is mail.on. If try to use TurboMail without setting this option as True, you will get an exception like this immediately: “MailNotEnabledError: An attempt was made to use a facility of the TurboMail framework but outbound mail hasn’t been enabled in the config file [via mail.on]. This was done on purpose because the default transport does not send out mails (so you don’t spam real people with mails accidentally generated in your unit tests [5]) but just not sending the mails could be frustrating experience for new users when their mails go to /dev/null. Furthermore this saves some bytes of RAM because neither a manager nor a transport will be loaded when TurboMail is not enabled.

• mail.on (boolean, default: False) Enable TurboMail.
• mail.manager (string, default: ‘immediate’) Specify the name of the manager to use
• mail.transport (string, default: ‘debug’) Specify the name of the transport to use
• mail.message.<property name> (string or tuple) – default values for properties of the Message class (see Message properties)

The individual Managers and Transports - Your Work Horses can have their own configuration options. These are explained in the section where the specific manager/transport is explained in more detail. We just make an exception to explain a mandatory option for the SMTPTransport (which is probably by far the most widely used one):

• mail.smtp.server (string, mandatory for SMTP transport) host name or IP address of the SMTP server which will take care of mails sent by TurboMail (only if SMTP transport was enabled). [6]

[5]

You do use foo@example.com as recipient for all your test emails, right? The example.com domain is reserved for documentation so you will never bother someone if you sent a mail to this domain. Usage of existing domains for test purposes is really a pain and kills kittens. And yes, donotreply.com does exist.

[6]

We deliberately did not set ‘localhost’ as default value for that because on Un*x machines there is often an SMTP server running on the local machine which may decide to deliver messages to real mail servers. After you mailed a badly written test email to a few thousand customers accidentally you’ll understand our paranoia here :-)

How transports, managers and extensions are found by TurboMail

TurboMail heavily utilizes setuptools to build a modular, loosely coupled and extensible system. When you specify “mail.transport = ‘smtp’” in your configuration, TurboMail tries to load the class associated to the entry point ‘turbomail.transports.smtp’. If no class could be loaded, an error will be logged and TurboMail will be disabled.

If you try to use TurboMail after the an error, you will get the same MailNotEnabledError as if you forgot to specify “mail.on = True”.

The same holds true for managers (entry point group ‘turbomail.managers’) and extensions (entry point group ‘turbomail.extensions’). The only difference related to extensions is that TurboMail won’t be disabled if an extension could not be loaded.

Add new transports without using setuptools

Sometimes you don’t want to rely on setuptools for manager/transport discovery. For example if you run the unit test suite for TurboMail itself, we don’t want to force you to install the TurboMail egg in your environment. Therefore we added a mechanism so that you can use transports and managers in TurboMail without the requirement that setuptools must be able to find them via its normal entry point mechanism:

config = {'mail.on': True, 'mail.manager': 'foo'}
# assuming you built a manager class called MyOwnManagerClass
interface.start(config, extra_classes={'foo': MyOwnManagerClass})

If extra classes where provided, TurboMail will look in this dict before the lookup with setuptools. If you use this mechanism, you can be sure that the specified manager class is used regardless of the environment configuration which is nice especially for unit tests.

Logging

TurboMail supports logging through Python’s standard logging module. All loggers utilized by TurboMail start with ‘turbomail.’ so you can easily disable all logging (don’t forget to stop propagation if you want to silence TurboMail). However we recommend setting the log level for TurboMail to WARNING so you will be notified of potential problems.

Furthermore there is a special delivery log where all delivery attempts are noted. To handle mail delivery log messages differently from other TurboMail log messages (e.g. for later auditing), you can define other handler/log level settings on ‘turbomail.delivery’ (mail deliveries are logged with log level ‘INFO’).

Unit Tests with TurboMail

Testability was advertised as a feature in the beginning. So how to use TurboMail in your (unit) tests?

First of all, you should use the ImmediateManager and the DebugTransport for testing so you get sent messages as fast as possible (ImmediateManager) and you can collect your messages easily without any interaction with your environment (DebugTransport). Fortunately, this is just the default configuration for TurboMail if you don’t configure any managers or transports explicitly.

A simple test case looks like this:

import unittest

from turbomail.control import interface
from turbomail.message import Message

class TestYourAppWithTurboMail(unittest.TestCase):
    def setUp(self):
        config = {'mail.on': True}
        interface.start(config)

    def tearDown(self):
        interface.stop(force=True)
        interface.config = {'mail.on': False}

    def test_fetch_sent_messages(self):
        # call your app - do some interesting things
        # ...
        # no you can retrieve the sent mail:
        send_mails = interface.manager.transport.get_sent_mails()
        self.assertEqual(1, len(send_mails))
        sent_mail = send_mails[0]
        # check that the mail contains the expected information

This test assumes that TurboMail is installed in your environment with all the necessary egg information so that it can load the ImmediateManager and the DebugTransport. If you want to be independent of setuptools, please have a look a TurboMail’s test_debug_transport.py test case does not rely on setuptools too.

If you want to test a TurboGears 1.0 application and you use a test case which calls turbogears.start() you don’t have to issue interface.start and interface.stop because TurboGears will handle that for you. Additionally you should not configure TurboMails’ interface but just use turbogears.config to set the appropriate values for TurboMail.

If you want to see some real test cases with TurboMail, you may want to look in TurboMail’s own test suite for examples.

Compatibility with TurboMail 2.x

For TurboMail 3 we reworked all internals. Nevertheless we tried to maintain compatibility for the most important interfaces. The idea is that 90% of all applications using TurboMail 2 will “just work” with TurboMail 3 without the need to change any code (just one line in the configuration). However we deprecated some interfaces and properties. If you use one of these, a DeprecationWarning will be raised. You can disable these warnings in your application easily:

from warnings import filterwarnings
filterwarnings('ignore', 'warning text specified with a regular expression',
               category=DeprecationWarning)
# Now you can use the deprecated interface and no warning will be given

The most important changes for upgraders are probably that the default manager is now the ImmediateManager (message delivery is synchronous by default, your code needs to deal with exceptions raised during the mail delivery) and the default transport is the DebugTransport which means that no mail is delivered unless you use the SMTPTransport in the configuration.

Configuration

One of the main features of TurboMail 3 are different types of transports. In order to send messages with SMTP in your production system you must add this configuration option:

mail.transport = "smtp"

If you forget to do this, no messages will be delivered!

In TurboMail 2 all parameter keys used the naming convention ‘mail.<name>’. With TurboMail 3 this often does not make sense because we have different transports now (not only SMTP) and many options are only useful in a SMTP context. Therefore all SMTP related parameters are renamed from ‘mail.foo’ to ‘mail.smtp.foo’. The options mail.server, mail.username, mail.password, mail.debug and mail.tls were renamed. The old parameter names will continue to work but a DeprecationWarning is given when TurboMail encounters one of the old configuration names.

Message class

All properties from the old Message class should be still there. Please note that we renamed the ‘recipient’ property to ‘to’ to be more clear which mail header is meant. ‘smtpfrom’ and ‘replyto’ were renamed in accordance to PEP 8 (the official style guide for Python code) to ‘smtp_from’ and ‘reply_to’.

The most important change functionality-wise is the rename of ‘sender’. We felt that ‘sender’ is ambiguous because it was used for the From header but there is another header that is called ‘sender’, too. Therefore the attribute to set the From header is now called ‘author’ and there the property ‘sender’ will now set the Sender header. If you don’t set the ‘author’ property but use ‘sender’, your ‘sender’ property will be used for the ‘author’ so we keep the backwards compatibility.

Sending Messages

In TurboMail 2 you submitted messages for sending by doing this:

from turbomail import Message, enqueue
msg = Message(...)
enqueue(msg)

In TurboMail 3 messages can send themselves so the code above should be written as:

from turbomail import Message
msg = Message(...)
msg.send()

# alternatively you can send messages by doing:
from turbomail import send
send(msg)

Furthermore we felt that ‘enqueue’ is not a good name any more because now we have different strategies of sending messages and not all managers will queue your message. Therefore the ‘enqueue’ method is still there but is marked as deprecated.

Broken Compatibility

We deliberately broke compatibility for some add-on kludges made in TurboMail in order to get a much nicer interface in TurboMail 3. The test mode of TurboMail 2 is completely unsupported although the same functionality is provided by the DebugTransport (see unit testing with TurboMail). Any code that accessed turbomail._queue directly will be broken in TurboMail 3.

As noted above, we changed the default manager behaviour to be synchronous and the default transport (DebugTransport) does not deliver any messages to the outside world.