Simple Mail Transfer Protocol (SMTP)

Simple Mail Transfer Protocol is used by Mail Transfer Agents that allows message transport even when there is no direct connection between the sender and the recipient.

E-mail has a special status in DNS: While most other services are directly associated with IPv4 or IPv6 addresses, e-mail will be sent to servers that are listed in special MX records.

For improved fault tolerance, e-mail servers may customarily store the received messages for up to a week until they can be delivered further. A server may be temporarily disconnected or the mailbox of the recipient be over quota. Usually such problems will be resolved within a few days.

Domain names may be associated with several MX records, so that when the primary server is loaded, alternative servers may be used.

E-mail is extremely robust thanks to its distributed, fault-tolerant and queue-based implementation.

The Spam Problem

With the commercialisation of the Internet in the 1990s, an unfortunate observation was made that e-mail is a cheap way for a sender to reach a large number of recipients, of which a large enough portion will take up the offer or fall victim to the scam.

As late as in the early 2000s some SMTP servers were wide open to the entire world, accepting messages sent in anyone’s name, claiming to be from anywhere, and destined to anywhere. Spammers could freely exploit the storage space for e-mail queues that had been paid by others, and recipients had a harder problem to thwart spam traffic, because it was arriving from a very large number of SMTP servers.

Well into the 1990s, a typical European university would manage its own Class B IPv4 address space (x.y.0.0/16, 65,536 addresses) and divide it into smaller segments by faculty or department. A static address was manually configured into each workstation or server, typically running a commercial version of Unix or the rising star GNU/Linux. Each workstation would also run its own SMTP server, communicating directly with the Internet. There were no firewalls, no NAT, no VPN. The only centralized component were the primary or departmental mail servers of the institution, which would buffer incoming messages in case the personal workstation of some researcher happened to be switched off.

With the advent of POP3 and IMAP it became more convenient to access remote e-mail mailboxes, rather than reading the local mail spool via a terminal connection to a VAX/VMS or Unix workstation or server. With the advent of mobile workstations and wireless networking, a switch to a more centralized infrastructure was necessary. Manually configured static IP addresses were replaced with dynamic ones obtained via BOOTP and later DHCP, and users were forced to switch to dedicated e-mail servers. Interestingly, DHCP does not provide any SMTP settings; they have to be configured manually.

Until early 2000s, Internet connectivity providers were also expected to provide e-mail service. Nowadays, one could switch connectivity providers multiple times per day (think about mobile devices and WiFi), and the e-mail service provider can be totally independent of them.

Inbound only to our users, outbound only from our users

The current practice is that an SMTP server will only accept messages from other SMTP servers or unauthenticated users to the addresses that it is responsible for.

End users are supposed to send messages via the SMTP server of their own e-mail service provider. Usually, a TLS connection, user name and password will be required.

Back in the day, it was common to send spam via hijacked devices in domestic networks. Nowadays it would be prevented for example by Traficom Regulation 67 on information security in telecommunications operations, which obliges service providers to block consumer-grade Internet connections from directly connecting to public SMTP servers outside the network of the service provider.

Thus, nowadays, one practically must send e-mail via the SMTP server of the service provider. This allows efficient spam protection. The receiving servers often make use of DNS based block lists of servers that are suspected of sending spam.

Differentiate from spam: SPF, DMARC, DKIM

Some of the most efficient ways of fighting spam are the DNS TXT record based DMARC and DKIM, which have largely superceded the old SPF.

Why will many webmail users not receive my messages?

Many e-mail service providers artificially delay or classify as spam some messages that do not conform to these policies.

The first send attempt may be rejected temporarily, requiring the sender to retry later. A later attempt from the same address or with the same metadata may be let through. The idea behind this is that not all spammers will implement message queues according to the specification, which means that some of the spam will be rejected efficiently.

The messages that are let through may still be flagged as spam, for example if the sender is missing from the recipient’s address book.

Because the Internet is a distributed system, improvements will take place gradually and transition periods can be long. Just like HTTP has been superceded by HTTPS, the transition to DMARC and DKIM will be completed eventually.

For example, there is a SPF TXT record for 208.fi that indicates the address of the sending e-mail server. In this way, a receiving SMTP server could easily reject messages that appear to originate from an invalid address.

What if 208.fi sends a message to a recipient who uses a forwarding service A, which will forward the message to a final address B? In this case, A will observe that the message arrives from the 208.fi mail server. But the SMTP server of B would observe that A is sending something in the name of 208.fi without being listed!

This design mistake of SPF was solved by DKIM: The original sender will sign the content of its message with its private key, whose corresponding public key is available from the DNS. For instance, the TXT record mail._domainkey.208.fi contains the public key of mail.208.fi. The receiving server can check if the DKIM-Signature header that was added by the sender corresponds to the contents and the public key.

What if a message is sent in good faith from 208.fi to a recipient that uses a forwarding service X, which sends the message to a million other addresses? Could mail.208.fi end up in some DNS block lists? (The forwarding service X should.)

SPF records are augmented by or have been replaced with DMARC, which indicates the preferred action for messages that have been classified as spam. It is common to request notifications. The postmaster would then investigate and reply to the message if needed, promising to address the situation. In this way, the block should be avoided.

The corrective action depends on the situation. If the complaint is about an individual user, the postmaster would contact the customer to resolve the situation. In an extreme case, it is possible to temporarily block the sending of further messages.