1

Dr. David MacQuiggResearch AssociateAutonomic Computing Laboratory

Email System – The most important application of computer networks

University of ArizonaECE 4783 December 2009

April 18, 2023 2

• The “intractable” problems with email– Spam & lost messages, $20B/year

– Fraud and other serious crimes

– Enabler for most malware

– Threats to critical infrastructure

• Reasons for these problems– Ignorance

– Identity fraud (can’t separate the good guys)

– Investment in the status quo ($2B per year)

• Possible technical solutions

– More of the same (IP blacklists, statistical filters)

– Reputation-based systems

April 18, 2023 3

• The problems with email– Spam problem, $20B/year, “intractable”

– Fraud and other serious crimes

– Enabler for most malware

– Threats to critical infrastructure

• Reasons for these problems– Ignorance (users and admins)

– Identity fraud (can’t separate the good guys)

– Investment in the status quo ($2B per year)

• Possible technical solutions

– More of the same (IP blacklists, statistical filters)

– Reputation-based systems

April 18, 2023 4

• The problems with email– Spam problem, $20B/year, “intractable”

– Fraud and other serious crimes

– Enabler for most malware

– Threats to critical infrastructure

• Reasons for these problems– Ignorance

– Identity fraud (can’t separate the good guys)

– Investment in the status quo ($2B per year)

• Possible technical solutions

– More of the same (IP blacklists, statistical filters)

– Reputation-based systems

EmailSystem

detailed model

http://en.wikipedia.org/wiki/Internet

April 18, 2023 5

April 18, 2023 6

The Internet Today

Our Domain

Trusted Domains

X

???

April 18, 2023 7

Textbook Model of the Email System

Figure 9.1 Sequence of mail relays store and forward email messages

{Peterson & Davie, Computer Networks, 4th ed.}

April 18, 2023 8

Real Mail Handling System

P. Faltstrom, mail-flows-0.4, Jan 6, 2004, http://www.ripe.net/ripe/meetings/ripe-47/mailflows.pdf

April 18, 2023 9

D. Crocker, "Internet Mail Architecture", 2009, http://tools.ietf.org/html/rfc5598.

Function modules and the protocols used between them

Relay-Level Model

April 18, 2023 10

Administrative-Level Model

+--------+ +---------+ +-------+ +-----------+ | ADMD1 |<===>| ADMD2 |<===>| ADMD3 |<===>| ADMD4 | | ----- | | ----- | | ----- | | ----- | | | | | | | | | | Author | | | | | | Recipient | | . | | | | | | ^ | | V | | | | | | . | | Edge..+....>|.Transit.+....>|-Edge..+....>|..Consumer | | | | | | | | | +--------+ +---------+ +-------+ +-----------+

Legend: === lines indicate primary (possibly indirect) transfers or roles ... lines indicate supporting transfers or roles

D. Crocker, "Internet Mail Architecture", 2009, http://tools.ietf.org/html/rfc5598.

Administrative Management Domains (ADMD)

April 18, 2023 11

The Email System ( a better textbook model )

  |--- Sender's Network ---|           |-- Recipient's Network -|                                  /  Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient                                /                             Border

http://en.citizendium.org/wiki/Email_system

April 18, 2023 12

Shorthand Notation for Email System Models

Simple Setup with four Actors

|--- Sender's Network ---| |-- Recipient's Network -| /Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient / Border

Actors, Roles and NotationActors include Users and Agents.

Agents may play more than one role, but no role has more than one Actor.Typical roles include Transmitting, Receiving, Forwarding, and Delivery.A Border occurs when there is no prior relationship between Agents.--> Direction of mail flow (no statement as to relationship)~~> Indirect relationship (e.g. both directly related to Recipient)==> Direct relationship between Actors (e.g. a contract)A/B Roles A and B both played by the same Actor

April 18, 2023 13

Other Common Setups

Simple Forwarding is quite common

|-------- Recipient's Network ---------| /--> / --> Receiver/Forwarder ~~> MDA ==> Recipient / Border

Chain Forwarding should be discouraged

|------------ Recipient's Network ------------| /--> / --> Receiver ~~> Forwarder(s) ~~> MDA ==> Recipient / Border

Open Forwarding must be banned

/ / |-- Recipient's Network -|--> / --> Forwarder --> / --> Receiver/MDA ==> Recipient / / Border Border

April 18, 2023 14

Roles and ResponsibilitiesAuthor- Originate messages- Provide a password or other means of authentication

MSA - Mail Submission Agent- Authenticate the Author- Manage Author accounts

Transmitter- Spam Prevention - rate limits, content analysis, alerts - respond to spam reports - maintain reputation- Authentication - RFC compliance - IP authorization (SPF, SID, CSV, ...) - signatures & key management (DKIM ...)

Receiver- Block DoS- Authenticate Sender - HELO, Return Address, Headers, Signature - reject forgeries- Assess reputation - whitelists- Filter spam- Add authentication headers- Manage Recipient accounts/options - whitelisting, blacklisting, filtering, blocking, forwarding- Process spam reports

April 18, 2023 15

Roles and Responsibilities (continued)Forwarder- Authenticate upstream Agent- Set up forwarding to downstream Agent - check RFC compliance - set up authentication records - submit forwarding request, wait for approval- Manage Recipient accounts - maintain database of forwarding addresses - suspend account when a message is rejected - communicate w Recipient re " "- Maintain reputation as a trusted Forwarder - certifications

MDA - Mail Delivery Agent- Authenticate upstream Agent- Sort and store messages- Provide access for Recipients - POP3, IMAP, Webmail- Manage Recipient accounts/options- Relay spam reports to Receiver (or don't accept them)

Recipient- Set up accounts with each Agent- Select options in each account- Report spam to Receiver

April 18, 2023 16

Secure Communications

Secure communications may require any or all of:

1)authentication of the source (individual or organization identity)2)verification of content (digital signature)3)confidentiality of content (encryption)4)originality (no duplicates)5)timely delivery (no unexpected delays)6)hidden communication (keeping an enemy unaware)

Solving the problems of bulk email abuse (spamming, phishing and other bulk mail scams) requires that we address items 1 and 4.

To be useful in email authentication, an identity must have three characteristics. It must be unique, verifiable, and suitable for accumulation of reputation.

http://en.citizendium.org/wiki/Email_authentication

April 18, 2023 17

Identities in an Email Session

$ telnet open-mail.org 25

220 open-mail.org ESMTP Sendmail 8.13.1/8.13.1; Wed, 30 Aug 2006 07:36:42 -0400

HELO mailout1.phrednet.com

250 open-mail.org Hello ip068.subnet71.gci-net.com [216.183.71.68], pleased to meet you

MAIL FROM:<macquigg@box67.com>

250 2.1.0 <macquigg@box67.com>... Sender ok

RCPT TO:<jman@box67.com>

250 2.1.5 <jman@box67.com>... Recipient ok

DATA

354 Enter mail, end with "." on a line by itself

From: Dave\r\nTo: Test Recipient\r\nSubject: SPAM SPAM SPAM\r\n\r\nThis is message 1 from our test script.\r\n.\r\n

250 2.0.0 k7TKIBYb024731 Message accepted for delivery

QUIT

221 2.0.0 open-mail.org closing connection

RFC-5321

Helo Name

Envelope Addresses: Return Address Recipient Addresses

RFC-5322

Header Addresses: From Address Reply-To Address

1

1

3

3

6

6

4

45

2 Network Owner

Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient

April 18, 2023 18

Email Authentication – The Challenge

SMTP makes forgery easy

Forger -------> / /Author ==> MSA/Transmitter --> / --> Receiver/MDA ==> Recipient / / / / Border / / / -- Secure Channel --

TCP makes IP addresses (relatively) secureThe source address is real, but it may be only a zombie!

DNS offers a (relatively) secure channelDomain owners can publish their transmitter addressesOr they can publish a public key

Nothing else can be trusted

April 18, 2023 19

Email Authentication Summary

IP-based Authentication (SPF, SenderID, CSV): Sender provides a list of authorized transmitter addresses via DNS. Can be very efficient (no data transfer) but may have a “forwarding problem” if the MDA thinks it is the Receiver.

Signature-based Authentication (DKIM): Sender provides a Public Key via a DNS. Messages are signed with the related Private Key. Message content can be very secure, but an un-trusted Forwarder can replay it to millions.

|--- Sender's Network ---| |--------- Recipient's Network --------| /Author ==> MSA/Transmitter --> / --> Receiver/Forwarder ~~> MDA ==> Recipient / / / / Border / / / ------ DNS -------

April 18, 2023 20

Analysis of SPF using our models

Simple Forwarding

|-------- Recipient's Network ---------| /MSA/Transmitter --> / --> Receiver/Forwarder ~~> MDA ==> Recipient / Border

SPF correlates the Return Address to the Transmitter’s IP address.Forwarders are expected to re-write the Return Address.Very few forwarders are doing that. “Proselytizing” has failed.A misconfigured MDA sees the forwarded message as forgery.The message is quarantined, and possibly lost.Senders are avoiding the loss by publishing “neutral” SPF records.Forwarders will not change until senders demand it by publishing “enforceable” SPF records.

Senders don’t care.SPF is stuck.

http://en.citizendium.org/wiki/Sender_Policy_Framework

April 18, 2023 21

Reputation – the other half of trust

Millions of legitimate senders are simply unknown

Aggregation of data is essential Ground Up: Gossip Top Down: Proprietary Systems Both: Registry of Internet Transmitters

Some legitimate senders are not qualified to operate a transmitterMake outsourcing the Transmitter role easy.

Accountability is essential – no excuses.

April 18, 2023 22

Suggested Receiver Setup

April 18, 2023 23

So why isn’t it happening?

Hurdles that proposed solutions have failed to avoid or overcome, in order of decreasing severity:

1) Required simultaneous upgrades in software or setup (Flag Day)

2) Required widespread adoption by Agents before any benefit is realized by Users

3) Required widespread adoption of one company's method or service (Microsoft patent)

4) Changes that cause a temporary degradation in service

5) Changes in current practicesa) A well-established and standards-compliant practice.b) A widespread but non-standardized practice. ("Misuse" of Return Address)c) A widespread but non-compliant practice. (bad HELO name)d) An already unacceptable practice. (open relays)

6) Costs to senders a) Must pay a fee, install new software, or incur some administrative cost. b) Worry about lost messages c) Need to keep track of their transmitter addresses

The real reason: Reversed incentives – more spam for everyone else = more money for us

April 18, 2023 24

Bibliography

• “Email System”, http://en.citizendium.org/wiki/Email_system - cluster of articles on how the email system works, email security, authentication methods, etc.

• "Internet Mail Architecture", D. Crocker, http://tools.ietf.org/html/rfc5598 - best relay-level model, with references to all the relevant RFC standards.

• Computer Networks, Peterson & Davie, 4th ed. – good on all relevant technologies except email.

• TCP/IP Illustrated, vol. I, The Protocols, W. Richard Stevens, 1994. Very thorough, yet readable. Good illustrations.

• Pro DNS and BIND, Ron Aitchison, Apress 2005. – Very readable book on the Domain Name System.

• "CircleID", http://www.circleid.com – a "Collaborative Intelligence Hub for the Internet's Core Infrastructure & Policies" – current articles by top industry experts.

Project Links• https://www.open-mail.org – current status of our Identity and Reputation System• http://purl.net/macquigg/email – articles and notes from early development.

A short list of the most useful books and articles on email and its underlying technology.

April 18, 2023 25

Economics of Email Abuse

$200B   annual benefit of email   $20B   cost of abuse          100M users x ($.25/day deleting spam + $100/yr lost emails)

    $2B   benefit to anti-spam industry         100 companies x $20M/yr

  $0.2B   benefit to spammers          10K spammers x $20K/yr

  $0.02B  cost of an effective authentication/reputation system         10M users x $2/yr          100K companies x $200/yr (90% internal, 10% external services)