IndiMail - The Flexible Messaging Platform

IndiMailThe Flexible Messaging Platform

Manvendra Bhangui

http://www.indimail.org/


Strengths & AdvantagesComponentsFeature ListArchitecture

Installation & ConfigurationAdministrationMaintenance

Long-term GoalsSupport

Hardware RequirementsQuestions



Strengths & Advantages–All parameters customizable through environment variables (around 200 parameters).

–Customizable globally, locally, dynamically or specifically for a user or group of users.

–Mail can be intercepted before injection, after queuing or before final delivery

–Single instance multiple queues. Concurrency customizable

–Horizontally scalable architecture.

–Crash proof

–Faster than qmail, postfix (the only two noteworthy competitors)

–Full featured (SMTP, POP3, IMAP, SPAM Filter, Virus Scanning, SSL, Domainkeys, DKIM, BATV, etc)

–Highly secure. Can protect vulnerable messaging servers like MS Exchange by acting as frontend SMTP, IMAP and POP3 servers.

–Open Standards – Compliant with most of the messaging RFCs

–Open Source – Source code available (GNU GPL V3).

–accesslist – control email exchanges between senders & recipients. Control exchanges between internet and users. (unique to indimail & satisfies corporate requirements)

–High speed virus/spam filtering by using inline scanning.



ComponentsMTA -Based on qmail. Extensive changes to make it a modern

MTA and achieve highest possible delivery speed.

IMAP/POP3 - Based on courier-imap. Changes made to support multiple open standards authentication modules (v4.10.x)

Virus Filter - clamAV with automatic signature updates using freshclam (v0.96.x)

SPAM Filter- Bogofilter – robinson fisher algorithm (v1.2.2)



Feature List– Speed

● Probably the fastest MTA. Multi-queue architecture allows 1.5 millions+ deliveries on a intel commodity hardware

– Setup● automatic adaptation to your UNIX variant

● Linux, SunOS, Solaris, and more

● automatic per-host configuration - gnu autoconf

● High degree of automation of configuration through svctool

● RPM packages for multiple Linux Distros.



Feature List (cont...)– Security

● clear separation between addresses, files, and programs

● minimization of setuid code (qmail-queue, qhspi, qscanq, systpass)

● minimization of root code (qmail-start, qmail-lspawn)

● five-way trust partitioning---security in depth

● optional logging of one-way hashes, entire contents, etc. (EXTRAQUEUE, mailarchive control file)

● virus scanning through qscanq, clamav.

● Extensible plugin feature for virus scanners

● Inbuilt virus scanner

● sender/recipient accesslist, hostaccess using tcprules



Feature List (cont…)– Message construction

● RFC 822, RFC 1123

● full support for address groups

● automatic conversion of old-style address lists to RFC 822 format

● sendmail hook for compatibility with current user agents

● header line length limited only by memory

● host masquerading (control/defaulthost)

● user masquerading ($MAILUSER, $MAILHOST)

● automatic Mail-Followup-To creation ($QMAILMFTFILE)

● ability to add signature/content to messages using altermime

● Abuse report format (ARF) generator using qarf


http://tools.ietf.org/html/rfc822




Feature List (cont…)– SMTP

● RFC 2821, RFC 1123, RFC 1651, RFC 1652, RFC 1854, RFC 1870, RFC 1893

● 8-bit clean

● 931/1413/ident/TAP callback

● relay control---stop unauthorized relaying by outsiders (control/rcpthosts)

● no interference between relay control and aliases

● automatic recognition of local IP addresses

● per-buffer timeouts

● hop counting * parallelism limit (tcpserver)

● per host limit (tcpserver - MAXPERIP)










Feature List (cont…)– SMTP (cont…)

● refusal of connections from known abusers(tcpserver, badmailfrom, badmailpatterns, badhelo, blackholedsender, blackholedpatterns, badhost, badip)

● goodrcptto, goodrcptpatterns which override the above

● blackholercpt, blackholercptpatterns for blackholing mails to specific senders.

● Control files spamignore, blackholedsender, badmailfrom, relaymailfrom, badrcptto, chkrcptdomains, goodrcptto, blackholercpt, badip can be specified in plain text, cdb format as well as stored in MySQL tables.

● relaying and message rewriting for authorized clients

● authenticated SMTP PLAIN, LOGIN, CRAM-MD5, CRAM-SHA1, CRAM-RIPEMD, DIGEST-MD5 HMAC (RFC 1321, RFC 2104, RFC 2554, RFC 2617)







● STARTLS extension, TLS

● Support for SMTPS

● POP/IMAP before SMTP

● ETRN (RFC 1985)

● ODMR (RFC 2645)

● RBL/ORBS support (rblsmtpd)

● DNSBL support using plugin

● SURBL (SURBL Blacklist) using surblfilter

● SPAM Control (Reject/Tag/Accept) using Bayesian techniques

● High Performance MS Virus Control via control file viruscheck andcontrol file signatures

● Content Filtering and blocking of prohibited attachments via control file bodycheck






● Ability to reject/bounce mails for unknown/inactive users (CHECKRECIPIENT)

● ability to have the RECIPIENT check for selective domains using control file chkrcptdomains

● Antispoofing mode (turned on by environment variable ANTISPOOFING)

● Masquerading ability.

● Multiline greetings via control file smtpgreeting

● Message Submission Agent – MSA (RFC 2476)

● Domain IP address pair access control via control file hostaccess

● Per User accesslist via control file accesslist

● SPF – Sender Permitted From





● Per User control of environment variable by envrules(rules file set by environment variable FROMRULES)

● Greylisting[3] capability using qmail-greyd or greydaemon

● Bounce Address Tag Validation (BATV)[4]

● Notify recipient when message size exceeds databyte limits (by setting environment variable DATABYTES_NOTIFY)

● SMTP Plugins using external plugins in /var/indimail/plugins

– Queue management● instant handling of messages added to queue

● parallelism limit (control/concurrencyremote, control/concurrencylocal)

● split queue directory---no slowdown when queue gets big


http://en.wikipedia.org/wiki/Greylisting

http://en.wikipedia.org/wiki/User:Mbhangui/IndiMail#cite_note-2

http://en.wikipedia.org/wiki/Bounce_Address_Tag_Validation



Feature List (cont…)– Queue Management (cont…)

● quadratic retry schedule---old messages tried less often

● independent message retry schedules

● automatic safe queueing---no loss of mail if system crashes

● automatic per-recipient checkpointing

● automatic queue cleanups (qmail-clean)

● queue viewing (qmail-qread)

● detailed delivery statistics (qmailanalog)

● Ability to hold local, remote or both deliveries (holdlocal, holdremote control file)

● Qmail Queue Extra Header – Ability to pass extra headers to local and remote deliveries via qmail-queue (Environment variable QQEH).




● Configurable number of queues and time slicing algorithm for load balancing via qmail-multi. A queue in indimail is configurable by three environment variables QUEUE_BASE, QUEUE_COUNT, and QUEUE_START. A queue in IndiMail is a collection of queues.Each queue in the collection can have one or more SMTP listener but a single or no delivery (qmail-send) process. It is possible to have the entire queue collection without a delivery process (e.g. SMTP on port 366 – ODMR). The QUEUE_COUNT can be defined based onhow powerful your host is (IO bandwidth, etc).NOTE: This configurable number of queues is possibe with a single installation and does not require you to install multiple instances of qmail.




● External Virus scanning via QHPSI – Qmail High Performance Scanner Interface

● Ability to extend QHPSI interface through plugins. The keyword plugin:shared_libdefined in the environment variable QHPSI denotes 'shared_lib' to be loaded.

● Virus scanner qscanq. Ability to detect virus via a third party scanner defined by SCANCMD environment variable (clamscan, clamdscan, etc)

● Blocking of prohibited filename extensions via qscanq program

● Domainkeys (qmail-dk) RFC 4870 * DKIM[5] with ADSP/SSP (qmail-dkim) RFC 4871


http://en.wikipedia.org/wiki/DomainKeys


http://en.wikipedia.org/wiki/DomainKeys_Identified_Mail





● Set all header values listed in envheader control file as environment variables.

● Log all headers listed in control file logheaders to stderr.

● Remove all headers listed in control file removeheaders from email.

● Ability to do line processing instead of block processing.

● qmail-nullqueue – blackhole the mail silently.

● rule based mail archival using control file mailarchive (SOX, HIPAA compliance)

● Added additional recipients for a message using extraqueue or mailarchive control file.

● X-Originating-IP header to record the original IP from which the mail originates


http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act

http://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act


Feature List (cont…)– Bounces

● QSBMF bounce messages---both machine-readable and human-readable

● HCMSSC support---language-independent RFC 1893 error codes

● double bounces sent to postmaster

● Ability to discard double bounces

● Ability to preserve MIME format when bouncing.

● Control of bounce process via envrules (rules file controlled by environment

● variable BOUNCERULES or control files bounce.envrules)

● limit size of bounce using control file bouncemaxbytes

● Ability to process bounces using external bounce processor (environment variable BOUNCEPROCESSOR)




Feature List (cont…)– Routing by domain

● any number of names for local host (control/locals)

● any number of virtual domains (control/virtualdomains)

● domain wildcards (control/virtualdomains)

● configurable percent hack support (control/percenthack)

● Clustered Domain. Same virtual domain can exist on multiple hosts, each having its own set of users. Provides Load Balancing and infinite scalability.



Feature List (cont…)– Remote SMTP delivery

● RFC 2821, RFC 974, RFC 1123, RFC 1870

● 8-bit clean

● automatic downed host backoffs

● Configurable tcp timeouts for downed host backoffs.

● automatic switchover to next best MX

● artificial routing---smarthost, localnet, mailertable (control/smtproutes)

● Support for jumbo ISP (control/smtproutes.cdb)

● per-buffer timeouts

● passive SMTP queue---perfect for SLIP/PPP (serialmail)

● AutoTURN support (serialmail)

● Spam control (SPAMFILTER environment variable)







Feature List (cont…)– Remote SMTP delivery (cont…)

● Authenticated SMTP (userid/passwd in control/smtproutes) - PLAIN, LOGIN, CRAM-MD5, CRAM-SHA1, CRAM-RIPEMD, DIGEST-MD5

● STARTTLS, TLS

● Static and Dynamic Routing. (SMTPROUTES environment variable)

● Environment variable control via envrules (rules file controlled byenvironment variable RCPTRULES) * QMAILREMOTE environment variable to run any executable/script instead of qmail-remote

● QMTP support, artificial routing using (control/qmtproutes)

● ONSUCCESS_REMOTE, ONFAILURE_REMOTE scripts run on successful or failed remote deliveries environment variables SMTPTEXT, SMTPCODE, ERRTEXT available for these scripts

● IP address binding on domain, sender address, recipient address and random selection from a pool of IP addresses



Feature List (cont…)– Local delivery

● user-controlled address hierarchy : fred controls fred-anything

● mbox delivery

● reliable NFS delivery (maildir)

● user-controlled program delivery: procmail etc. (qmail-command)

● optional new-mail notification (qbiff)

● detailed Delivered-To Headers

● optional NRUDT return receipts (qreceipt)

● autoresponder RFC 3834 compliance (provide Auto-Submitted,In-Reply-To, References fields (RFC 3834))

● conditional filtering (condredirect, bouncesaying, vfilter)





Feature List (cont…)– Local delivery (cont…)

● Environment variable control via envrules (rules file controlled by environment variable RCPTRULES)

● Eliminate duplicate messages

● QMAILLOCAL environment variable to run any executable/script instead of qmail-local

● X-Forwarded-To, X-Forwarded-For headers

– Other● Unix Client Server Program Interface (UCSPI) through

programs tcpserver and tcpclient

● Change concurrency of tcpserver without restart


http://cr.yp.to/proto/ucspi.txt


Feature List (cont…)– Other

● TLS/SSL Support in tcpserver

● STARTTLS extension in IMAP, STLS extension in POP3

● Ability to restrict connection per IP (MAXPERIP)

● run shutdown script if present on svc –d

● ability to log svscan output using multilog

● nssd Name Service Switch which allows extending of the system passwd database to IndiMail's database.

● pam-multi - Generic PAM module allows any external programs to authenticate against IndiMails database.

● multiple checkpassword modules sys-checkpwd, ldap-checkpwd, pam-checkpwd, vchkpass, systpass

● Proxy for IMAP/POP3 Protocol


http://en.wikipedia.org/wiki/Name_Service_Switch

http://en.wikipedia.org/wiki/Pluggable_Authentication_Modules



● inlookup – High Performance User Lookup Daemon.

● indisrvr – Indimail Administration Daemon.● spawn-filter - Ability to add disclaimer, run multiple filters before

local/remote delivery.

● Post Execution Handle - Allows functionality of indimail to be extended by writing simple scripts

● On the fly migration of users by defining MIGRATEUSER environment variable.

● ready to use QMQP service

● ability to distribute QMQP traffic across multiple servers

● sslerator - TLS/SSL protocol wrapper for non-tls aware applications




● svctool – Configuration tool for IndiMail.

● adminclient protocol – Interface for external programs to administer IndiMail

● mrtg graphs for detailed statistics

● ability to specify commands in control files


IndiMail Architecture

Online Replication

Mailstoreimap/pop3/smtp

Inlookupvdelivermail

WebmailProxy

imap/pop3inlookup

WebmailProxy

imap/pop3inlookup

WebmailProxy

imap/pop3inlookup

IncomingRelayInlookupClamd

bogofilter


bogofilter

Internetmx.xxxx.com (25)

imap.xxxx.com (110, 143)mail.xxxx.com (80)

Load Balancer 1


bogofilter

Load Balancer 2

OutgoingRelayInlookup

clamd

OutgoingRelayInlookup

clamd Mailstoreimap/pop3/smtp


Mailstoreimap/pop3/smtp


InternetSMTPPort 25

SMTP

SMTP(for local domains)

MySQLHost Control

DB(Master)

MySQLHost Control

DB(Slave)

MySQL MySQLMySQL

SMTP(for authenticated

userto submit theiroutgoing email)

Internet SMTP

qmail-remote(external domain)

IMAP/POP3

MySQL Connection

MySQL ConnectionMySQL

Connection

smtp.xxxx.com (25/587)

imap.xxxx.com (110/143)mail.xxxx.com (80)

mx.xxxx.com (25)

MySQL Connection

SMTP(delivery from

mailstore)














Installation

– RPM for both 32 / 64 Bit

– rpm -ivh rpm_filename

– openSUSE

• openSUSE 11.3• openSUSE 11.2• openSUSE 11.1• SUSE Linux Enterprise 11 SP1• SUSE Linux Enterprise 11• SUSE Linux Enterprise 10• OpenSUSE Factory

– Red Hat

• Fedora14• Fedora13• Fedora12• RHEL6• RHEL5• Centos 5

– Mandriva Linux

• Mandriva 2010.1• Mandriva 2010

– Yum Repository

– http://download.opensuse.org/repositories/home:/indimail/

– Copy repo file to /etc/yum.repos.d for hands-free installation/upgrade

● % sudo yum install indimail.x86_64● % sudo yum install indimail.i386

●


IndiMailThe Flexible Messaging PlatformConfiguration

– svctool (service Tool)• SMTP, IMAP, POP3, ssl, fetchmail, certificates, verification of

installation

• Backups of data, configuration

• Repair of Database, queue

• Configuration for MySQL, Spamfilter, virus filter, qmail

• Reports

– GUI• Iwebadmin, indium, shit, cindimai, osh

– Supervise for 100% uptime• Envdir

• Logging


● Administration● Web Administration – iwebadmin

● ncurses GUI – Secure Host Interface

– Configurable menu & commands● Indium – TCL/TK GUI

– Powerful & provides in-depth administration● Operator Shell – osh

– Bash like shell with restricted access to commands

– Access to commands based on user roles● cindimail – Small shell providing access to all indimail commands

with tool tips




Maintenance– Backup of Configuration

● /var/indimail/control● /var/indimail/etc● /var/indimail/users● /var/indimail/share

– Mail Backup

– Update of Clamav Software

– Update of IndiMail (rpm, yum)



Long Term Goals●To provide a modern technologically superior technology and solutions specifically related to Messaging Protocols (SMTP, IMAP, POP3)

●Provide a highly scalable, bug free, MSP class mailing solution using the Bazaar Model under GNU GPL V3

●Provide configuration for any property that can potentially affect the behaviour of messaging (Install the software once).

●FHS 2.3 compliance

●Build a community and get shipped with Linux distros by default



Support● indimail-support - You can subscribe for Support at

● https://lists.sourceforge.net/lists/listinfo/indimail-support. You can email [email protected] for posting messages to this list.

● indimail-devel - You can subscribe at https://lists.sourceforge.net/lists/listinfo/indimail-devel

● indimail-announce - This is only meant for announcement of New Releases or patches. You can subscribe at http://groups.google.com/group/indimail. You can email [email protected] for posting messages to this list.

● Archive at Google - http://groups.google.com/group/indimail. This group acts as a remote archive. Any discussions posted here goes to indimail-support.

● There is also a Project Tracker for IndiMail (Bugs, Feature Requests, Patches, Support Requests) at http://sourceforge.net/tracker/?group_id=230686


https://lists.sourceforge.net/lists/listinfo/indimail-support

https://lists.sourceforge.net/lists/listinfo/indimail-devel

http://groups.google.com/group/indimail

http://groups.google.com/group/indimail

http://sourceforge.net/tracker/?group_id=230686


Documentation– INSTALL, INSTALL-RPM

– IndiMail Documentation – indimail.pdf

– WIKI - http://en.wikipedia.org/wiki/User:Mbhangui/IndiMail

– Frequently Answered Questions – FAQ.pdf

– Man Pages - /var/indimail/man

– All docs in /var/indimail/docs


http://en.wikipedia.org/wiki/User:Mbhangui/IndiMail


Hardware Requirements (100,000 users)– Messaging Server

• DL 360 (Relay Server) – 2 Nos

• DL 380 (Message Store) – 1 Nos

• Storage (MSA 1000/MSA 2000)

- Groupware & Chat• DL380 (Egroupware & Jabber) – 1 Nos

- Load Balancer – 1 Nos



Timelines– Messaging Server – 1 day installation + 4 weeks

training

– Groupware & chat - 6 weeks installation & configuration + 2 weeks training


Technology

IndiMail - The Flexible Messaging Platform