IMAP mail sync for the Frappe Framework
Mailsync keeps Frappe's Communications in step with your IMAP mailbox:
- One login per account. Every two minutes one background job per Email Account logs in once and runs the inbox pull, the Sent folder import and the status sync over the same connection.
- Sent folder import. Mails you send from any client (Apple Mail, Thunderbird, your phone) show up as Sent Communications. Mails that Frappe sent itself or already knows are skipped.
- Status from the mailbox. A received mail becomes Replied when a reply sits in the Sent folder, and Closed once it is moved out of its folder (archived or deleted).
- Import of past mail. A one-off import pulls old mail from chosen folders of any IMAP mailbox. You decide first which unknown addresses become contacts, and only mails with a known contact are imported.
Mailsync only reads the mailbox. It never moves, flags or deletes mail and never sends email.
bench get-app https://github.com/EffizienzAgent/mailsync --branch version-16
bench --site <site> install-app mailsyncCompatibility: Frappe v16. The version-16 branch always points to the latest release; tagged releases
and the changelog are on the releases page.
Set up an Email Account with IMAP and Enable Incoming as usual. Mailsync picks it up on the next run.
mailsync.mail_sync.sync_all runs every two minutes. For each incoming account it enqueues one job on the default queue with a 600 second timeout. The job name is Frappe's own pull_from_email_account|<account>, so Mailsync and Frappe's stock pull never run at the same time for one account. IMAP accounts get mailsync.mail_sync.sync_account. POP3 and Frappe Mail accounts get Frappe's stock pull.
sync_account logs in once and hands the open connection to three steps. A failing step is logged and the other steps still run. If imap_timeout is not set in the site config, it is set to 60 seconds for the job.
- Inbox pull. Frappe's own
EmailAccount.receive, reusing the connection. - Sent folder. The folder is found by the
\Sentflag, then by the account's Sent Folder Name, then by common names (Sent,INBOX.Sent,Sent Items,Gesendet, ...). On the first run Mailsync only remembers the current position and imports nothing older. After that it imports up to 50 new mails and up to 100 MB per run. Mails over 25 MB are skipped and logged. A mail is skipped when its Message-ID is already a Communication, is in the Email Queue or was generated by the site. If the folder's UIDVALIDITY changes, Mailsync starts again from the current position. The position is stored as a default namedmailsync_sent_folder:<account>. - Status. Open and Linked received mails from the last 14 days are checked. Mails whose Message-ID has characters outside printable ASCII, spaces, quotes, backslashes or angle brackets are left alone. If the server rejects a search, only the mails in that search keep their status. A reply in the Sent folder (by
In-Reply-ToorReferences) sets Replied. A mail that is no longer in its IMAP folder is set to Closed. Closed mails are never reopened.
Frappe sets only In-Reply-To on outgoing replies. Mailsync adds a References header as RFC 5322 asks, so mail clients keep the whole thread. It finds the Communication with the parent's Message-ID and follows its In Reply To links up to the first mail of the thread. The header lists the Message-IDs from the oldest to the parent. Long threads keep the first mail and the 19 newest, and the header line stays within 998 characters, because Frappe writes headers without folding. Message-IDs that are not plain printable ASCII without spaces or angle brackets are skipped; if In-Reply-To holds several ids, the last one is the parent. Message-IDs that Frappe never stored, for example the References of a received mail, are not known and are left out. If anything goes wrong, the error is logged and the mail is sent without References.
A Mail Import pulls old mail from chosen folders of an IMAP Email Account into Communications, once. It is meant for history that sits in other mailboxes, for example customer folders in a personal mailbox.
Account. Use a normal Email Account. For a mailbox that should only be imported from, turn Enable Incoming off: Frappe and Mailsync only poll accounts with Enable Incoming on, but host, login and password stay stored. Frappe hides the IMAP settings while Enable Incoming is off, so set the account up with it on (Frappe checks the connection), then turn it off. Accounts that are polled can be imported from too.
Who. System Managers and the users who have the Email Account in their User Emails (User settings) can create, see and run imports for it.
Steps.
- Create a Mail Import, pick the Email Account and optionally a date range (by the mail's
Dateheader, both days included). Choose Folders lists the folders on the server, subfolders included. - Scan reads the headers of the chosen folders in a background job. Mails over 25 MB are counted as Too Large. Mails whose Message-ID is already a Communication, is in the Email Queue or was generated by the site count as Already Present.
- Review Addresses lists the addresses in the scanned mails that are no Contact yet. For each: Create Contact (name from the mail header, optionally linked to a company), Add to Contact or Ignore. Ignored addresses stay ignored in all later imports; this does not affect the regular mail sync. Own addresses (all Email Accounts and enabled system users) are never listed.
- Import imports every mail in which at least one Contact takes part, as a Closed and seen Communication: Sent when it comes from an own address, otherwise Received. Frappe links the Contacts and their links (for example their company) as usual. Mails whose other participants are all ignored are Filtered Out. Mails with only undecided addresses wait: decide and run Import again. Links are set once, when a mail is imported.
The import never creates Contacts on its own, even when the Email Account has Create Contact on. Running Scan again picks up new mail; running Import again or importing the same folders in a new import never creates duplicates. Mails without a Message-ID are recognized by account, folder and UID.
Scan and import run on the long queue in jobs of about 20 minutes and continue in a new job until they are done. The form shows the counters and refreshes when a job finishes. If a job dies, start the step again after 25 minutes; it continues where it stopped. If a folder's UIDVALIDITY changes on the server, its mails are not imported and the import asks for a new scan.
While the import inserts a Communication and then sets it to Closed, frappe.flags.mailsync_import is True, so other apps can skip notifications or other work for old mail.
Python's imaplib and smtplib do not check the server certificate unless they are given an SSL context, and Frappe passes none for IMAP STARTTLS, SMTP STARTTLS and SMTP over SSL. Mailsync fixes this in every request and background job, including Frappe's own inbox pull and the outgoing email queue. These connections now check the certificate and the host name against the system's trusted CAs. IMAP over SSL was already verified by Frappe.
For a test server with a self-signed certificate, you can turn the check off for one site:
bench --site test.site set-config mailsync_allow_self_signed_starttls 1 # incoming IMAP
bench --site test.site set-config mailsync_allow_self_signed_smtp 1 # outgoing SMTPDo not set these in production.
Other apps can extend Mailsync in their hooks.py.
Hook arguments come from mail headers and the IMAP server, so treat them as untrusted strings. Use them in database filters and escape them before rendering them as HTML. Never pass them to a shell, eval or raw SQL.
Adds timeline links to imported Sent mails. Called once per recipient in To and Cc, except the account's own address and system users. For mails from a Mail Import it is called once per participant in From, To and Cc, except own addresses. Return a list of links.
mailsync_recipient_links = ["myapp.mail.recipient_links"]
def recipient_links(address: str) -> list[dict]:
customer = frappe.db.get_value("Customer", {"email_id": address})
return [{"link_doctype": "Customer", "link_name": customer}] if customer else []Frappe still links known Contacts on its own.
Called after the status sync changed Communications, with a dict of Communication name to the new status ("Replied" or "Closed").
mailsync_status_changed = ["myapp.mail.status_changed"]
def status_changed(changes: dict[str, str]) -> None:
frappe.publish_realtime("inbox_changed", after_commit=True)Called for every mail Mailsync reads from the mailbox, from the Sent folder and from a Mail Import, with the original bytes of the mail. Use it to archive the original .eml.
mailsync_mail_read = ["myapp.mail.mail_read"]
def mail_read(communication, raw: bytes, account: str, folder: str, sent_or_received: str, **kwargs) -> None:
archive(communication.name, raw)communication: the Communication document. For a newly imported mail it is called after the Communication is inserted. For a mail that Frappe already knows (its Message-ID belongs to the site or to an existing Communication), it gets the existing Sent Communication with that Message-ID. Mails that are only in the Email Queue have no Communication and are not passed on.raw: the mail as read from the server. Mails over 25 MB are skipped and never passed on.account: the name of the Email Account.folder: the IMAP folder the mail was read from.sent_or_received:"Sent"or"Received", as on Communication. Always"Sent"for the Sent folder.
A Mail Import calls it only for newly imported mails, after the Communication is inserted.
Each mail is passed on once, when Mailsync first reads its UID. Mails that are already in the folder on the first run are never passed on; a mail that is moved or copied into the folder later gets a new UID and is passed on then, whatever its date. The same mail is passed on again only when it lands in the folder a second time or when two Email Accounts read the same mailbox, so make the method idempotent. Every call runs in its own savepoint: if the method raises, its database writes are rolled back, the error is logged and the sync goes on.
Hooks run synchronously inside the sync job, which has a 600 second timeout for the whole account. Keep them fast and enqueue heavy work, such as uploads, as a job of your own. Do not call frappe.db.commit() in a hook: Mailsync commits once per step, and a commit ends the savepoint. If a hook commits and then raises, Mailsync rolls back the open transaction, logs the error and goes on with the batch.
raw is the complete mail, including the Bcc header and all attachments. Treat it as sensitive: restrict who can read what you store and never log it.
Tells the import which addresses are your own, in addition to the Email Accounts and enabled system users. Own addresses are never listed for review, never decide whether a mail is imported, and make a mail Sent when they are the sender. Called with a lowercased address; return True for own addresses.
mailsync_own_address = ["myapp.mail.is_own"]
def is_own(address: str) -> bool:
return address.endswith("@mycompany.example")A plain value, not a method: the DocType a new Contact from the review is linked to as its company. The review then shows that DocType in the company field, and Create new in that field creates one. Without it, the user picks the DocType.
mailsync_contact_link_doctype = "Customer"Prefills the review for an unknown address. Called with a lowercased address; return a dict with contact (an existing Contact for Add to Contact) and/or link_name (a record of mailsync_contact_link_doctype). Results of several apps are merged; a failing method is logged and skipped.
mailsync_address_suggestion = ["myapp.mail.suggest"]
def suggest(address: str) -> dict:
domain = address.split("@")[-1]
customer = frappe.db.get_value("Customer", {"website": ("like", f"%{domain}%")})
return {"link_name": customer} if customer else {}Tests run on a bench with a test site:
bench --site test.site set-config allow_tests true
bench --site test.site run-tests --app mailsyncThe tests in test_imap_server.py talk to a real IMAP server. They run when MAILSYNC_TEST_IMAP_HOST (and optionally MAILSYNC_TEST_IMAP_PORT, default 3143) is set. CI uses GreenMail:
docker run -d -p 3143:3143 -e GREENMAIL_OPTS="-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.auth.disabled" greenmail/standalone:2.1.5
MAILSYNC_TEST_IMAP_HOST=127.0.0.1 bench --site test.site run-tests --app mailsyncThe tests in test_tls.py and test_smtp_tls.py need IMAP and SMTP servers with a self-signed certificate. They run when MAILSYNC_TEST_STARTTLS_HOST (IMAP STARTTLS on port 31143) and MAILSYNC_TEST_SMTP_HOST (SMTP STARTTLS on 31587, SMTP over SSL on 31465) are set. CI uses Dovecot:
docker run -d -p 31143:31143 -p 31587:31587 -p 31465:31465 -e USER_PASSWORD=secret dovecot/dovecot:2.4.1
MAILSYNC_TEST_STARTTLS_HOST=127.0.0.1 MAILSYNC_TEST_SMTP_HOST=127.0.0.1 bench --site test.site run-tests --app mailsyncTo check that a CA-signed server still connects, set MAILSYNC_TEST_CA_SMTP_HOST to a public SMTP host. This test needs network access and does not run in CI.
Need help setting this up or integrating it into your Frappe stack? Write to oss@effizienzagent.de (English or German). German-speaking SMEs: see mittelstandshub.de.
MIT, see license.txt.