mRpostman

An IMAP Client for R

Overview

mRpostman is a session-based IMAP client that implements the full functionality of the IMAP4rev1 protocol (RFC 3501), allowing you to perform virtually all e-mail operations from within R. The aim of this package is to pave the way for email data analysis in R. To do so, mRpostman makes extensive use of the {curl} package and the libcurl C library.

mRpostman’s official website: https://allanvc.github.io/mRpostman/

Cite mRpostman: A. V. C. Quadros, “mRpostman: An IMAP Client for R”, Journal of Open Research Software, vol. 12, no. 1, p. 4, 2024, doi: 10.5334/jors.480. http. Refer to citation("mRpostman").

IMPORTANT:

  1. Old versions of the libcurl C library ({curl}’s main engine) will cause the malfunction of this package. If your libcurl’s version is above 7.58.0, you should be fine. In case you intend to use OAuth 2.0 authentication, then you will need libcurl >= 7.65.0. To learn more about the OAuth 2.0 authentication in this package, refer to the “Using IMAP OAuth2.0 authentication in mRpostman” vignette.

  2. Most mail providers discontinued less secure apps access. If it is still available and you are comfortable with this type of access you can enable this option for your account on your mail provider. Some providers, such as Yahoo Mail, also offer the option to generate a password to be used by third-party apps such as mRpostman. The other option, as mentioned above, is to set up OAuth2 (two-factor authentication) in order to access your mailbox. Please also refer to the “Using IMAP OAuth2.0 authentication in mRpostman” vignette.

Providers and their IMAP urls

Provider IMAP Server
Gmail imap.gmail.com
Office 365 outlook.office365.com*
Outlook.com (Hotmail and Live.com) imap-mail.outlook.com
Yahoo Mail imap.mail.yahoo.com
iCloud Mail imap.mail.me.com
AOL Mail imap.aol.com
Zoho Mail imap.zoho.com
Yandex Mail imap.yandex.com
GMX Mail imap.gmx.com
Mail.com imap.mail.com
FastMail imap.fastmail.com

* For Office 365 accounts, the username should be set as user@yourcompany.com or user@youruniversity.edu for example.

Introduction

From version 0.9.0.0 onward, mRpostman is implemented under the OO paradigm, based on an R6 class called ImapCon. Its derived methods, and a few independent functions enable the R user to perform a myriad of IMAP commands.

Below, we present all the available methods and functions, grouped by type of operation:

  • configuration and connection methods: configure_imap(), disconnect(), noop(), reset_url(), reset_username(), reset_password(), reset_verbose(), reset_use_ssl(), reset_buffersize(), reset_timeout_ms(), reset_xoauth2_bearer();
  • server information methods: list_server_capabilities(), id(), namespace(), get_quota_root(), get_quota();
  • mailbox operations methods: list_mail_folders(), list_subscribed_folders(), list_special_use_folders(), select_folder(), examine_folder(), status(), create_folder(), rename_folder(), delete_folder(), subscribe_folder(), unsubscribe_folder(), close_folder(), unselect_folder(), list_flags();
  • single-search methods: search_before(), search_since(), search_period(), search_on(), search_sent_before(),search_sent_since(), search_sent_period(), search_sent_on(), search_string(), search_flag(), search_smaller_than(), search_larger_than(), search_younger_than(), search_older_than();
  • the custom-search method and its helper functions: search();
  • server-side sort and thread methods: sort(), thread();
  • fetch methods: fetch_body(), fetch_header(), fetch_text(), fetch_metadata(), metadata_options(), fetch_attachments_list(), fetch_attachments();
  • attachments methods: list_attachments(), get_attachments(), fetch_attachments_list(), fetch_attachments();
  • complementary methods: copy_msg(), move_msg(), append_msg(), esearch_min_id(), esearch_max_id(), esearch_count(), delete_msg(), expunge(), add_flags(), remove_flags(), replace_flags();
  • MIME-decoding and message-text helper functions: decode_mime_header(), clean_msg_text().

Supported IMAP commands and capabilities

The IMAP protocol has a mandatory core — the IMAP4rev1 commands defined in RFC 3501, which every compliant server must implement — plus a set of optional extensions, each advertised by the server in its CAPABILITY response. mRpostman covers both. For the extension-based methods, mRpostman checks the server’s advertised capabilities and, if the required one is missing, raises an informative error instead of letting the server reply with a cryptic BAD Unknown command. You can inspect what your server supports with list_server_capabilities().

Core commands (RFC 3501 — always available)

IMAP command mRpostman method(s)
CAPABILITY list_server_capabilities()
NOOP noop()
LOGIN / AUTHENTICATE configure_imap()
LOGOUT disconnect()
SELECT / EXAMINE select_folder() / examine_folder()
CREATE / DELETE / RENAME create_folder() / delete_folder() / rename_folder()
SUBSCRIBE / UNSUBSCRIBE subscribe_folder() / unsubscribe_folder()
LIST / LSUB list_mail_folders() / list_subscribed_folders()
STATUS status()
APPEND append_msg()
SEARCH search(), search_before(), search_since(), search_string(), … (all search_*)
FETCH fetch_body(), fetch_header(), fetch_text(), fetch_metadata(), fetch_attachments()
STORE add_flags(), remove_flags(), replace_flags()
COPY copy_msg()
CLOSE close_folder()
EXPUNGE expunge(), delete_msg()

Optional extensions (server-dependent — capability-checked)

IMAP command mRpostman method(s) Capability RFC
SORT sort() SORT 5256
THREAD thread() THREAD=REFERENCES / THREAD=ORDEREDSUBJECT 5256
GETQUOTA / GETQUOTAROOT get_quota() / get_quota_root() QUOTA 2087
NAMESPACE namespace() NAMESPACE 2342
ID id() ID 2971
UNSELECT unselect_folder() UNSELECT 3691
LIST (special-use) list_special_use_folders() SPECIAL-USE 6154
MOVE move_msg() MOVE 6851
SEARCH RETURN (ESEARCH) search(esearch = TRUE), esearch_count(), esearch_min_id(), esearch_max_id() ESEARCH 4731

Availability varies by provider. Gmail, for instance, supports every extension above except SORT and THREAD, which it has never implemented; to exercise sort() and thread() you need a server that advertises them (e.g. Dovecot-based hosts, Yandex, or Outlook/Office 365). Two extensions remain intentionally unimplemented: CONDSTORE/QRESYNC (out of scope) and IDLE (not feasible with libcurl’s one-shot request model — see the Basics vignette).

Installation

# CRAN version
install.packages("mRpostman")

# Dev version
if (!require('remotes')) install.packages('remotes')
remotes::install_github("allanvc/mRpostman")

Trying it without a mail account

The package ships a disposable local IMAP server (Dovecot, in a Docker container) plus a deterministic synthetic corpus generator, so every feature can be exercised offline and reproducibly — no credentials, OAuth2 setup, or provider rate limits involved:

# after starting the container (see the "sandbox" vignette):
con <- configure_imap(url = "imap://localhost:1430", username = "testuser",
                      password = "sandbox", use_ssl = FALSE)
populate_sandbox(con, n = 200) # uploads the corpus with the package's own APPEND

Real data works too: ingest_maildir() uploads any local maildir-style directory to the server via APPEND, and enron_sandbox() builds on it to download (once, with consent, cached) the public Enron corpus and ingest a subset selected by custodian, folder, and date — turning the sandbox into a full e-mail data-analysis laboratory.

See the “A reproducible IMAP sandbox with Docker” vignette for the guided tour. The sandbox’s Dovecot server also advertises SORT and THREAD, making it a convenient place to try the extensions your provider may lack.

Basic Usage

1) Configure an IMAP connection and list the server’s capabilities


library(mRpostman)

# Outlook - Office 365
con <- configure_imap(url="imaps://outlook.office365.com",
                      username="your_user@company.com",
                      password=rstudioapi::askForPassword()
)

# other IMAP providers that were tested: Hotmail ("imaps://imap-mail.outlook.com"),
#  Gmail (imaps://imap.gmail.com), Yahoo (imaps://imap.mail.yahoo.com/), 
#  AOL (imaps://export.imap.aol.com/), Yandex (imaps://imap.yandex.com)

# Other non-tested mail providers should work as well

con$list_server_capabilities()

2) List mail folders and select “INBOX”


# Listing
con$list_mail_folders()

# Selecting
con$select_folder(name = "INBOX")

3) Search messages by date


res1 <- con$search_on(date_char = "02-Jan-2020")

res1

4) Customizing a search with multiple criteria

Executing a search by string:


# messages that contain either "@k-state.edu" OR "ksu.edu" in the "TO" header field
res2 <- con$search(OR(
  string(expr = "@k-state.edu", where = "TO"),
  string(expr = "@ksu.edu", where = "TO")
))

res2

5) Fetch messages’ text using single-search results


res3 <- con$search_string(expr = "Welcome!", where = "SUBJECT") %>%
  con$fetch_text(write_to_disk = TRUE) # also writes results to disk

res3

6) Attachments

You can list the attachments of one or more messages with:

1) the list_attachments() function:


con$search_since(date_char = "02-Jan-2020") %>%
  con$fetch_text() %>% # or with fetch_body()
  list_attachments() # does not depend on the 'con' object

… or more directly with:

2) fetch_attachments_list()


con$search_since(date_char = "02-Jan-2020") %>%
  con$fetch_attachments_list()

If you want to download the attachments of one or more messages, there are also two ways of doing that.

1) Using the get_attachments() method:


con$search_since(date_char = "02-Jan-2020") %>%
  con$fetch_text() %>% # or with fetch_body()
  con$get_attachments()

… and more directly with the

2) fetch_attachments() method:


con$search_since(date_char = "02-Jan-2020") %>%
  con$fetch_attachments()

Future Improvements

  • add further IMAP features;
  • eliminate the {stringr} dependency in REGEX;
  • implement a progress bar in fetch operations;

Known bugs

  • search results truncation: This is a libcurl’s known bug which causes the search results to be truncated when there is a large number of message ids returned. To circumvent this problem, you can set a higher buffersize value, increasing the buffer capacity, and verbose = TRUE for monitoring the server response for truncated results when executing a search. When possible, mRpostman tries to issue a warning for possible truncated values.

  • verbose = TRUE malfunction on Windows: This seems to be related to the {curl} R package. When using the verbose = TRUE on Windows, the flow of information between the IMAP server and the R session presents an intermittent behavior, which causes it to not be shown on the console, or with a considerable delay.

  • shared mailbox access not working: This seems to be another libcurl’s bug, although more tests need to be done to confirm it. It does not allow the user to connect to a shared mailbox. To circumvent this, if the shared mailbox has a password associated with it, you can try a direct regular connection.

  • xoauth2_bearer SASL error: This is related to old libcurl’s versions which causes the access token to not be properly passed to the server. This bug was fixed in libcurl 7.65.0. The problem is that many Linux distributions, such as Ubuntu 18.04, still provide libcurl 7.58.0 in their official distribution (libcurl4-openssl-dev). If you use a newer Linux distro such as Ubuntu 20.04, you should be fine as the distributed libcurl’s version will be above 7.65.0. Another alternative is to use plain authentication instead of OAuth2.0.

License

This package is licensed under the terms of the GPL-3 License.

References

Crispin, M. (2003), INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1, RFC 3501, March 2003, http.

Heinlein, P. and Hartleben, P. (2008). The Book of IMAP: Building a Mail Server with Courier and Cyrus. No Starch Press. ISBN 978-1-59327-177-0.

Ooms, J. (2020), curl: A Modern and Flexible Web Client for R. R package version 4.3, http.

Quadros, A. V. C. mRpostman: An IMAP Client for R, Journal of Open Research Software, vol. 12, no. 1, p. 4, 2024, doi: 10.5334/jors.480. http.

Stenberg, D. Libcurl - The Multiprotocol File Transfer Library, http.