A new version of my R package mRpostman landed on CRAN. It is the most important update to the package since I rebuilt it around an R6 object-oriented structure back in September 2020. This version incorporates several intermediate updates that were only available on GitHub in the previous weeks.

Some context

I released mRpostman in 2019, and of all the packages I have published, it was the one that caught the most attention from the R community (it even got a retweet from Hadley Wickham at the time). The reason is simple: e-mail is a rich source of data that usually sits out of reach, behind IMAP servers. mRpostman turns MIME content into plain text you can actually analyze.

What is new

The package now implements the full functionality of the IMAP4rev1 protocol from within R, including commands that were missing until recently, such as APPEND, STATUS and CLOSE, plus a set of capability-checked extensions such as SORT, THREAD, MOVE and QUOTA.

Decoding grew up with it: message headers and bodies now honor the character set declared in the message, so mail in Russian, Greek, Japanese, Chinese or any other charset that iconv supports comes out as readable text instead of gibberish. Searching accepts non-ASCII terms as well, sent to the server as UTF-8.

Reproducibility

Reproducibility was another thing that bothered me all these years: you needed a real mail account to try any of it, and no two mailboxes are alike. This release fixes that. The package now ships a small disposable IMAP server (Dovecot, via Docker) plus a synthetic corpus generator with known properties: dates, sizes, encodings, attachments, reply chains. populate_sandbox() loads it all through the package’s own APPEND implementation, so you can test and learn every feature locally, against exact ground truth. There is a new vignette to help users with that.

And since the sandbox can hold real data too, 1.4.0 adds enron_sandbox(), which downloads the public Enron corpus from 2001, filters by custodian and date, and ingests it into the local Dovecot server. From there, you can score sentiment over time, map who was writing to whom, or run whatever text mining you have in mind. Underneath it sits ingest_maildir(), a general function that uploads any maildir archive to any IMAP folder.

install.packages("mRpostman")

• Official website: https://allanvc.github.io/mRpostman
• Sandbox vignette: https://allanvc.github.io/mRpostman/articles/sandbox.html
• CRAN version: https://cran.r-project.org/package=mRpostman
• Dev version: https://github.com/allanvc/mRpostman

I am happy to share treeSS, a new R package now available on CRAN.

It helps you detect a special kind of cluster in your data: spatial clusters, meaning groups of nearby areas where something happens more often than expected. But it does one extra thing. It looks at where something happens (space) and what kind of thing happens (a category in a tree/hierarchy) at the same time. This is an advance over spatial-only or tree-only algorithms, and it does so without inflating the family-wise error rate.

For example, you can use it to find which specific cause of death, type of crime, accident, or condition is unusually common in a group of nearby areas.

The method, called the tree-spatial scan statistic, comes from our paper:

Cançado, A. L. F., Oliveira, G. S., Quadros, A. V. C., & Duczmal, L. H. (2025). A tree-spatial scan statistic. Environmental and Ecological Statistics, 32(3), 953-978. https://doi.org/10.1007/s10651-025-00670-w

Try it in R:

install.packages("treeSS")

A Python version is coming soon to PyPI.

• CRAN version: https://cran.r-project.org/package=treeSS
• Dev version: https://github.com/allanvc/treeSS
• Article: https://doi.org/10.1007/s10651-025-00670-w

I am excited to announce the publication of my recent paper in the Journal of Open Research Software, titled “mRpostman: An IMAP Client for R”. This work introduces a package designed to facilitate the retrieval of email data directly within R.

Quadros, A. V. C. (2024). mRpostman: An IMAP Client for R. Journal of Open Research Software, 12(1). https://doi.org/10.5334/jors.480

I acknowledge my potential bias, but I believe mRpostman stands out for its user-friendly interface and implementation using an elegant object-oriented (OO) approach.

Here are some of the applications in which mRpostman can be particularly helpful:

Email content analysis. With mRpostman, researchers and analysts can dive deep into email content to uncover trends, patterns, and themes. This capability is invaluable in organizational contexts, enabling a deeper understanding of communication flows, for example. The package is the first step for advanced text mining techniques for sentiment analysis, keyword discovery, and thematic exploration using email data.

Attachment analysis. mRpostman simplifies the process of downloading and analyzing email attachments. Whether it is automating attachment fetching, extracting text from PDFs, scrutinizing images, or leveraging spreadsheet data in R for comprehensive statistical analysis, mRpostman provides the necessary functionality to expand the analysis beyond the email text.

Network analysis. By leveraging email metadata, such as sender, recipient, and CC details, mRpostman facilitates the construction of intricate communication networks. This opens the door to sophisticated social network analysis, offering insights into the dynamics of communication within organizations or social groups.

Spam detection and filtering. mRpostman serves as a foundational tool for developing advanced spam detection algorithms. By enabling easy access to extensive email datasets, it aids in training machine learning models to distinguish between spam and legitimate messages efficiently.

Time series analysis. Using the temporal data available in email metadata, mRpostman enables users to perform time series analysis. This can reveal communication patterns, peak activity times, and email flow trends, offering valuable insights into organizational or group communication behaviors.

• Article: https://doi.org/10.5334/jors.480
• Official website: https://allanvc.github.io/mRpostman
• CRAN version: https://cran.r-project.org/package=mRpostman

Our paper “ROC App: an application to understand ROC curves”, with Georges von Borries (University of Brasilia), was published in the Brazilian Journal of Biometrics.

von Borries, G. F., & Quadros, A. V. C. (2022). ROC App: an application to understand ROC curves. Brazilian Journal of Biometrics, 40(2). https://doi.org/10.28951/bjb.v40i2.566

What the paper is about

The Receiver Operating Characteristic (ROC) curve is one of those concepts that every student of statistics, data science or medicine meets early on, and that many keep finding confusing: what exactly moves along the curve, how it relates to sensitivity and specificity, and why a threshold that looks good for one population is bad for another.

The ROC App is a Shiny application built to make these ideas concrete. Using a diagnostic test as the motivating scenario, the user controls the parameters of the healthy and diseased populations (their means, variances and the cutoff of the test) and watches, in real time, how the ROC curve and the accuracy rates (sensitivity, specificity, false positive and false negative rates) respond to each change. The paper describes the underlying concepts, the app’s functionalities and how it can be used in the classroom.

The app is freely available online and runs in the browser, so it can be used in class without any installation.

Abstract

We present a software application to help students understand the Receiver Operating Characteristic (ROC) curve and other concepts associated with binary classification models. We use the diagnostic test scenario as a motivation to explain the underlying concepts and the app functionalities. The ROC App enables students to interactively learn why/how the ROC curve closely relates to the accuracy rates, by seeing how these curves and rates respond to modifications on the population’s parameters.

• ROC App: https://gfvonborries.shinyapps.io/roc_app/
• Article: https://doi.org/10.28951/bjb.v40i2.566
• Full list of publications: /publications/

mRpostman is an R package to help you to easy connect to your IMAP (Internet Message Access Protocol) server and execute commands, such as listing mailboxes, fetching and searching for messages in a tidy way. It heavily relies on {curl} for issuing the IMAP commands.

So far, mRpostman have been tested with Gmail, Yahoo Mail and AOL Mail, but it should also work with other mail providers. I would be happy to hear other successful experiences from users.

ATTENTION: Before you start, you have to enable “less secure apps access” in your mail account settings.

There is a detailed vignette HERE showing how to use the package!

That’s all!

• Official website: https://allanvc.github.io/mRpostman
• CRAN version: https://cran.r-project.org/package=mRpostman
• Dev version: https://github.com/allanvc/mRpostman

I am proud to announce emstreeR - my first R package available on CRAN.

emstreeR is a package for fast and easily computing Euclidean Minimum Spanning Trees (EMST). It heavily relies on RcppMLPACK and Rcpp to work as wrapper to the EMST Dual-Tree Boruvka algorithm (March, Ram, Gray, 2010)^[March, W. B., and Ram, P., and Gray, A. G. (2010). Fast euclidean minimum spanning tree: algorithm analysis, and applications. 16th ACM SIGKDD International Conference on Knowledge Discovery and Data mining, July 25-28 2010. Washington, DC, USA. doi:10.1145/1835804.1835882.] implemented in ‘mlpack’ - the C++ Machine Learning library (Curtin et al., 2013)^[Curtin, R. R. et al. (2013). Mlpack: A scalable C++ machine learning library. Journal of Machine Learning Research, v. 14, 2013.]. With emstreeR, R users have access to the C++ fast EMST algorithm without having to deal with the R-Rcpp-C++ integration. The package also provides functions and an S3 method for readily plotting the Minimum Spanning Trees (MST) using either {base}, {scatterplot3d} or {ggplot2} style.

Working with emstreeR is easy because you just have to pass a matrix or data.frame of points as an argument to the main function computeMST() in order to get things working.

But, before we begin, what exactly is a Minimum Spanning Tree?

I will try to explain that without bringing any graph concept to the discussion.

Imagine you have to connect a group of South American cities with pipelines in order to distribute the natural gas extracted from an offshore platform on the coast of Rio de Janeiro. To optimize your operations, you need to know the most efficient route, i.e. the route that connects all the cities within the minimum possible distance. That optimal route among the cities is a Minimum Spanning Tree. Therefore, the MST problem is, in its essence, an optimization problem.

The distances between two points can be calculated using various methods, such as Manhattan, Euclidean, Minkowski, etc. This package uses Euclidean distances. Hence, the name Euclidean Minimum Spanning Trees.

So, let’s try to reproduce the aforementioned example.

First, we are going to use the {ggmap} package . We will pass the cities names as arguments to ggmap::geocode() function in order to get their latitude and longitude coordinates. For this, we need the Geocoding Google API. For more instructions on how to enable the API and how to obtain a API key, please refer to the documentation of the ggmap::register_goole() function.

library(ggmap)

## cities:
cities_location <- data.frame(location = c("Passo Fundo, Brazil", 
                                           "Buenos Aires, Argentina",
                                           "Assuncion, Paraguay",
                                           "Campinas, Brazil",
                                           "Cubatao, Brazil",
                                           "Mendoza, Argentina",
                                           "Corrientes, Argentina",
                                           "Porto Velho, Brazil",
                                           "Manaus, Brazil",
                                           "Santa Cruz de La Sierra, Bolivia",
                                           "Belo Horizonte, Brazil",
                                           "Brasília, Brazil",
                                           "Rio de Janeiro, Brazil",
                                           "Campo Grande, Brazil",
                                           "Recife, Brazil"), 
                              stringsAsFactors = FALSE)

## Warning in readLines(con <- file("../../../API_key.txt")): cannot open file
## '../../../API_key.txt': No such file or directory

## Error in `readLines()`:
## ! cannot open the connection

## Error:
## ! object 'API_key' not found

## register your Google API
register_google(key = rstudio::"AIza...8RLaPBOQ")

## get latitude and longitude
geo_location <- geocode(cities_location$location)

## Error in `geocode()`:
## ! Google now requires an API key; see `ggmap::register_google()`.

## combine data:
df_location <- cbind(cities_location, geo_location)

## Error:
## ! object 'geo_location' not found

Let’s plot the map, so we can have an initial idea about what we are dealing with.

# coordinates for South America
map_grid <- c(left = -80, bottom = -40, right = -20, top = 0) 

library(dplyr) # we wanna use pipes

get_stamenmap(map_grid, zoom = 5) %>% ggmap()+
  geom_point(data = df_location,
           aes(x = lon, y = lat), size=2)

## Error in `get_stamenmap()`:
## ! Stamen map tiles are now hosted by Stadia Maps, use `get_stadiamap()`.

Now is when emstreeR comes in handy. We pass the point columns as argument to computeMST() and it calculates the minimum route connecting all the points, i.e. the Euclidean Minimum Spanning Tree.

## MST:
library(emstreeR)
out <- ComputeMST(df_location[,2:3])

## Error:
## ! object 'df_location' not found

out

## Error:
## ! object 'out' not found

As a result, we have a data.frame containing our original data and three more columns: from, to, and distance. This information is the Minimum Spanning Tree and will be used for plotting.

You can silence the output of the algorithm by setting verbose=FALSE in computeMST().

To better visualize the result, we plot the map again, but now using the {ggplot2}’s extension provided in emstreeR: stat_MST().

## Plot:
# we use the same grid map    
get_stamenmap(map_grid, zoom = 5) %>% ggmap()+
  stat_MST(data = out,
           aes(x = lon, y = lat, from=from, to=to), 
           colour="red", linetype = 2)+
  geom_point(data = out, aes(x = lon, y = lat), size=3)

## Error in `get_stamenmap()`:
## ! Stamen map tiles are now hosted by Stadia Maps, use `get_stadiamap()`.

Once you have the information about the minimum route, it is easy to see that you will have to install gas distribution hubs in cities that will have more than two incoming or outgoing links, such as Assuncion in Paraguay. Based on that information or other criteria such as connectivity or even by cutting the largest edge, you can also define or separate regions of operations. What I mean by this is that you can also use Minimum Spanning Trees in clustering problems. Nonetheless, let’s save that topic for another post.

Below, I will present other things you can do with ‘emstreeR’.

‘emstreeR’ provides an S3 method for the {base} function plot(). It is easy to make simple 2D plots with that.

## artifical data for 2D plots:
set.seed(1984)
n <- 15
c1 <- data.frame(x = rnorm(n,-0.2, sd=0.2), y = rnorm(n,-2,sd=0.2))
c2 <- data.frame(x = rnorm(n,-1.1, sd=0.15), y = rnorm(n,-2,sd=0.3)) 
d <- rbind(c1, c2)
d <- as.data.frame(d)
  
## MST:
#library(emstreeR)
out <- ComputeMST(d, verbose = FALSE)  

## simple 2D plot:
plot(out, col.pts = "red", col.segts = "blue")

The stat_MST() extension also makes it easy plotting 2D plots, but using {ggplot2} as we showed before.

## 2D plot with ggplot2:
library(ggplot2)
ggplot(data = out, aes(x = x, y = y, from = from, to = to))+ 
  geom_point()+ 
  stat_MST(colour="red")

## Warning: The following aesthetics were dropped during statistical transformation: from
## and to.
## ℹ This can happen when ggplot fails to infer the correct grouping structure in
##   the data.
## ℹ Did you forget to specify a `group` aesthetic or to convert a numerical
##   variable into a factor?

If you want something fancier, you can use the geom=curve argument:

## 2D plot with ggplot2:
#library(ggplot2)
ggplot(data = out, aes(x = x, y = y, from=from, to=to))+ 
  geom_point()+ 
  stat_MST(geom="curve")

## Warning: The following aesthetics were dropped during statistical transformation: from
## and to.
## ℹ This can happen when ggplot fails to infer the correct grouping structure in
##   the data.
## ℹ Did you forget to specify a `group` aesthetic or to convert a numerical
##   variable into a factor?

You can also make 3D MST plots with emstreeR. It provides a wrapper for plotting 3D scatter plots, using the {scatterplot3d} package.

Let’s create 3D artificial data and compute the MST.

## artificial data for 3D plots:
n = 99
set.seed(1984)
d1<-matrix(rnorm(n,mean=-2,sd=.5), n/3, 3) # 3d
d2<-matrix(rnorm(n,mean=0,sd=.3), n/3, 3)
d3<-matrix(rnorm(n,mean=3,sd=.4), n/3, 3)
d<-rbind(d1,d2,d3) # just to show a matrix input
  
## MST:
library(emstreeR)
out <- ComputeMST(d, verbose = FALSE)

We can do a simple 3D plot using the function plotMST3D().

## simple 3D plot:
plotMST3D(out, xlab = "xaxis", main="A simple MST 3D plot")

But, again, if you like fancier plots, we can also make an interactive 3D MST plot using {plotly}:

# some adjustments needed
ord_id <- Reduce(rbind, rbind(t(out[, c("from", "to")]), NA))
ord_data <- out[ord_id, c("V1", "V2", "V3")]

library(dplyr) # we will use pipes
library(plotly)

plot_ly(showlegend = T) %>%
  add_markers(data=ord_data, x = ~V1, y = ~V2, z = ~V3,
              marker=list(opacity=0.5, size=2.5), name="points") %>%
  add_paths(data=ord_data, x = ~V1, y = ~V2, z = ~V3,
            color = I("red"), name="edges")

## Error in `loadNamespace()`:
## ! there is no package called 'webshot'

That’s all!

• CRAN version: https://cran.r-project.org/package=emstreeR
• Dev version: https://github.com/allanvc/emstreeR

Updated on 2026-08-25. This is an enhanced and substantially different version of the original article published in R-Bloggers in 2018.

This is an example of how to scrape Google News website with the rvest package.

First off, you should take a look at the Google News website HERE, which I reproduce below:

You may notice, on the right side of the page, that we are using Google Chrome dev-tools. This is necessary in order to identify the html nodes we need. You can access this tool by hitting the F12 key. The html nodes are passed as arguments to the rvest functions.

Basically, the idea is to extract the communication vehicle (vehicle), the time elapsed since the news was published (time), and the main headline (headline).

The code and comments are presented below:

# loading the packages:
library(dplyr) # for pipes and the data_frame function
library(rvest) # webscraping
library(stringr) # to deal with strings and to clean up our data

# extracting the whole website
google <- read_html("https://news.google.com/")

As we can see, the Google News website is divided in rectangular chunks of headlines and other info. Therefore, our strategy is to first scrape the whole chunks, and then, for each chunk scrape the information of interest: vehicle, time, and headlines.

We start scraping the whole chunks of articles. By using the “inspect” tool of our browser, we can see that the “article” \HTML node is the one that identifies each chunk.

# extracting the headlines
# and using stringr for cleaning
article_all <- google %>% html_nodes("article")

article_all

## {xml_nodeset (0)}

Having the whole chunks, now we can separately scrape the information of interest:

times <- article_all %>%
  html_node("time") %>%
  html_text()

vehicles <- article_all %>%
  html_nodes("a.wEwyrc.AVN2gc.uQIVzc.Sksgp") %>%
  html_text()

headlines <- article_all %>%
  html_nodes("a.DY5T1d") %>%
  html_text()

Let’s take a look at these vectors:

# take a look at the first ten
headlines[1:10]

##  [1] NA NA NA NA NA NA NA NA NA NA

vehicles[1:10]

##  [1] NA NA NA NA NA NA NA NA NA NA

times[1:10]

##  [1] NA NA NA NA NA NA NA NA NA NA

It seems all good!

Then, we can proceed to generate our final tibble:

tb_news <- tibble(headlines, vehicles, times)

tb_news

## # A tibble: 0 × 3
## # ℹ 3 variables: headlines <chr>, vehicles <chr>, times <chr>

That’s all!