# Commitments of Traders: Update 12 August 2014

# Plotting Flows with riverplot

I have been looking for an intuitive way to plot flows or connections between states in a process. An obvious choice is a Sankey Plot, but I could not find a satisfactory implementation in R… until I read the riverplot post by January Weiner. His riverplot package does precisely what I am need.

Getting your data into the right format is a slightly clunky procedure. However, my impression is that the package is still a work in progress and it’s likely that this process will change in the future. For now though, here is an illustration of how a multi-level plot can be constructed.

## Assembling the Data

The plan for this example is to have four nodes at each of six layers, with flows between layers. The data are a little contrived, but they illustrate the procedure quite nicely and they produce a result which is not dissimilar to the final plot I was after. We have to create data structures for both nodes and edges. I will start with the edges and then use these data to extract the nodes.

The edges data frame consists of records with a “from” node (N1) and a “to” node (N2) as well as a value for the flow between them. Here I systematically construct a grid of random flows and remove some records to break the symmetry.

> edges = data.frame(N1 = paste0(rep(LETTERS[1:4], each = 4), rep(1:5, each = 16)), + N2 = paste0(rep(LETTERS[1:4], 4), rep(2:6, each = 16)), + Value = runif(80, min = 2, max = 5) * rep(c(1, 0.8, 0.6, 0.4, 0.3), each = 16), + stringsAsFactors = F) > > edges = edges[sample(c(TRUE, FALSE), nrow(edges), replace = TRUE, prob = c(0.8, 0.2)),] > head(edges) N1 N2 Value 1 A1 A2 2.3514 4 A1 D2 2.2052 5 B1 A2 3.0959 7 B1 C2 2.8756 9 C1 A2 4.5099 10 C1 B2 4.1782

The names of the nodes are then extracted from the edge data frame. Horizontal and vertical locations for the nodes are calculated based on the labels. These locations are not strictly necessary because the package will work out sensible default values for you.

> nodes = data.frame(ID = unique(c(edges$N1, edges$N2)), stringsAsFactors = FALSE) > # > nodes$x = as.integer(substr(nodes$ID, 2, 2)) > nodes$y = as.integer(sapply(substr(nodes$ID, 1, 1), charToRaw)) - 65 > # > rownames(nodes) = nodes$ID > head(nodes) ID x y A1 A1 1 0 B1 B1 1 1 C1 C1 1 2 D1 D1 1 3 A2 A2 2 0 B2 B2 2 1

Finally we construct a list of styles which will be applied to each node. It’s important to choose suitable colours and introduce transparency for overlaps (which is done here by pasting “60” onto the RGB strings).

> library(RColorBrewer) > # > palette = paste0(brewer.pal(4, "Set1"), "60") > # > styles = lapply(nodes$y, function(n) { + list(col = palette[n+1], lty = 0, textcol = "black") + }) > names(styles) = nodes$ID

## Constructing the riverplot Object

Now we are in a position to construct the riverplot object. We do this by joining the node, edge and style data structures into a list and then adding “riverplot” to the list of class attributes.

> library(riverplot) > > rp <- list(nodes = nodes, edges = edges, styles = styles) > # > class(rp) <- c(class(rp), "riverplot")

Producing the plot is then simple.

> plot(rp, plot_area = 0.95, yscale=0.06)

## Conclusion

I can think of a whole host of applications for figures like this, so I am very excited about the prospects. I know that I am going to have to figure out how to add additional labels to the figures, but I’m pretty sure that will not be too much of an obstacle.

The current version of riverplot is v0.3. Incidentally, when I stumbled on a small bug in v0.2 of riverplot, January was very quick to respond with a fix.

# Commitments of Traders: Update 5 August 2014

Here are the data I distilled from the most recent CFTC Commitments of Traders report. Relatively large changes in CHF and JPY.

# Commitments of Traders: Moves in the Last Week

In my previous post I gave some background information on the Commitments of Traders report along with a selection of summary plots.

One of the more interesting pieces of information that one can glean from these reports is the shift in trading sentiment from week to week. Below is a plot reflecting the relative change in the number of long and short positions held by traders in each of the sectors (Commercial, Non-Commercial and Non-Reportable).

The changes are normalised to the total number of positions (both long and short) held in the previous week. To illustrate how this works, consider the JPY.

> tail(subset(OP, name == "JPY" & sector == "Commercial"), 2) name date sector long shrt 11842 JPY 2014-05-13 Commercial 125523 -35537 11845 JPY 2014-05-20 Commercial 117310 -48851

This indicates that the number of positions that are long relative to the JPY has decreased while the number of positions that are short on the JPY (given by a negative number) has increased. Both of these changes are consistent with the fact that traders are selling the JPY in favour of other currencies.

A synopsis of these data for a range of currencies is given in the plot below. This is how the plot works. Again we will consider Commercial trades involving the JPY. We are thus looking at the second to last row and first column. Here there are two cells: long trades on the left and short trades on the right. The coloured bars indicate the relative change for long (blue) and short (orange) trades. The relative changes are normalised to the total number of trades for the currency and sector in the previous week. We can see here that the orange bar is broader than the blue bar indicating that the change in short trades is larger than the change in long trades. The grey boxes show the 95% confidence interval for the expected range of these changes. The closer the bars come to the edge of the boxes, the more significant the change. So, in this case, the change in the number of short trades is significant and is probably an indication of a change in sentiment regarding the JPY.

If anyone is interested in updated charts like this, please just let me know.

# Charting culture

This is well worth watching. Good idea. Great presentation. Effective use of the media to publicise a scientific paper.

# Whistlers and Volcanic Lightning

A paper authored with my very talented student, Claire Antel, entitled “Investigating Dunedin whistlers using volcanic lightning” has just been published in Geophysical Research Letters. The paper looks at electromagnetic signals (“whistlers”) received at Dunedin, New Zealand, caused by lightning over volcanoes thousands of km away near the Aleutian Chain.

## Abstract

Whistlers detected at Dunedin, New Zealand are an anomaly: there is little lightning around Dunedin’s conjugate point yet whistlers appear in relatively large numbers. These surplus whistlers have consequently inspired investigations into their origins. Dunedin’s lightning-sparse conjugate point lies in the Aleutian Islands, a region populated with active volcanoes. Their presence has allowed us to perform a novel analysis: the correlation of whistlers to volcanic lightning. We report on our investigation, which successfully yielded the first observations of “volcanic whistlers.” It was found that the single July 2008 Mount Okmok eruption had an impressive effect on the number of whistlers at Dunedin. The eruptions at Mount Redoubt in 2009 also caused a sporadic flow of whistlers in Dunedin.

## In Other Media

Claire has also given an interview about the paper.

# Comrades Marathon: A Race for Geriatrics?

It has been suggested that the average Comrades Marathon runner is gradually getting older. As an “average runner” myself, I will not deny that I am personally getting older. But, what I really mean is that the average age of *all* runners taking part in this great event is gradually increasing. This is not just an idle hypothesis: it is supported by the data. If you’re interested in the technical details of the analysis, these are included at the end, otherwise read on for the results.

## Results

The histograms below show graphically how the distribution of runners’ ages at the Comrades Marathon has changed every decade starting in the 1980s and proceeding through to the 2010s. The data are encoded using blue for male and pink for female runners (apologies for the banality!). It is readily apparent how the distributions have shifted consistently towards older ages with the passing of the decades. The vertical lines in each panel indicate the average age for male (dashed line) and female (solid line) runners. Whereas in the 1980s the average age for both genders was around 34, in the 2010s it has shifted to over 40 for females and almost 42 for males.

Maybe clumping the data together into decades is hiding some of the details. The plot below shows the average age for each gender as a function of the race year. The plotted points are the observed average age, the solid line is a linear model fitted to these data and the dashed lines delineate a 95% confidence interval.

Prior to 1990 the average age for both genders was around 35 and varies somewhat erratically from year to year. Interestingly there is a pronounced decrease in the average age for both genders around 1990. Evidently something attracted more young runners that year… Since 1990 though there has been a consistent increase in average age. In 2013 the average age for men was fractionally less than 42, while for women it was over 40.

## Conclusion

Of course, the title of this article is hyperbolic. The Comrades Marathon is a long way from being a race for geriatrics. However, there is very clear evidence that the average age of runners is getting higher every year. A linear model, which is a reasonably good fit to the data, indicates that the average age increases by 0.26 years annually and is generally 0.6 years higher for men than women. If this trend continues then, by the time of the 100th edition of the race, the average age will be almost 45.

Is the aging Comrades Marathon field a problem and, if so, what can be done about it?

## Analysis

As before I have used the Comrades Marathon results from 1980 through to 2013. Since my last post on this topic I have refactored these data, which now look like this:

> head(results) key year age gender category status medal direction medal_count decade 1 6a18da7 1980 39 Male Senior Finished Bronze D 20 1980 2 6570be 1980 39 Male Senior Finished Bronze D 16 1980 3 4371bd17 1980 29 Male Senior Finished Bronze D 9 1980 4 58792c25 1980 24 Male Senior Finished Silver D 25 1980 5 16fe5d63 1980 58 Male Master Finished Bronze D 9 1980 6 541c273e 1980 43 Male Veteran Finished Silver D 18 1980

The first step in the analysis was to compile decadal and annual summary statistics using plyr.

> decade.statistics = ddply(results, .(decade, gender), summarize, + median.age = median(age, na.rm = TRUE), + mean.age = mean(age, na.rm = TRUE)) > # > year.statistics = ddply(results, .(year, gender), summarize, + median.age = median(age, na.rm = TRUE), + mean.age = mean(age, na.rm = TRUE)) > head(decade.statistics) decade gender median.age mean.age 1 1980 Female 34 34.352 2 1980 Male 34 34.937 3 1990 Female 36 36.188 4 1990 Male 36 36.440 5 2000 Female 39 39.364 6 2000 Male 39 39.799 > head(year.statistics) year gender median.age mean.age 1 1980 Female 35.0 35.061 2 1980 Male 33.0 34.091 3 1981 Female 33.5 34.096 4 1981 Male 34.0 34.528 5 1982 Female 34.5 35.032 6 1982 Male 34.0 34.729

The decadal data were used to generate the histograms. I then considered a selection of linear models applied to the annual data.

> fit.1 <- lm(mean.age ~ year, data = year.statistics) > fit.2 <- lm(mean.age ~ year + year:gender, data = year.statistics) > fit.3 <- lm(mean.age ~ year + gender, data = year.statistics) > fit.4 <- lm(mean.age ~ year + year * gender, data = year.statistics)

The first model applies a simple linear relationship between average age and year. There is no discrimination between genders. The model summary (below) indicates that the average age increases by about 0.26 years annually. Both the intercept and slope coefficients are highly significant.

> summary(fit.1) Call: lm(formula = mean.age ~ year, data = year.statistics) Residuals: Min 1Q Median 3Q Max -1.3181 -0.5322 -0.0118 0.4971 1.9897 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -4.80e+02 1.83e+01 -26.2 <2e-16 *** year 2.59e-01 9.15e-03 28.3 <2e-16 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.74 on 66 degrees of freedom Multiple R-squared: 0.924, Adjusted R-squared: 0.923 F-statistic: 801 on 1 and 66 DF, p-value: <2e-16

The second model considers the effect on the slope of an interaction between year and gender. Here we see that the slope is slightly large for males than females. Although this interaction coefficient is statistically significant, it is extremely small relative to the slope coefficient itself. However, given that the value of the abscissa is around 2000, it still contributes roughly 0.6 extra years to the average age for men.

> summary(fit.2) Call: lm(formula = mean.age ~ year + year:gender, data = year.statistics) Residuals: Min 1Q Median 3Q Max -1.103 -0.522 0.024 0.388 2.287 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -4.80e+02 1.68e+01 -28.57 < 2e-16 *** year 2.59e-01 8.41e-03 30.78 < 2e-16 *** year:genderMale 3.00e-04 8.26e-05 3.63 0.00056 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.68 on 65 degrees of freedom Multiple R-squared: 0.937, Adjusted R-squared: 0.935 F-statistic: 481 on 2 and 65 DF, p-value: <2e-16

The third model considers an offset on the intercept based on gender. Here, again, we see that the effect of gender is small, with the fit for males being shifted slightly upwards. Again, although this effect is statistically significant, it has only a small effect on the model. Note that the value of this coefficient (5.98e-01 years) is consistent with the effect of the interaction term (0.6 years for typical values of the abscissa) in the second model above.

> summary(fit.3) Call: lm(formula = mean.age ~ year + gender, data = year.statistics) Residuals: Min 1Q Median 3Q Max -1.1038 -0.5225 0.0259 0.3866 2.2885 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -4.80e+02 1.68e+01 -28.58 < 2e-16 *** year 2.59e-01 8.41e-03 30.79 < 2e-16 *** genderMale 5.98e-01 1.65e-01 3.62 0.00057 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.68 on 65 degrees of freedom Multiple R-squared: 0.937, Adjusted R-squared: 0.935 F-statistic: 480 on 2 and 65 DF, p-value: <2e-16

The fourth and final model considers both an interaction between year and gender as well as an offset of the intercept based on gender. Here we see that the data does not differ sufficiently on the basis of gender to support both of these effects, and neither of the resulting coefficients is statistically significant.

> summary(fit.4) Call: lm(formula = mean.age ~ year + year * gender, data = year.statistics) Residuals: Min 1Q Median 3Q Max -1.0730 -0.5127 -0.0492 0.4225 2.1273 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -460.3631 23.6813 -19.44 <2e-16 *** year 0.2491 0.0119 21.00 <2e-16 *** genderMale -38.4188 33.4904 -1.15 0.26 year:genderMale 0.0195 0.0168 1.17 0.25 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.679 on 64 degrees of freedom Multiple R-squared: 0.938, Adjusted R-squared: 0.935 F-statistic: 322 on 3 and 64 DF, p-value: <2e-16

On the basis of the above discussion, the fourth model can be immediately abandoned. But how do we choose between the three remaining models? An ANOVA indicates that the second model is a significant improvement over the first model. There is little to choose, however, between the second and third models. I find the second model more intuitive, since I would expect there to be a slight gender difference in the rate of aging, rather than a simple offset. We will thus adopt the second model, which indicates that the average age of runners increases by about 0.259 years annually, with the men aging slightly faster than the women.

> anova(fit.1, fit.2, fit.3, fit.4) Analysis of Variance Table Model 1: mean.age ~ year Model 2: mean.age ~ year + year:gender Model 3: mean.age ~ year + gender Model 4: mean.age ~ year + year * gender Res.Df RSS Df Sum of Sq F Pr(>F) 1 66 36.2 2 65 30.1 1 6.09 13.23 0.00055 *** 3 65 30.1 0 -0.02 4 64 29.5 1 0.62 1.36 0.24833 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Lastly, I constructed a data frame based on the second model which gives both the model prediction and a 95% uncertainty interval. This was used to generate the second set of plots.

fit.data <- data.frame(year = rep(1980:2020, each = 2), gender = c("Female", "Male")) fit.data <- cbind(fit.data, predict(fit.2, fit.data, level = 0.95, interval = "prediction"))

# Where to Put EAs and Indicators in New MT4 Builds

If you are creating an EA or indicator from scratch, then the MetaTrader editor places the files in the correct location and the terminal is automatically able to find them. However, if the files originate from a third party then you will need to know where to insert them so that they show up in the terminal. For older builds of MetaTrader 4 the directory structure was fairly simple. Everything was to be found in a folder under *Program Files*. Its contents looked like this:

EAs would go in the *experts* folder, while indicators would go one level further down in the *indicators* sub-folder.

But everything is different with the new builds of MetaTrader 4.

## Windows

On Windows (and here I am referring specifically to Windows 7, although the setup will be similar on other flavours) the structure of the MetaTrader folder (still found under *Program Files*) now looks like this:

There seems to be something missing, right? Where do the EA and indicator files go? To find these you will need to look elsewhere. There is now a separate directory hierarchy for each user, in which these files are stored. Browsing under the *AppData* folder for your user you should find

Locating this folder can be a little tricky. First look under *Users* to find the folder associated with your user name. Within this folder you may or may not be able to see an *AppData* folder. If you can’t see it then you will need to reveal hidden files by changing the folder options.

Once you have dug down to your *Terminal* folder it should contain a set of sub-folders like this:

If you descend further into one of those obscurely named folders, then you will find further sub-folders.

Finally, within the *MT4* folder, you will find this:

The *Experts* and *Indicators* folders are where you will insert those EAs and indicators.

## Linux

If you are running MetaTrader under Wine on Linux, then the new directory structure looks like this:

Evidently the structure is a little simpler here, because all of the files and directories are still lumped in a single location. The *Experts* and *Indicators* directories are to be found under *MQL4*.

# Comrades Marathon Negative Splits: Cheat Strikes Again

It looks likes one of the suspect runners from my previous posts cheated again in this year’s Comrades Marathon. Brad Brown has published evidence that the runner in question, Kitty Chutergon (race number 25058), had another athlete running with his number for most of the race. This is a change in strategy from last year, where he appears to have been assisted along the route, having missed the timing mats at Camperdown and Polly Shortts.