A few months ago we launched a scientific Prediction Market pilot at Poznan University of Economics (PUE) called Poznan Electronic Markets (PEM) (paying tribute to the original Iowa Electronic Markets (PM) at the University of Iowa). This pilot was aimed at testing the feasibility of an applied long-term research project in Prediction Markets. For instance, we needed to estimate how large an operational crew would be needed and if the scientific community at the university as well as student traders would support such research (see the blog post on the launch of PEM).
After almost 4 weeks of trading (26 trading days) on a total of 44 markets the operational part of the pilot was successfully completed. From running PEM for a short time we saw that running a PM research project is possible with reasonable resources. Now the way forward is to gather support from local scientists as well as to organize a larger scale second pilot early next year involving all local universities (there’re more than 125,000 students in Poznan) to achieve a significant trader base before commencing any research. Also a research programme will be drawn up a draft of which is already being circulated among the current PEM team. Also, anybody interested in Prediction Market research is invited to contact the PEM team or Analyx to discuss a possible cooperation.
Besides the positive conclusion on the resources needed for running a PM for research we also looked into the performance of PEM. Here we’d like to share some of those results.
There were a total of 36 binary forecasts, some of them simple single binary YES/NO forecasts and some multi binary choose-the-best-option forecasts, as well as 8 index forecasts, i.e. markets where we forecasted a number not a probability, e.g. the central bank’s interest rate. We had a total of 131 student traders out of the 15,000 students of PUE, which we recruited with a bit of advertisement. None of the participants was trained in using the Prediction Market but we provided videos on the web site which explained how the system worked. Trader statistics show that about one third of the traders had 26 or more orders during the 4-week trading period. Most people made only one order on each market they decided to participate in. Only few traders were highly active – these were our marginal traders.
Although forecasting accuracy was not a primary concern in this pilot (we even tried some really weird payoff rules, just to see if forecasts would be completely off) the calibration of the binary forecasts is very good and the error percentage of index markets is very low, respectively.
We grouped all binary contracts in 6 groups of approximately equal size and then calculated the calibration of these markets. While the result is in no way statistically significant (the PEM pilot had other objectives, see above) we are glad to see that even a PM with a small and untrained trader population can forecast well. The black line in the figure below shows the perfect forecast. The linear correlation of the PEM pilot markets with an 93% R-squared compares quite well to that.
Looking at the index markets, we calculated a MAPE of 22% (mean absolute percentage error). When excluding one of the 8 forecasts, the remaining ones achieve a MAPE of 15%. Errors of this magnitude are comparable to errors of professional forecasting processes in large corporations and we are quite satisfied with this result given that it was somewhat of a “side product” of this pilot. Some of the index forecasts had a very complex and rather unintuitive payoff rule because we were interested to see if this would completely blow the forecasts. Indeed, it made forecasting more difficult and those markets had an above average error which weighted into the overall MAPE of 22%.
Again, all of these results are only given as a side information to the interested reader. The main goal of the first PEM pilot was to test if Prediction Markets would be well received and if we could sustain a long-term PM research programme. We’re looking forward to the next stage of PEM and invite comments from anybody interested in Prediction Market research.
