Price is what you pay. Value is what you get. Warren Buffet
Quant Finance is one of the toughest areas to apply Machine Learning. In recent times, it has undergone tremendous iterations and changes in terms of how things are being practised. There has been a constant attempt to democratise the space with several quantitative platforms in the market.
I have been involved in this space for some time through my association with WorldQuant as a research consultant. The program with an idea to democratise alpha generation was an eye-opening experience. As I’m getting back into the fold of Quant Finance with a full-time role, I wanted to quickly recap all my understanding and basics.
In this attempt to get a good overall overview, I collected and collated feedback and inputs from multiple sources namely Reddit, personal bookmarks, books and plain old google search.
This is a basic compendium of major resources, I found. Will keep updating this as I go ahead.
Provides a nice little refresher into possibilities of using ML in Quant Finance and uses Python (Pandas) for processing. Gives a neat introduction.
- Udacity: Machine Learning for trading
There were a couple of books widely recommended from multiple sources especially from several handles on Reddit. The first one is a nice hands-on book, allowing you to build on the course of Udacity.
These are some of the blogs which have interesting commentary, a couple of them are by the same authors as the books.
Hard technical topics are best leveraged using notes besides the already available resources, it helps to retain and recollect things fast.
- Numerai: Had participated on this portal way back when they had just got started. Had even received some bitcoin as a part of the reward. Ever since they have gone on to create their cryptocurrency and made several changes to the overall platform. Think ML for finance in Kaggle style.
- Quantopian: Another popular platform but more traditional in nature with a much more simpler scheme. Has plenty of data sources, nice research environment and a thriving community on forums.
- Websim: Another simulation platform, by WorldQuant. This one has a nice payout scheme if you do well with revenue sharing and fixed stipends.
The entire trade cycle can be divided into four parts based on the different aspects which go into researching, building, deploying and then evaluation of an alpha. Each node provides ample opportunities to employ data science and machine learning. I also collated Reddit thread replies and comments.
Data & Pre-Processing
The edge here is in terms of size, latency and novelty of the type of data being used. Price and order book data are heavily used with Alternate Data emerging up.
- Gaining access to quality data is the biggest challenge in terms of the entry barrier.
- Cost is extremely prohibitive.
- Openly available data is ubiquitous and has low signal power.
- Non-stationary, Non-IID & Non-Normal price data points result in the violation of several Machine Learning algorithmic assumptions.
- Instead of sampling data in terms of time, we sample data in volume terms called volume bars. This has a dual advantage :
- The corresponding volume has better statistical properties (iid & gaussian)
- This takes into account also the volume aspect of information and we manage to capture more information due to higher sampling during higher activity
Signal Generation & Processing
This is one area where the majority of ML applications is being explored. Everything around converting the data sets into useful signals comes here.
- Overfitting to data is the biggest challenge with ML models which coupled with relentless backtesting can result in lots of spurious results.
- Feature selection and not back-testing is where the edge is. Use simple models to understand and interpret the top features/ predictors.
- Model ML problems as classification over regression. Simple models over complex approach: Occam’s Razor
- Follow a research-driven approach(EDA, summaries) contrary to back-testing heavy model.
- Split Data into Train, Validation & Test.
- Normalise the values.
- Plot the error values after each epoch. This will help in understanding if we are overfitting, generalisability of models etc.
- Order book “pictures” used and trained using transfer-learning to predict the next set of movements.
- LSTMs: Any paper on time series will have some relevant stuff for financial data sets. The pre-processing steps can be replicated for financial data sets as well.
Portfolio Allocation & Risk Management
Besides signal generation, a portfolio allocation processes also involve several risk management principles. I have seen firms employ strict risk management constraints such as not more than 2% liquidity in one stock etc.
- Kelly Criterion & Portfolio Allocation theory to distribute funds to signals. Make assumptions about a known mean, variance of returns which is a huge assumption.
- Extreme returns(primarily negative) have a higher probability than the traditional normal distribution. This is important from a risk management point of view. This is called fat tails in returns.
- This can result in greater drawdown during live markets compared to backtesting.
- A large & diverse portfolio can bring the excess kurtosis close to zero.
- This assumption of independence can be dangerous, the fat tails aspect coupled with highly correlated asset movements resulted in a failure of risk management during the 2008 crisis.
- The covariance between assets is constantly changing and is highly correlated with negative stock movements.
Evaluation & Execution
Post-simulation of the model under conditions of risk etc, they still need to be evaluated. Several metrics and evaluation techniques have been developed but the most prominent one is the historical Sharpe Ratio during a pre-defined data.
- Sharpe Ratio: The best single criteria to evaluate stocks under the assumption that returns have a normal distribution.
- Returns typically are known to have a high kurtosis and long negative tail so the probability of high drawdowns is greater in real scenario compared to back-tests.
- Bonferroni Test: The p-value for significance is adjusted as we carry out more back testing operations. This allows to ensure new max Sharpe >> old max Sharpe for it to be an actual signal.
- Transaction costs are absolutely critical and important to check for, often not accounted for in back-tests and modelling.
- Execution Strategy :
- You can’t execute at midpoint prices, so need to include price.
- Trade execution needs to include the volume aspect as well.
- You need at the minimum last trade price with volume. Best is to have order book depth.