The Mechanics of Predicting the Stock Market, from a Financial Engineering Major

Earlier this week, I had the privilege to speak with Jeremy Coste, a current student at Columbia University who studies quantitative finance and financial engineering. This was a very valuable and insightful conversation for me, as a prospective student in these disciplines. It is always great to hear from a student, as they tend to give honest feedback on the work they’re doing (they don’t have as much to lose as someone who runs their own trading firm, and whose income relies on convincing others to invest in their outlooks). Mr. Coste is in a similar situation as my last interviewee, who also graduated from a competitive engineering school and later transitioned into a prominent trading institution. Some differences are that Mr. Coste works at Bank of America, while Mr. Conerly works at a more specialized high frequency trading firm. This is an interesting note, as contrasting these two interviews could give insights into general bank trading strategies vs. HFT trading strategies.
Once again, due to the secretive nature of quantitative trading, I was unable to get a full transcript of the interview, but the answers I provide are accurate renditions of Mr. Coste’s answers.
S.A: Could you give me a general idea of what you do for work? Did you get your knowledge through school, or does this field require more personal, independent research? 
J.C: I’m currently studying at Columbia’s Financial Engineering department, but I also work as an intern with Bank of America as a quant. I have experience in deep learning, reinforcement learning, unsupervised learning. Financial engineering as a study is challenging mathematically, and about 60% of the students I work with are from China, 20% from India, with a few from France and U.S as the rest. There’s a lot of applied math, stochastic calculus, and every course has a lot of fairly advanced math. It’s sort of building up from multivariable calculus, and then moving into more programming (languages, datasets). But, like with most studies, there’s definitely a lot of independent work that needs to be done as well. 
S.A: In your experience of writing/learning about trading algorithms, is there a stronger emphasis on mathematical modeling, such as statistical probabilities, or on deep learning/big data strategies?
J.C: There’s a few main branches to working on trading algorithms, and optimization is a big one (math heavy). The stochastic calculus aspect is for modeling how the market moves, while the machine learning side is very data heavy. For all of the data courses, this is clumped into the data analyst section, while core courses focus more on quantitative aspect.
Data is a much more prominent aspect, which is where I, as a quant, focus a lot of my work. From what I’ve seen, everyone is using deep data. Data is the future, so it’s a good deal, but with all the hype that’s starting to surround AI and finance, data scientists are coming up with automated methods that can compare/contrast all these different data analyzing strategies. This is why right now, the shift is more towards jobs in researching new machine learning strategies which aren’t already being used. Deep learning algos are something of a black box, which is why there’s so much room for opportunity and research, as there’s so many variables and parameters you can change. One technique I think you should look at when working with this many variables is cross validation, as this can be super useful. These deep learning tools can also be more specific, like researching new models that could perform better in certain specific circumstances.
S.A: What’s your opinion on using deep learning to map similar patterns and events in the markets? Do you think this could be a profitable strategy? (some say the behavior of the markets don’t follow easily-recognizable patterns, others say they work in identical cycles) If so, would this type of strategy be better suited for long-term or short-term projections?
J.C: First off, I think it’s important to define what a stock event is when you’re talking about mapping similar events.  
There is definitely some correlation between patterns and stocks, even just looking at charts of the S&P, booms follow recessions. There’s definitely patterns we can see aside from that, too. Tick data could be interesting to look at, which are small pieces of stock data for say every tenth of a second, and we could use this to predict whether the next tick will go up or down. Small scale can yield higher results, as we have a bigger abundance of data on a smaller scale rather than historical stock data going 20 years back. RNNs on these small scales should give a better percentage, like 60%.
One other interesting thing to look at is, like you mentioned, predicting whether or not a stock will go up or down in the next day. This is a much more difficult task, and therefore would have a much lower accuracy of around 50%, but it would also yield bigger rewards. One thing to remember though is that even if your accuracy is at 55% for predicting next day stock performance, this is great, as you’re getting it right more than half of the time and the rewards are so large that they outweigh the losses. Obviously, a better percentage is more profitable and desirable, but don’t be discouraged by a low percentage to begin with.
Also, what banks and hedge funds will do is hedge bankruptcies through puts and calls, which is the primary use of these options. These options can be incredibly useful, as they’re very cheap. It’s important to know that standard deviations are used to calculate prices of puts/calls, and that you can use volatility to decide whether or not to buy puts/calls in certain situations.
S.A: What’s one factor you would consider most heavily when gauging the value of an asset (say: stock, bond, future, currency)?  
J.C: I don’t know if you can value a company in just one factor, and volume definitely doesn’t work to estimate value. I would think market cap is good for this, another thing to consider is enterprise value which is based on belief in the efficient market theory. Neural nets can handle a lot of factors, so it doesn’t really make sense to limit yourself to just one, especially when looking at something as complex as stock price prediction. I would consider using variable reduction (principle product analysis). With this tool, you can include say 100 factors, and then variable reduce to find which of these 100 factors are most significant.  
So, to recap, use as many features as you can find in the data, and my belief is that the more information the better. Eliminate as many heuristics as you can, because these can be especially problematic and commonplace with stock data
S.A: What are some good ressources you recommend I look at, or people I talk to, as a beginner in quantitative trading?
J.C: If you’re interested in this field, the math used, the programming used, I would take a look at: A Practical Guide to Quantitative Finance Interviews, the Green Book. This one is referenced in a lot of quant finance classes, and can familiarize you with what employers look for in recruits for their trading algorithm sectors. Other than that, most books I’ve used are just text books.
To find people to interview, I wouldn’t start with students in quant finance, because most of them are really busy with work and internships. I would use LinkedIn as a tool, and message people through LinkedIn. LinkedIn is great because you can use keywords to find people working in similar fields, so you can for instance use quantitative finance keywords to find relevant people. Some schools I know have great quantitative finance programs are UC Berkeley, NYU, and obviously Carnegie Mellon and MIT.  
Image result for takeout boxImage result for takeout box


Mr. Coste’s point about low-percentage accuracy is quite interesting, and super inspiring as well. If you think about it, consistently predicting next-day price changes with an accuracy of only 50% sounds really bad, but in the long run, you would be breaking even. Say you’re flipping a coin, and heads means you’re right about what a stock will do tomorrow (+$$) and tails means you’re wrong (-$$). In a hundred days, you should have about 50 heads and 50 tails, meaning an equal amount of profits and losses (if the magnitude of the investments are the same every time). What this also means is that a consistent accuracy of 51% would ensure profit in the long run, albeit very small. This doesn’t mean that I’m shooting for a low accuracy, because the better the accuracy, the more often you’re profiting. I mention this because I think it’s interesting how people consider stock predictions to be exercises in futility, while forgetting that you don’t need to be right 100% of the time (this is impossible), you just need to be right enough of the time while hedging losses in times when you’re wrong.
Image result for flip a coin
But, Mr. Coste also emphasizes that predicting how much a stock will gain/dip tomorrow is very profitable (especially if you predict the magnitude at which the price changes), and much more profitable than predicting tick-data, which is what HFT traders do. Again, we see that the trade-off is high accuracy for smaller profits, or small accuracy for bigger profits. Considering that HFT is dominated by the “speed arms race” (cool blog about this) and the fact that tick-data requires many more transactions and therefore many more fees, it makes sense for me to work on predicting day-to-day behavior of stocks. Also, with the datasets I have found and the revelations I made in my last post, it seems like this is the right direction to take my project.
A second compelling takeaway is Mr. Coste’s point about using as much data as possible, rather than limiting myself to a single factor to predict stock price shifts. This seems like the obvious answer, but I don’t know much about coding variable reduction and cross validation systems, so this might be another area I need to study. I wonder if variable reduction is affected by the ‘noise’ of too much data, as I have talked about how neural networks can often find misleading patterns if you give it too much information.
A third interesting note is about using options, specifically puts and calls, to hedge potential losses. I am familiar with options, and have traded them in the past, but for some reason I have neglected using them in my trading algorithm (the benefits of talking to people!). I don’t think deciding whether or not to purchase hedge options is a job for A.I, as I think this is more intuition based on volatility and how much you are investing in a certain position. With the stock data I have, I can definitely calculate volatility, and factor this into some investments.
A final take away from this conversation is that Mr. Coste acknowledges patterns in market behavior, and that A.I could be used to find these patterns. This is very reassuring, as this is the premise of the project I am working on.

Leave a Reply

Your email address will not be published. Required fields are marked *