I'm thinking of trading on either the FX or Stock markets using ML algorithms (primarily neural networks) on an intraday timescale. Does anyone have any experience in doing this? Any tips/stories/recommendations on brokers, data, etc? Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)?
I don't know much about stock markets, but I did a consulting gig for a large currency trading company a while back where I was doing a bit of work on their algorithmic trading platform. I learned that unless you are trading at least a million dollars a trade it's very unlikely that you will make money.
The reason for this is that the large companies won't talk to you unless you're trading at those volumes since they charge pretty high transaction fees. The smaller companies that will talk to you all have accounts with the larger companies and make money by increasing the spread between the bid and ask price they receive from the large companies. The expected return on a trade is usually somewhere in the region of 1-10 pips (where a pip is one ten thousandth of a dollar), with 10 pips being a really good return and 3-4 being more realistic. The smaller companies usually increase the spread by up to 3 pips in each direction on the price they get from the larger company which swallows up your return.
While millions of dollars per trade might sound like a lot the leverage in this market is also very high at 50-100 times meaning that with your 10k you could trade 500k-1M though on a sum that low you might get less leverage. However if you lose money on a trade you could eat substantially into your capital. So if your $1M trade goes against you by 10 pips you'd lose 1k.
I'm not ruling out making money on that market, but I'd say it's very unlikely unless you have a lot more money to start with. I am not an expert in this area though. This is just what I learned from the small amount of time I spent working with the company.
Yes, I did work on that. I used a neural network to trade FX. I wrote my own neutral network framework using code generators for x86 and CUDA.
The neural network was a fairly normal feed-forward network with GA-inspired, but heavily modified learning process. It combined layers of neurons with memory and memory-less neurons. It is essential to keep the network size small, because a big network will just memorize the data you use to train it.
Also, I used several other inputs, such as the frequency of trades per minute, overnight rates for each currency, and stock market indexes for the relevant countries.
Finding historic data was the biggest problem. I used Bloomberg data from my university, but that was a pain in so many ways. There is NO free data for most of the inputs I needed. Good data from Thomson Reuters or something like starts at $5k/mo.
Overall, the model worked fairly well, generating about 10% annual returns even with a few handicaps I put to be on the safe side. This was about an year ago and the only reason I stopped working on it was that my partner in the financial startup quit. I'm sorry if some of my terminology is wrong - I haven't been working in that field since then. If you have any questions, I'd be happy to talk about this. My email is in my profile.
I have friends who do this, and they told me, anecdotally, that for every 10 trades they do, 8 or 9 lose money.
While they do make it up on the rest, you have to ask yourself if you can afford the costs (brokerage commissions and fees) of executing your strategy in that context (they all work for banks and hedge funds).
If you really want to get into this, you might be better off applying to a quant-based hedge fund.
A firm like that will have the capital and brokerage connections to make the game profitable.
All I can say is that 8/10 are losers is highly improbable. Given no other information, if you enter a trade there is a 50/50 chance that the next tick is going to be up or down. But notice I said given no other information, there is much more information from the behavior of the charts. If you are a small player the charts will help you be right more than 50% of the time. The rest is how your algorithms and trading methodology take advantage of the information.
In general, machine learning would probably not be a great approach to day trading. There are several problems:
1) Limited market data: As an individual, you have limited access to market data. At best, you probably have real time top of book data but without paying for it, you almost certainly aren't getting a real time depth feed. Going off of only last trade data gives you even less to work with. In most cases free quotes are delayed making them worthless to day traders (although not longer term investors).
2) Limitations of market data: Even if you have a lot of historical data with a full order book, that is still of limited utility for simulations. The problem is that everything you do affects the market so any trade you simulate needs to be doing small enough quantity to not make your simulation unreasonable. How to deal with simulating market-affecting trades is a complex problem and the only way to truly know is to run your trades in the market.
3) For day trading, there is a lot of profit to be captured simply from the microstructure of the market (its moves back and forth, etc.). To capture much of this it helps to be fast but you won't be able to colo boxes at an exchange with only 10K.
Machine learning seems more suited to longer term investments and trading on a slower platform than making a quick profit arbing or scalping. You may want to consider this rather than day trading since not only does day trading require much larger amounts of capital, but its also extremely risky.
Also, if you plan to trade equities, enjoy all of the ridiculous SEC regulation (order marking, etc.).
Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)?
Yes. The intraday movement of stocks is small enough that you'd need to be using a large fraction of your capital on every single trade in order to overcome trading costs; and based on the Kelly criterion you'd need to have unrealistically high odds of betting correctly in order to avoid going broke.
More suitable, sure. But I still doubt you'd get anywhere -- people who run hedge funds are very much aware of ML techniques, so I'd expect them to run circles around you.
There's a saying in the industry: your strategy will go south as soon as you put real money on it -- you can't raise capital without a track record of real money success.
If you want more information on this, there was an IAMA on Reddit about 2-3 months ago, on a guy who claimed to do exactly this. It was incredibly informative, and went into a lot of the details that you ask for here.
My understanding is this is a pretty huge field : http://en.wikipedia.org/wiki/Computational_finance and that if you try to roll your own, you are computing against a variety of math and CS phds that big trading firms hire to do analytics.
Can't you just download a historical set and use that to test whatever algorithms you think you would use? Then when you are confident enough in their performance, you can take the risk? I wouldn't say it would be a complete waste of time because you would be sure to learn something. If you think you can realistically make money--well then, that's a different question that I wouldn't be fit to answer.
> Can't you just download a historical set and use that to test whatever algorithms you think you would use?
Be careful - much of the historical data has been pruned to eliminate some data from companies that didn't survive.
I've heard that one of the major quant houses (Shaw?) can often guess what data set you used to test your strategy by looking at the strategy.
Plus, there's a selection or fitting problem. Your strategy may have worked well on average from 1972-1999, but that doesn't mean that it will work today.
And there's a velocity problem. There are pricing errors in old data to exploit. And, if you had today's computers then, you might have been able to find them fast enough because your systems are faster than what folks had then. However, today you're competing with folks who have faster systems than you do.
You make some good points, but isn't this always going to be a problem when training your algorithms? You can't really train in real time so by the nature of the industry you're always going to be put at a disadvantage by using older data.
The only tip is: do it on paper first, with the exact setup, lag times, etc. that you're going to have when doing it live. Do it for a while.
And realize that this is a fanciful pursuit which is likely to lose you money, because the guys you're going up against have $ millions, ns-scale lag times, free or near-free transaction costs, and PhDs in theoretical physics.
10K is too little. Statistically speaking, a small draw down can wipe out your acount. Most professionals have stop losses to limited their loss to 1-2% of their account. Google "trading money-management". You probably want to paper trade for a while before risking real money. If you are really interested in markets you can follow it as a hobby while working as a programmer. I used Interactive Brokers for trading futures, fx, options and stocks. They are low-cost and provides an API for automation. I write java code for my own personal "proprietary desk trading".
Being a "small fish" is actually an advantage since you have less capital you need to put to work and thus can more easily get fills and catch the bigger waves generated by the bigger fish institutions like mutual and hedge funds.
The institutions have to hold on for longer periods and can't really "scalp" trades per se given that the bid/ask volume would collapse if they didn't distribute and "average in" slowly overtime to accumulate a position vs. buying or dumping their entire position all at once.
True. Each side has their own set of advantages/disadvantages. And so you'd have to adapt and take that into consideration in addition to bid/ask spread. It's not impossible (nothing is impossible), and you'd only get killed by brokerage fees if you're overtrading.
If you can find an algorithm that can trade only a few times a day and ride stocks that has a volatility of 1% or more then even at retail commission levels you can still do pretty well as you slowly build up your account to trade strategies that require more churn.
intraday NN is irrelevant because NN are prone to overlearning, so the stuff becomes even more horrible - actually in reality you don't care which way it moves, as long as it moves: this is why the "big fish". It is like owning the place in Vegas: you don't care about the bets (much less about their substance or "direction"), as long as there are bets.
if you really want intraday, think about news.google.com and http://rs2007.limsi.fr/index.php/TLP:Page_12 to automatically tweak the few k base parameters (SVM, associative memory etc).
intraday NN is irrelevant because NN are prone to overlearning
I call bullshit. A lot of the most successful applications in machine learning are modeled using neural networks.
To regularize a neural network, you use standard machinery that controls the capacity of the machine. The scale of the weights cannot be too large (this is a famous result from IIRC Haussler in 1992) so penalize the weights using an l2 (Gaussian) prior. This reduces the model complexity and the potential for overfitting.
How much do you know about trading now? If you know little, I would recommend the textbook Trading & Exchanges by Larry Harris. It's somewhat out of date but still very good. I work at a high-frequency trading shop and that is the first book they shove at you when you walk in the door.
Other than that, I'd like to second what several other people have already said: getting a job at a firm is a good way to learn, you're outgunned in terms of brains and money, and 10k is probably too little. Now, you can do a lot of learning for 0k while you save money, and every startup is outgunned by goog, msft, but that doesn't mean there isn't room in the ecosystem. Like a startup against an established player, you could find a niche that is going unexploited. What makes trading harder is the established players don't tell you much about what they're doing, so you don't know if you are doing something new or walking into a trading strategy perfected 10 years ago by DE Shaw. With a software startup, you have more information about what your competitors are selling, what their products are, how you might differentiate yourself, what's missing from the internet, etc. Software companies promote their products and try to get public attention; trading shops try to keep as many secrets as possible.
The majority of people who have responded to you with various pieces of opinion/information forgot to answer this question:
Does anyone have any experience in doing this?
So take everything with a grain of salt-- there seems to be a lot of advice going around by people who don't have skin in the game-- especially those who say it can't be done. Unless they've put money down to test this, then their opinion should be discounted.
For those saying commissions are to high for a shorter term timeframe: trading has been commoditized. Lightspeed offers .40 per lot, and IBKR offers around that given a certain amount of trading volume. Forex brokers make money off the spread, and so the commission that you pay is only from the bid ask. Since you are trading small amounts, you will get hit on the comission structure, but if you pay per share rather than per trade it won't hurt as bad. You also might want to look into equity futures as well-- very liquid, good tax advantages, and good commission structures.
Biggest thing about small sizes is that you're subject to the pattern daytrader rule-- you can't make >3 intraday trades if your portfolio is <25k. SEC rules.
I'm a discretionary trader, and not a system trader. I am currently developing some systems to go live but they are not ready as of yet.
I also know of many who are quants, and it's possible and profitable.
You might find it interesting to look at this site, which gives you a lot of the infrastructure for testing your FX strategies with real time data (although not direct access to the data).
Strategies are independently monitored and ranked on the site, and can be switched from simulations to real accounts when you're ready. If your strategy performs well, it will be noticed and you can get paid commissions on it (potentially helpful for raising capital).
For FX and equity data, you might as well use Yahoo finance. I run a Slicehost server that scrapes it once every minute with no problem. It's not ultra high frequency but you can't hope to compete in that space anyway, as others have pointed out. You should start building a historical database now before you do anything else, so that you'll have something ready when you need it. You probably don't need years and years of it.
I worked as a quant for a while and would support the other comments that standard ML techniques are unlikely to outperform, but you might be the exception.
I just want to add that I read some of the Turtle Trader books and one more on Trend Following. The last book finally got across to me the point that the trend followers used computers and math not to try and predict where the markets were headed but rather to try and classify markets as rising, falling, or side to side. Their theory is that once a market has a direction it is likely to keep that direction - until it changes.
This is very different from most of the stuff I read about with neural nets. In most of the other books the idea to try and predict when markets are over sold or about to change so that you can be in on the ground floor.
I'm doing a webapp about card counting at black jack and the trend following approach seems a lot more like the reality of card counting. When the deck is hot or in your favor you bet more. Most people think that card counting is a super complicated exercise in combinatorics where the player is recalculating basic strategy in real time and able to prove that certain plays are the best.
If by reality you mean Buffett or Soros, consider the thousands (hundreds of thousands?) of extremely smart and hardworking people who fail to replicate their success.
If you're interested in this, why not join a firm that does it? They will have a large capital base to start with and eventually start your own venture. Unfortunately 10k won't get you very far in this game
Practice some with historical data, get a job with a firm that does this, and eventually strike out on your own. If you prove yourself, people will hand you millions to manage. Starting with your own money seems like a bit of a fools game.
This is not what you are asking but a market with more inefficiencies and less transaction costs is Betfair. There is a simple api and free historical data.
I spent a bunch of time with Neural Nets and SVM and that data. Sadly I didn't find a profitable strategy. I would love to hear about anyone else who is looking in that area.
I would generate a back or lay output signal for 'Correct score' bets for soccer matches before the game started. Input would be everything you can pull out of the the data set especially including the odds on other more standard bets like the match odds and the over / under odds.
There are also arb opportunities between the various arcane bets, but that involves speed more than ML.
If you know .NET, Do have a look at NinjaTrader and their forums. You will find quite a few folks who are in to automated trading.
I ventured into retail trading (manual) and lost around 5$. Email me if you are interested, I can give you more details (email address available in my profile).
Noticing comments on the high degree of proficiency of the competition(or so is the perception.)
Anyone know of any particular markets where a small timer might have a better go at it due to insufficient volume to attract the bigger players, or for any other factors?
Besides, the bigger players could easily afford to risk a few hundred grand finding out whether they can plug the existing system into the small market profitably.
of course, get MetaTrader4, open up a demo account on FXCM, Alpari, dbFX or whatever and write your own software and backtest it without losing any money. Boston Technologies has a free expert advisor, maybe you can build up on it - http://www.bostontechnologies.com/expertadvisors.html . I am not keen on automated trading though, since I do a bit of TA and FA, no software I know can do this - and where is the fun in it.
D.E. Shaw, Renaissance Tech, Goldman Sachs, etc. have already hired hundreds (if not thousands) of people smarter than you to do exactly this. They have more Math, Physics, Econ, CS PhDs than you can count/
More to the point! The big disadvantage ML methods is that they are just fancy statistics. The main faulty assumption here is that they believe the future looks a lot like the past, so they try to find even more complex patterns in historical data to extrapolate future trends slightly faster than everybody else.
In order for this approach to work, you need very precise market conditions. When markets are stable, your edge is zero and you would be better off investing in some sort hedged index (minimum effort, decent result). When markets are not that stable, it is when you can anticipate trends and get a decent profit for your effort.
However, when markets suffer big unexpected turns is when fortunes change hands... and it is when your algorithms give you negative advantage! While you are warrantied to keep following the numbers wherever they take you,a savvy finance human agent has a chance to smell the blood and discretely head for the exit before everyone else.
So, it is your call guys. In my opinion, this will not be a viable approach until we get some type of semantic-web-ly mine-able news feeder.
The reason for this is that the large companies won't talk to you unless you're trading at those volumes since they charge pretty high transaction fees. The smaller companies that will talk to you all have accounts with the larger companies and make money by increasing the spread between the bid and ask price they receive from the large companies. The expected return on a trade is usually somewhere in the region of 1-10 pips (where a pip is one ten thousandth of a dollar), with 10 pips being a really good return and 3-4 being more realistic. The smaller companies usually increase the spread by up to 3 pips in each direction on the price they get from the larger company which swallows up your return.
While millions of dollars per trade might sound like a lot the leverage in this market is also very high at 50-100 times meaning that with your 10k you could trade 500k-1M though on a sum that low you might get less leverage. However if you lose money on a trade you could eat substantially into your capital. So if your $1M trade goes against you by 10 pips you'd lose 1k.
I'm not ruling out making money on that market, but I'd say it's very unlikely unless you have a lot more money to start with. I am not an expert in this area though. This is just what I learned from the small amount of time I spent working with the company.