News, events and the financial markets have always been linked, with one reacting to the other. Whether by mathematical alchemy or intuition, the holy grail of stock market prediction has turned its attention to the almost unimaginable amounts of public data now generated on a daily basis.  As the activity that surrounds the markets grow, so have the ways of collecting and distributing its news.  In efforts to harness the full potential of this ever increasing amount of information, machine-learning is used to measure sentiment scores and is applied to financial models.  The connection between sentiment scores derived from social media and news can be applied to improved trading strategies and risk control.

Before the advent of social media and blogs, traders have always searched for improved methods of finding the best quality news in the shortest amount of time to gain a competitive advantage. Terminals include newswires with social feeds used to cross reference historical and real time indexes from which new financial models have evolved around patterns that have emerged from the coloration between events reported through these channels and market movements. Events can vary from a volcano in Iceland tweeted minutes before the newswires to individual views shared about a new product on the market. There is no doubt that publicized events can affect investor sentiment and there is an increased demand for the ability to interpret larger groups of smaller events that can be then evaluated to improve trading and risk management decisions.

The tech behind fields like big data, data analysis and machine learning has made it possible to read larger data sets and also to balance the advantages found within these growing volumes against the difficulty in extracting the relevancy from this increasing wave of information.  Advanced NLP and machine learning has taken the human element out of the equation to sort through the white noise of irrelevancy which ultimately means a clearer picture of market sentiment in a fraction of the of time. The ambiguities of language that machines (and sometimes humans!) struggle with is being narrowed down with advances in NLP, the new stars within this field hitting the headlines on a regular basis.

With real time delivery of market moving news, decisions can be automated and high frequency trading that relies on this model now accounts for around 50% of exchange trading orders.  This is a clear endorsement of the accuracy and dependability that social and news data can provide.  The interpretation of this data can also be helpful for individuals who can use this information as an added metric to augment existing tools and financial models.  Experimenting against historical data and benchmark indexes can reveal new insights into theories already being developed by individual retail investors. A viable discussion around calculating the sentiment behind different data sets and the type of actions that could be appropriated to these scores is currently underway.

Interestingly, in the case of data gathered and analysed from social media sources, high street retail brands have had a head start and are approximately two years ahead of stock market trading in the application of social analytics.  The main difference being that unlike HFT, they make human decisions on the back on new data by evaluating public opinion on products and services through sentiment scores.  By bringing the human element back into this process and building on the manual skills of directing and acting upon sentiment scores, it is possible to increase its versatility within the financial market sector beyond the successes of HFT.  The increased application of social data developed by sectors like health, government, finance and retail have resulted in new insights and academics and software companies are now well positioned to aggregate this knowledge for better performing services.