|
 |
| |
 |
| |
|
| |
 |
Coauthored with Ted Goertzel
Webmind Inc. was a company with a long-term mission: creating real AI. But it also, like any other company, had the goal of making money. And it wasn’t endowed with a large enough amount of cash to enable a total focus on achieving real AI, and making money after the long-term goal was complete and a real AI was constructed. We had to build products too, products we could sell in the immediate term, to increase shareholder value and fund further research. From the very start our idea with the company was to proceed on two tracks: long-term AI research and short-term AI-based product development.
The most interesting product we made was our first one, the Webmind Market Predictor -- a relatively simple software system whose purpose was to predict the daily prices of various currencies and futures, based on looking at the prices of various financial instruments over the past few days, and (this was the really innovative part) recognizing patterns among the words, phrases and concepts used in the recent news. MP (our nickname for the product) didn’t use nearly all of the Webmind AI Engine, and the version we finally productized didn’t rely on the main AI Engine codebase at all, instead relying on simplified software produced by combining more traditional statistical pattern recognition based methods with some key features extracted from the AI Engine (and specialized for the particular context of financial prediction). In this case our would-be general AI served the role of a prototyping framework in which ideas leading to a specialized AI system were developed. Not a realization of our grand goal by any means, but interesting nonetheless.
It’s not conventional to think of finance as an important aspect of the evolution of intelligence, of the path toward the Singularity. But the fact is that as computers get smarter and smarter, a lot of what they’ll be doing is managing money. The finance industry and the military have long been the two biggest users of AI. Nearly all of Hillis’s Connection Machines were sold within these sectors. And as much as the left-wing part of my mind doesn’t want to admit it, the tech explosion that’s bringing the Singularity about is essentially a capitalist phenomenon. Money is not as interesting to me as computer technology or neuroscience or genetics or ethics, but it’s certainly right at the heart of what’s happening….
A question that naturally arises at this point is: If you created such a great AI-based market prediction system, how come you’re not rich? How come Webmind Inc. went under instead of making it big on the markets? The answer to that is simple, albeit a little embarrassing in retrospect. The company made the mistake, in mid-1999, of back-burnering the Market Predictor project, and focusing the attention of its product development and marketing groups on a different product: the Webmind Classification System, WCS, which divided documents into categories automatically, and was sold to a number of customers, mostly dot-com firms.
I was never terribly thrilled by this shift in product focus, in fact I argued against it vehemently, but even so I was never 100% convinced that the decision was wrong until events had proved my intuition correct. It was our CEO Andy Siciliano who made this decision, based on his years of experience in the financial trading world, and his painful awareness of the years it can sometimes take to turn an apparently effective financial prediction system into a real moneymaker. In 1999, the dot-com boom was at its peak, and the years it could take to build up a fund based on a trading system seemed like a slowpoke’s path to fortune, compared to building some kind of Internet-focused firm and taking it public. Lisa and I and the other founders always regretted the way the MP was underemphasized within the company, but I chose to ignore this worry and focus on the AI R&D that was my main passion. Of course I also wound up spending a huge amount of time dealing with the development and marketing of WCS, even though I didn’t feel the product was the direction we should be going in. And it turns out that, although Webmind Inc., couldn’t build a business around WCS, it’s possible to create a successful smaller business around a similar piece of software. The guys from the Webmind Inc. New Zealand office have started a new company, ReelTwo, which is having some success with their Tui product, a machine learning categorization system very much in the WCS vein.
Anyway, Webmind Inc. dissolved in March 2001, due in substantial part to the failure of the WCS business model. And ironically, in one of those twists so typical of the business world, Andy Siciliano took the MP code and began using it and further developing it on his own, together with 5-6 former Webmind Inc. staff who had been on that project during late 2000 (not including me). The legal aftermath of the Webmind Inc. dissolution goes on. But the bottom line technically is that the Webmind MP does appear to work, and will potentially create a significant profit for someone over the next few years, although neither I nor the shareholders of Webmind Inc. are likely to be in this particular category of “someones.” A whole text could be written on the lessons about business that I derived from this and other experiences with Webmind Inc., but that would be a totally different book....
Investing is a lot more popular these days than it was in 1997 when I and my colleagues first designed the Webmind Market Predictor -- let alone back when I was a kid, when it was something hardly anyone ever talked about. Today everyone trades stocks, and half the cab drivers of the world seem to believe they know how to beat the market somehow -- maybe by surfing the net to find a valuable piece of research everybody else missed. Even if they can’t beat the market at least they can gamble on it. When I lived in Vegas in the early 90’s I got really baffled at the reluctance of most US states to legalize gambling – I figured it was because they didn’t want to cut off the revenue they made from lotteries. But now people have a new outlet for the innate urge to gamble, and it’s perfectly legal. The crash of the Internet bubble definitely put a damper on the “man in the street’s urge to gamble on stocks; but even post-crash, E-trade is doing a healthy business.
Some people believe there’s really nothing but gambling going on here. There’s a classic book by Burton Malkiel called A Random Walk Down Wall Street. Malkiel argues that a blindfolded monkey, picking stocks at random, could do as well as the best professional financial analysts. Since monkeys can't read, presumably the blindfold isn’t much of a handicap ... the point is that, in his view, the financial markets are not predictable.
Now one might think: “If everyone else in the market is a blindfolded monkey, then, I should be able to make a lot of money by entering the market and trading more intelligently.” But as it is now, there are so many people trying to trade intelligently, that any insight you might have has likely been roughly simultaneously exploited by a dozen or a thousand others, and hence "priced into the market." For instance, if you have good reason to believe that IBM stock is overvalued, and its price is going to drop soon, then so do others, and these others will start selling IBM at the same time as you. But this rush to sell IBM will make the price drop. So you won't gain anything by selling it, because it won't be overvalued anymore. This is called the "efficient market hypothesis" -- the hypothesis that there is no one on earth who can systematically do better on the markets than a blindfolded monkey.
Until I began studying finance with a view toward Webmind applications, this wasn't my view of the markets at all. Rather, coming from a democratic socialist family background, my perspective was that the markets were a way for the rich and well-connected to profit at the expense of the working and middle class. The markets, I was certain, were crooked. Efficient, yes -- efficient at dumping money into the laps of the richest and most dishonest individuals.
Well, I still see a certain amount of truth in this cynical view. Insider trading is more common than is widely appreciated. Maybe you have a friend who works at IBM and so you know that their new products don't work. Then you know their stock is going to drop and nobody else does. That's a way to beat the blindfolded monkey for sure. (Poor innocent beast!)
Another way, if you're an investment bank, is to create a complex financial instruments in order to get around government regulations regarding the kinds of investments mutual funds, pension funds, etc., can taken on. Risky investments in foreign currencies are cleverly disguised as investments in US government bond derivatives, and so forth, so that mutual and pension funds can "legally" buy them. Deception is everywhere. And outright illegal activities are also rampant. Mike Lissack, a friend of mine for several years, achieved some notoriety as a finance-industry whistleblower. Formerly an investment banker, he turned several large Wall Street firms into the Federal Government for defrauding the government out of billions of tax dollars. He had to hide out in an FBI safe house for a while to avoid possible assassination by his former employer.
And yet, in spite of all the crookedness, Wall Street also contains a great number of very smart people, honestly trying to beat the markets by one strategy or another. The motivation is mainly greed, to be sure -- this is not an altruist's game. But status is just as much a motivator, as well as pure curiosity, and the challenge of the task. After all, even if most trends and patterns are priced into the market, what if there's one that nobody sees but you? Maybe you can recognize from public data alone that the price of IBM stock is going to drop, while no one else is clever enough to see the truth. The efficient market view is basically that no one is this much smarter or more knowledgeable than everyone else, so that the rule holds 99.999% of the time -- the markets can't be predicted. But no trader believes this. They believe that it holds for almost everyone else, but not for the best of the best. They're smart enough, knowledgeable enough -- they can see the patterns that aren't priced in.
In the efficient markets view, which is conventional wisdom among finance professors in academia, it is not worth the cost to invest in financial advice in the hope of "timing the market" by buying stocks when they are low and selling when they are high. Malkiel's book, which came out in the early 70's and pushed the efficient market perspective hard, actually had a big impact on the market. It helped stimulate the development of index funds, which simply buy a wide selection of the most reputable stocks - those that are included in standard indexes such as Standard and Poor's or Dow-Jones. Index funds have done very well for a long time, although in the last year their performance has faltered relative to stocks of smaller, hi-tech firms.
The people out there trying to beat the blindfolded monkey fall into several different categories. First of all there are "fundamental" analysts, who study the economic fundamentals that determine an investment's true worth. How much profit has it generated in the past, and how much is it likely to generate in the future? How well is the company managed? What are the future prospects for the industries the company is competing in? These are, of course, difficult things to analyze, but many analysts are well trained and do a good job. This is what Lisa Pazer, my friend and Intelligenesis co-founder, did for a living for many years, before she quit Wall Street. She was a currency analyst. Lisa certainly believed that she was recognizing genuine patterns in market behavior, coming up with things that others didn't see. But she traded herself for years, and never made it big.
Why were Lisa and her colleagues, in spite of their insights, unable to beat the blindfolded monkey? In the efficient markets view, the answer is that, the market has already taken their analyses into account, before anyone can act on the analyses to exploit them in a significant way. Thousands of buyers and sellers, acting on this fundamental information, generally arrive at a price that reflects each investment's fundamental value. Or ... maybe they just weren't smart enough!
Of course, rational analysis of companies' prospects is not the whole story. Fundamental analysis clearly is not the sole driver of the markets! Consider, for example, market crashes such as those in 1929 and 1987. Suddenly, stock prices go down by as much as a third. Certainly, the true value of the companies' assets has not fallen that much overnight. The explanation is that markets are also a psychological phenomenon. They depend, not just on objective indicators, but on how people appraise those indicators. So you get periods of enthusiasm, when people build "castles in the air," persuading themselves that the future is unlimited for certain industries. And then the bubbles burst – just like the moving Internet bubble I talked about earlier.
Some financial analysts believe they can beat the market by analyzing the financial trends. "Technical" analysts try to do this by charting trends in stock prices. They make graphs of trends in stock prices, and believe that they can predict turning points by studying patterns in the graphs. Some use more sophisticated mathematical models rather than simple charts, especially now that computers are available to do the computations. None of these technical wizards has, however, established a really reliable track record. According to the efficient-markets view, this is because future trends in financial prices simply are not correlated with short-term fluctuations in future prices. One can predict long-term trends, within broad limits, but this is basically what the fundamental analysts do, and the results of their analyses are already incorporated in today's stock prices.
In addition to the "fundamental" and "technical" analysts, there are "behavioral" analysts who do their best to follow trends in investor opinion. They, in effect, try to psych out the market, anticipating when the climate of opinion is about to change. This is a very subtle field, depending on hunches and gut feelings, and it is difficult to test statistically. Some of these analysts have large followings, and make a lot of money selling newsletters to people who believe in their theories. They have stories to tell of great successes in predicting major turning points in markets. But many of the most successful have gone on to make dramatic bloopers. Just by luck, a certain number of soothsayers are always going to be right, but relying on the ones who were right in the past doesn't improve one's chances in the future very much, if at all.
In the 1970's, a new group of financial analysts emerged, called quantitative analysis. Unlike technical analysis which usually involves recognition of fairly simple patterns ("after three consecutive peaks, expect a big fall" and such), quantitative analysis uses highly sophisticated mathematics to analyze the markets. Practitioners are called "quants", or, more colorfully, "rocket scientists." Rocket science is big business on Wall Street, and has led to some huge successes and huge disasters. Last year, 1998, Long Term Capital Management, a hedge fund run by some Nobel Prize-winning rocket scientists, went under and lost billions of dollars. They were trading in a way that was mathematically guaranteed to succeed -- but it didn't. They lost anyway. The real world did not agree with the assumptions of their theorems. They held a number of investments that they believed to be uncorrelated, but actually, when the Russian economy crashed and a few other bad events occurred at the same time, all of their holdings simultaneously tanked.
As even this brief overview indicates, financial analysis is a very difficult and competitive field, and lots of clever schemes have failed. The market mechanism itself seems to guarantee that prices stick fairly close to their true value. But, Lisa was convinced, based on her work as a fundamental analyst, that there were patterns in the daily news that a computer could exploit for market prediction. She had been pretty good at picking up market-relevant news patterns, but she felt a computer could do even better. It could read more news than her, and study it more objectively. She convinced me, back in 1997 when Webmind was just a rough design sketch and a bunch of equations and concepts, that this was a good initial Webmind application. This would be our first Webmind "killer app" -- Webmind reading the news and predicting the markets. It was wild, it was crazy, but it made sense. Market prediction is a field where a small increase in intelligence can reap tremendous rewards.
When Webmind Inc. folded, the Webmind Market Predictor was still in a testing phase. The test were going remarkably well, however, and there was a lot of controversy within the company as to whether the business should focus on MP, or on another product that was also highly technically successful, the Webmind Classification System (WCS; a tool for dividing documents into categories). Ironically, the folks within the company who were most familiar with the trading business were eager to do with WCS, which played into the then-peaking dot-com craze, in that most of the customers for WCS were dot-com firms. Andy Siciliano, who had replaced Lisa as CEO in mid-1999, was firmly in this camp. He was all too familiar with the chancy nature of making money off the markets. On the other hand, the folks in the firm with more experience selling traditional software products were more bullish on MP, because trading seemed to them a much simpler and more direct way to make money – and they were all too familiar with problems like a long sales cycle, customer retention, and so forth. The WCS advocates won, which turned out to be a very bad thing for the company – which is easy to see in hindsight, but in foresight none of us projected the precise timing of the dot-com crash, nor its huge magnitude. We all knew the boom market couldn’t last forever, but, a smaller crash would have been survivable even with a business focused on WCS.
When the firm dissolved in March 2001, I hadn’t personally been closely involved with MP for quite some time; I’d been focusing on AI research, on endless business meetings involving fundraising and WCS, and on the design of yet another product, Webmind Search (a search engine). The MP team, a handful of people spread across the US and Australia, stayed together funded by Andy Siciliano personally. They’ve been testing the system extensively since that point, and Andy, with a fantastic finance-industry pedigree earned through years as a trader and banker, is well-positioned to transform an effective market-prediction system into a money-making trading firm. So I suspect we haven’t heard the last of Market Predictor. (And, incidentally, MP and Novamente are not the only surviving offshoots of Webmind Inc.; the core of the WCS team, the Webmind Inc. New Zealand office, has formed a new company called ReelTwo, selling a new text categorization product.)
In scientific terms, what the success of Webmind MP – and the handful of other advanced AI trading systems that are out there – shows is that the financial markets are not truly efficient. What they are is almost efficient with respect to human intelligence. There are enough smart humans trading the markets, that almost any inefficiency detectable by a smart human will be immediately detected, and priced in. But, there are inefficiencies in the market that are not easily detected by the human mind, but are nonetheless real. Webmind MP, as simple as it is to the full Webmind AI Engine or Novamente, is a non-human mind of a sort, and it can detect different patterns in the market, thus exploiting inefficiencies that humans cannot. Specifically, by reading the news and using concepts it extracts from the news to predict the markets, Webmind MP is detecting trends in human mass psychology that humans are not detecting. This is a fascinating accomplishment in itself, even if you have no interest in using it to make money.
There are dozens of AI products aimed at financial prediction -- using neural nets, genetic algorithms, and expert-system-type rules -- but most of these offer only small performance gains over the blindfolded monkey. This because they are really not all that intelligent. As artificial intelligence technology develops, we will see more and more situations like the one we currently have with Webmind MP -- exploitation of patterns in the market that no one has detected before, because there never before existed a mind with the proper orientation. Of course, if everyone started using tools Webmind MP to predict the markets, then Webmind MP's intuitions would become priced in, and the inefficiency would be gone. You'd need a new version of Webmind MP, or a different kind of AI, to gain an advantage. In the financial markets of the future, the spoils may to he or she who can develop a better, more distinctive, artificial brain.
The thing that’s unique about the Webmind MP, as opposed to other financial AI products, is its ability to analyze and synthesize both quantitative and qualitative data. It reads both text and numbers. In addition to following statistical trends and synthesizing the results of nonlinear predictive algorithms and standard financial indicators, Webmind reads the news, just as human analysts do. We simply feed in text from readily available financial news services. The system reads this news, not in an undirected, musing kind of way, but with a particular financial data set in mind, say the Dow Jones. It constructs concepts that capture themes in the news which are correlated with what the market is going to do the next day (or the next hour, or 2 weeks later, or whatever). The financial meaning of the text is thus boiled down to a collection of numbers - one for each concept extracted, representing the relevance of that concept to the text on a certain day. The numbers corresponding to the concepts can be computed anew every day, or even more often, and used for financial analysis purposes just like numbers coming from any other source. These numbers, representing the relevancies of text-derived concepts to the news at a particular time, are what we call text indicators.
The extraction of text indicators is the crux of Webmind MP's financial intelligence. It relies on Webmind MP's ability to intelligently judge relevance, which draws on all of Webmind MP's abilities at reasoning, language understanding, conceptualization, and so forth. But text indicator extraction is not the end of the story. In addition to the extraction of financially relevant concepts from news, there is an additional process of learning optimal trading models for particular financial markets. Just knowing the news concepts that tend to correlate with a certain financial market (say, knowing that trouble in foreign countries tends to drive the Dow) doesn't tell you enough to make accurate predictions. You have to get at the nonlinear interrelations between news concepts and numerical patterns in the data. This has to be done differently for the Dow, for IBM stock, for the Yen, for 30 year bonds, and so forth. For each market, Webmind derives a trading model that embodies the best way of incorporating text based information into decisions about that particular market.
Webmind's trading models are what computer scientists call "Boolean automata," simple logical decision rules, just as Webmind uses for making any kind of decision. They're basically the same kind of rules that are used, within the natural language system, to decide which sense of "Java," is intended in a sentence (the computer program, the island in Indonesia, or a copy of coffee). More generally, they are the rules that Webmind follows for learning abstract concepts and categories.
For reasons of efficiency, in building Webmind MP we adopted a special and simplified format for trading decision rules; a format derived from the prior work Jeff Pressing, an Intelligenesis co-founder. Jeff is yet another super-brilliant multitalented scientist: physics PhD; acclaimed jazz pianist, West African drummer, multi-instrumentalist and composer; currently working as a psychology professor … and does financial prediction on the side. For the few years prior to the founding of Webmind Inc., Jeff was trading the Australian bond market for a group of Australian investors, using trading rules of his own invention and making a fair amount of money. Jeff's rule format doesn't make Webmind MP perform any better than it would if it used its default decision rule module; but it does make it learn faster. In the current configuration, the system would take about an hour to learn a trading model using the generic decision module, as compared to about one minute using Jeff's streamlined framework. This is the kind of tradeoff that you face all the time doing AI engineering: the more specialized you get, the better your performance in one particular domain, but the less generalizable the performance is to other domains.
Initially, when Jeff and I developed this approach, Lisa didn't see why this decision-rule-inferring phase of the process should be necessary. "Why," she asked, "isn't it enough just to determine the concepts, occurring in news, that drive the markets? People are sheep," she said, “they just follow the herd.
My answer was simple: "People may be sheep, but they're not retarded sheep."
The serious answer, of course, is self-organization, nonlinearity. The markets are a mind of a sort -- which means that even when you know what motivates them, there is a great deal of subtlety to exactly how this motivation occurs. Just because the Dow tends to be driven by trouble in foreign countries, this doesn't mean that every time there's trouble in foreign countries, the Dow's going to jump. The conditions have to be right for this connection to manifest itself. An analogy is, suppose you've figured out that a given person is a sucker for beautiful women. This doesn't mean that every time the guy sees a beautiful woman, he's going to react in a certain way. If you want to be sure of his reaction, the conditions have to be right. You want to catch the guy at a good time of day; you want to catch him when it's been at least a few hours since he was with the last beautiful woman, etc. People, and markets, are predictable, but not linearly so. They react to complex combinations of stimuli, spread out over space and time. It takes an intelligent system, like a human or a Webmind AI Engine or Novamente, or at least a Webmind MP, to predict what they're going to do with reasonable effectiveness.
Table 2 below shows a small selection of the hundreds of amazing results we obtained in the early days of testing Webmind on financial analysis problems. In these simple experiments, we asked Webmind to learn optimal trading models for five major markets, first with and then without news-derived information. When it included the information derived from news archives, the performance increased tremendously, often by a factor of 3 or more.
|
Market
|
Entry type
|
profit/year without text
|
profit/year
with text
|
Increase in profit due to use of text
|
|
DJIA
|
Long
|
63.8%
|
172.3%
|
170.0%
|
|
Eurodollar futures
|
Long
|
22.9%
|
134.4%
|
486.9%
|
|
yen futures
|
Short
|
35.4%
|
118.3%
|
234.1%
|
|
US T‑bond futures
|
Short
|
7.8%
|
95.8%
|
1128.2%
|
|
Cocoa futures
|
Short
|
19.9%
|
30.6%
|
53.8%
|
|
| Table 2 - An illustration of the power of text based information in market prediction on 5 markets |
These results are old ones, obtained from “simulated backtesting” of the system; there are much more current results involving actual trading. But those are being held as proprietary by the current Webmind MP team; the reason I can show these results here is that they were presented by Jeff and myself in a published conference paper in 1999.
These results are dramatic in the context of trading, and they’re also of fairly obvious importance outside the financial arena, because the dynamics underlying Webmind MP 's performance on these sample tasks are the same ones underlying more general aspects of the Webmind AI Engine and Novamente. In addition to demonstrating that Webmind MP can produce highly profitable trading systems using textual information, these tests also show something more general: that Webmind MP can meaningfully extract concepts from large amounts of text data, relate these concepts to trends in numerical data, and use the relationships it has inferred to provide highly significant actionable information. This capability is of tremendous value beyond the world of finance, in areas such as risk assessment, demand planning, supply chain management, enterprise management, document retrieval and analysis, primary market analysis, medical diagnosis and research -- the list could be continued almost endlessly.
Wide implementation of this kind of financial prediction technology – which is all but inevitable, given a little time -- will have a globally dramatic effect. The financial component of the World Wide Brain will become vastly more intelligent than it is right now. I would hate to see the World Wide Brain develop with an overly financial bias -- the finance industry, for all its breadth, does after all represent a fairly narrowly biased view of the human race and all the information it has to offer. But the financial markets are already a globally integrated, perceiving and acting self-organizing system, and so they are a natural place for Internet intelligence to start.
The precise mechanisms underlying Webmind's text-based market prediction are, obviously, proprietary. They're also patent-pending: Lisa and I applied for a patent for the details of this process in mid-1998. But the basic character of the process is not a secret, as it's nothing but Webmind intelligence, applied to one particular domain. What I’ll describe here is a very early implementation of Webmind MP, back when market prediction was done in an old version of the Webmind AI Engine, before it was split off into a separate product on its own. This Webmind MP didn’t have a version number, so I’ll just refer to it as the “early Webmind MP.”
Recall the basics of Webmind architecture. Webmind, internally, consists of a collection of software objects called "nodes," each of which contains links to other nodes, representing inter-node relationships. Some nodes contain raw data such as text or numerical time series; others are more abstract and consist entirely of links to other nodes. A Webmind node is more like a "neuronal module" in the brain than it is like a single neuron. In Webmind , unlike in a neural network, link construction is carried out by a variety of intelligent software actors, and nodes and links are frequently created and destroyed as part of the learning process. Webmind 's internal intelligent actors use a variety of techniques such as genetic algorithms and statistical language processing.
In the early Webmind Market Predictor system, different types of nodes were used for representing different types of data. The node types most directly relevant to simple financial applications are:
· DataNode, which refers to a numerical data sets, e.g. financial time series
· TextNode, which refers to a text document, e.g. a market news report for a particular day
· TimeSet, which refers to a series of time-indexed nodes, e.g. a series of market news reports over several years
· ConceptNode, which refers to a concept either extracted from text or learned by Webmind
· TradingRuleNode, representing a financial trading rule
· TradingSystemNode, representing a trading system, which is a collection of trading rules
The process of learning “textual indicators” to aid in market prediction, trading and analysis was, in this early Webmind MP version, a natural outgrowth of Webmind 's intelligent self-organizing dynamics. Consider, for simplicity, the case where the market being trading is the Dow Jones Industrial Average, and the text being used is market news which is issued on a daily basis. In this case the central data structures are a DataNode pointing to the DJIA, and a TimeSet pointing to the series of market news articles. The text indicator learning process is carried out by an actor associated with these two nodes. The goal of this actor is to isolate a ConceptNode, representing one of Webmind 's internal ideas, with the property that the relevance of the ConceptNode to the TimeSet, on a given day, is a useful indicator for trading the data in the DJIA DataNode on that day. Having carried out this learning process, it then uses the ConceptNode it has isolated to create a new "text indicator" DataNode, each entry of which indicates the relevance of the ConceptNode to the market news TimeSet on a certain day. This text indicator DataNode can then be supplied to the user as, quite simply, a series of numbers. Each day, on reading the news, Webmind MP can supply a new value for the text indicator -- just as, on seeing the DJIA itself, it can supply new values for numerical indicators such as moving averages and the like.
The text indicator obtained by this process of internal conceptualization can be used for a variety of different purposes. It can be used by humans to form their own intuitive judgements, or incorporated in any computational trading framework that is sufficiently flexible to incorporate arbitrary numerical indicators. Most of our practical financial prediction work, however, involved incorporating text indicators into Webmind MP's own intelligent trading processes, which are ideally suited for incorporating text indicator information. A TradingRuleNode, in the early Webmind MP, encapsulated a trading rule, which was a logical combination of information deriving from a numerical predictor and a number of indicators, some numerical and some textual.
An indicator, generally speaking, provides a piece of information about the state of play at the time computed. There are hundreds of different indicators in common use in financial analysis today. For example, an indicator may be the value of the market series at that time, or the 5-session variability at that time, the "Relative Strength Index" of another data series at that time, or a text indicator -- the relevance of a particular concept within Webmind MP to the news at a particular time.
A predictor, on the other hand, doesn’t merely give you some information about the market -- it tries to tell you what the market is going to do. It takes in the past history of a market series, and perhaps other past information, and predicts the next value of the market series. The early Webmind MP used various types of predictors, based on such approaches as linear and nonlinear regression or pattern matching (e.g., based on fit to past history of the same market series, or changes in the series, or weighted historical combinations of various indicators). The parameters characterizing each predictor can be optimized by Webmind MP to produce maximally effective prediction. These parameters can then be stored for later recall. More sophisticated prediction in a certain market can be addressed by an optimally evolved combination of one or more predictors in that market, indicator or predictor information from other markets, and Webmind-MP-extracted text-based indicator series. Prediction can be tested by discrepancy between the actual market outcome and the prediction, or by a what-if analysis of the impacts of decision-making recommendations.
So, basically, once it had read text and made text indicators from them, the early Webmind MP used evolutionary methods to automatically "evolve" trading rules and trading systems utilizing the predictors and indicators it has at its disposal, optimized for performance on particular financial instruments. The evolution of trading rules provides a quantitative way to test the ability of Webmind to find exploitable windows of enhanced predictability for financial decision-making. Evolved trading rules may be either long or short. Trading rules may then be combined to produce trading systems, which consist of at least one long rule and at least one short rule. There are various technical criteria telling a rule when to exit the market after it’s identified a good time to enter.
A simple example of a trading rule this Webmind MP might have used is:
IF
Simplex predictor predicts > 0.25 % rise AND
Fast Stochastic Indicator (31 sessions) > 36 AND
Relevant Textual Indicator < .15
THEN ENTER LONG
EXIT after 1 session
This rule uses a predictor, one numerical indicator and one textual indicator; it combines these using numerical inequalities (involving numbers Webmind MP thought up) and logical operators (AND in this case). This rule finds a good time to enter, and then exits after one session, banking its winnings.
Basically all this work just confirms the wisdom that, while financial markets are complex and difficult to predict in general, there are windows of enhanced predictability, and if you’re smart enough, you can find them. With its text indicators and its evolutionary learning of rules that incorporate them, Webmind seems to be smart enough.
So far I’ve been fairly mysterious about one very crucial point in the "Webmind MP as financial guru" story. What exactly are these "concepts" that Webmind MP derives, by reading the news with financial data in mind? This is the most fascinating part of the whole tale! These are not simple, sensible English concepts. It's not like the Yen futures jump every time the word "hurricane" is mentioned, or every time a story about Japanese industry appears on the front page of the New York Times. The concepts that drive the market are a bit subtler and more abstract than this. Within the early Webmind MP, they were represented as abstract data structures -- ConceptNodes. They are associated with various words and documents and numerical data sets, but they are not truly described in any way except as abstract Webmind-internal data structures. Just as, while we can try to describe our intuitions about the world in words, we can never quite capture them. A good boxer, or financial trader, or mathematician can try to explain the intuitive ideas that guide his or her work, but the articulation never quite matches the reality.
For the period 1996-1997, for instance, the most prominent concepts driving the Dow and the Yen had to do with trouble in Asian economies, and Asian banks particularly. "Asian bank trouble" as a focus of the news was highly predictive of downturns in the Dow. One useful concept had to do with the relation between the British government and the Asian financial situation. Another had to do with European unification. But the important thing to remember is that the concept that Webmind MP found to be related to the market wasn't just "European unification," itself. It was, rather, a certain slant on European unification, a certain sense it got from the news, that had to do with European unification.
These are fairly concrete concepts. But the longer the time scale Webmind MP studies the news over, the more abstract its concepts become. Optimal prediction uses a combination of concrete, short-term-relevant concepts, with longer-term concepts getting at more fundamental underlying patterns. One of the most interesting concepts to come out of long-term analysis is general reluctance to take a stand. This is an internal concept which is activated whenever Webmind MP is reading news articles in which the author is apparently unwilling to state his own opinion, and is instead citing other sources, discussing the opinions of other analysts, etc. This concept is a good predictor, not specifically of increase or decrease in the market, but of increasing volatility in the market. When people start passing the buck to others, this means things are about to go nuts.
Intuitively, it seems that the long-term patterns Webmind <P picks up in its news-based financial analysis are archetypal, whereas the short-term patterns are situational. For example, we haven't actually seen this one yet, but I am quite confident that as we do more analysis of long-term news reports and market data, we're going to find the Good Guys/Bad Guys archetypes popping up. These may not come up explicitly, but they will come up as tones of expression, as general moods of the collective mind, just like the "reluctance to take a stand" collective mood that we've detected in our current experiments.
Whatever else it may be, Webmind MP is an outstanding tool for studying collective mind. It is a pattern-ometer, and an archetype-ometer. By looking at what humans write, with a specific goal in mind, it gets at the subtle patterns underlying what humans are saying and thinking at a particular point in time. This is the kind of thing that we have always sensed, but never before measured. Because this is the kind of thing it takes an intelligence to measure ... intelligence, but not necessarily human intelligence!
|
|
|
