Fake data, part 3: Bypassing the central limit theorem

May 2017: This is from my other blog that's no longer online. The original comments are no longer available, but you are welcome to add more.

Now that I have my black swan distribution, and I verified that it fits market returns remarkably well while being mathematically tractable, I want to use it to generate artificial market data.

Generating a series of closing prices is easy.

  1. Select a starting price, take the logarithm
  2. Select a mean μ and standard deviation σ of log returns ln(P0/P1) where P0 is the most recent price and P1 is the previous price.
  3. Generate a uniformly-distributed random probability p between 0 and 1. Plug it into the inverse black swan distribution (using a=1.6):
    $$B^{-1}(p;\mu,s) = \mu - 2 s \, \sinh \left[\frac {\tanh^{-1}(1-2p)} {a} \right]$$
  4. Add the result to a running total.
  5. Go to step 3. Repeat as often as desired.
Then simply calculate the antilog or exponential of the values in the running total to get prices.

This works fine. But what if we want to generate more than just single prices? How do I get a series of daily price bars, with high, low, and closing values for my fake market? The way to mimic real market behavior is to generate a series of n values for each "day" and record the highest, lowest, and final price for high, low, and close of the day. That is, define Sd as a daily step and Si as an intraday step:

$$S_d = \sum_{i=1}^n S_i$$ where $$S_i = B^{-1}(p,\mu_i,s_i) = \mu_i - 2 s_i \, \sinh \left[\frac {\tanh^{-1}(1-2p)} {a} \right]$$ $$\mu_i = \frac{\mu_d}{n}, \quad s_i = \frac {s_d}{\sqrt n}$$ Black swan random walk sampling every n steps

The scaling of the intraday mean μi and standard deviation σi (corresponding to the shape parameter si) is necessary to preserve the desired mean μd and standard deviation σd for daily values. As described in the previous part, the approximation for the distribution shape parameter s is:

$$s \approx \frac{\sqrt{6}\,\sigma} {\pi \tanh^{-1} \left(\frac{1}{a}\right) \sqrt{\tanh^{-1} \left(\frac{1}{a}\right)^2+2}}$$ ...using a=1.6. But look at what happened! A disaster. The figure shows the resulting distributions for 1, 10, and 100 intraday steps. Look at the big rounded top made up of green dots representing n=100. As the number of intraday steps n grows, the resulting distribution of daily values approaches a normal distribution.

I'm back where I started! The whole point was to find something better than a normal distribution to model the markets. I don't want to end up with a normal distribution, I want to end up with my black swan distribution.

So what happened? The central limit theorem, evidently the "sovereign law" of probability theory, came in and took over. It says that the sum of many random variables with a finite mean and variance will be normally distributed, regardless of the underlying distribution we start with.

Although the black swan distribution has infinite kurtosis, it does have a finite mean and variance. This means, if we generate a bunch of tiny black swan steps to build large "daily" steps in a random walk, the large steps will approximate a normal distribution.

How can I break this law?

After much experimentation, I found a way to get past the central limit theorem. If we perturb μd to vary black-swanly with each daily step (not each intraday step), we get values that still appear distributed according to black swan, and so are the sums! $$\large \varepsilon = \begin {cases} \frac{ B^{-1}(p,0,s_d)}{\sqrt{n}\sqrt{n+1}} = \frac{\mu_d - 2 s_d \, \sinh \left[\frac {\tanh^{-1}(1-2p)}{a} \right]}{\sqrt{n}\sqrt{n+1}} & n>1 \\ 0 & n=1 \end {cases}$$ Re-generate ε once per day, and use that value for all intraday steps (if you generate a new ε for each intraday step, you end up with a normal distribution again). For each intraday step Si, use these values of μi and si: $$\mu_i = \frac{\mu_d}{n}+\varepsilon, \quad s_i = \frac {s_d}{n}$$ For large n, the adjustment to μd basically perturbs the mean each day by a small black swan distribution having a standard deviation of σ/n. For n>50 or so, one can simply use n by itself in the denominator of the expression for ε. Again, p is a random probability between 0 and 1.

Here's how it worked out:

That seems to work. I can now generate an artificial series of high-low-close prices that have similar statistical properties to actual markets. Now I must investigate whether any dependencies exist between successive values in the series.

I want to mention that the plots in this article use the Rnd() function in Excel's Visual Basic. The Visual Basic random number generator is fairly crude by modern standards, which may explain the noisy tails of the distributions shown (using 16,000 samples) versus the relatively cleaner tails of the distributions measured from actual market data (using less samples). I can't know for sure unless I try a better random number generator, but I'm not inclined to do so now — I'm satisfied with these results.

Comments

Post a Comment

Popular posts from this blog

Syncing Office 365 Outlook to Google calendar using Power Automate

Image
When my employer switched over from the Google office suite to Microsoft's Office 365, it caused unexpected disruption in my family. I had been sharing my work calendar with my wife (sharing no details, just busy times). It became an integral part of her calendar, to help her coordinate (between her job and my job) who picks up and drops off my son from school, and to let her know when I was free to take a phone call during my work day. Since the switchover to O365, my ability to share a calendar externally was eliminated. Based on documentation I could find, it should be possible, but the option doesn't appear in my employer's implementation of Outlook. I can view my personal Google calendar in my work Outlook calendar, but I can't share it the other way. Microsoft's Power Automate provides a way to solve this problem. It lets you create flows of data, triggered manually or automatically by events, to move data around, not only among the Microsoft products but ...
Read more

Whose hands are biggest? You may be surprised.

Image
A recent project to create an ergonomic handle for 3D printing led me down a path that introduced me to anthropometric measurements of the human hand, which in turn revealed some interesting facts about hand sizes. I had no idea that hand size is dependent on nationality, but I found a number of research articles on this topic. For those who don't want to read further, the answer to the title question is: Based on the data found, Filipino males have the biggest hands, by a significant margin. In fact, the margin is large enough that this population may be considered an outlier from the general human population with respect to hand size. Even Filipina females have larger hands than most males of other nationalities. On the other end of the spectrum, Vietnamese females and Indian females have the smallest hands. Now you can read on for the whole story, or you can skip down to the chart at the end. Idea for a toy My unintentional investigation into hand sizes started while de...
Read more

New approach to screw threads in OpenSCAD

Image
This is an intellectual exercise with a practical end-result. There are better approaches to this problem, but the approach I describe here is interesting. A screw thread from a stack of discs Imagine a stack of 50 coins, stacked so that each coin is slightly offset from the one below it, with the offset moving around in a circle with each new coin. Here is what it looks like for fifty coins 20 mm in diameter and 0.5 mm thick, offset by 2 mm from a center of rotation, and stacked with a rotation offset of 30 degrees per coin ( click to enlarge any image in this article ): Obviously, this is a spiral stack of coins. But it also approximates screw threads. The thread profile looks like a sinewave rather than the trapezoidal profile of machine screw threads. Indeed, others have observed this, and used the freeware parametric CAD program OpenSCAD to make screw threads by extruding an offset circle while twisting it, as in this model on Thingiverse , for example. From a distan...
Read more