A Gentle Introduction To Time Series
Class 75
The following conversation from an alleged Class follower (CF) and an academic statistician (AS) may or may not have taken place. Whichever, it comes at a perfect time for our entry into Time Series. Note the brief video (at bottom) has material additional to this dialog.
CF: I couldn’t help but notice the graph on your screen. Would you mind if I looked closer?
AS: Not at all. Here:
CF: If we accept the veracity of the sources going into that picture, China’s fertility rate has really fallen over 70 years, and at last joined the rest of the civilized world. One child on average per woman is low.
AS: The sources are all official, so they’re likely right. I can see why you’d say the rate as fallen, but you know we have to do a statistical test to confirm that rate has indeed dropped.
CF: A test? I don’t understand. Let’s accept the data’s accuracy. You mean we can’t say the rate has dropped from its highs in the 1950s to 1970s without a statistical test? We can’t just look?
AS: Oh, no. That’s a common error many make, so don’t feel bad about it. Yes, we have to perform what’s known as a hypothesis test. We subject the data to a rigorous mathematical test, and this tells us whether the seeming drop is real or random.
CF: Random?
AS: That happens when it only appears changes in trend occur, but which aren’t real.
CF: Not real? So this change in trend we’re seeing might not be a change at all? The rate might not have dropped?
AS: That’s right. You can never tell without first testing. But in this case, I have already done the test. The decline we’re seeing is real.
CF: What do you mean by real?
AS: The drop is statistically significant. Hold up. I can see your next question. Statistical significance is given to us by the test. It is a point at which we judge the results to be real and not random. This is measured by the P-value.
CF: P-value? You’re not making a urology joke, are y0u?
AS: Very funny, I’m sure. No, no. It is a complex number, part of the difficult math of hypothesis testing. It’s too hard to explain simply. But when the test gives us a P-value less than 0.05, we are entitled to say the results are statistically significant.
CF: So a wee P means not random?
AS: That’s one way to put it.
CF: And not random means statistically significant?
AS: That’s right. You have it.
CF: And statistically significant means a wee P?
AS: That’s it. That’s the essence of testing. That’s more or less the whole theory, without the mathematics.
CF: Is it only time series that you need to test? I mean, can you just look at other data and say whether or why changes have taken place, or do you always need to test.
AS: Very good question. No, you always have to test. Testing is the only way to say whether changes are real or merely by chance.
CF: So, going back to pictures you have, if I wanted to know whether the trend changed at any point, I could just do a test.
AS: That’s it. They’re somewhat complicated, but software eases the burden for us.
CF: Can you show me how to use it?
AS: Why not.
[At this point, a lesson in how to use time-series change point testing supposedly occurred.]
CF: All right, so I’ve done the test, and you can check my code. And if I did it right, it confirms the fertility rate changed from before the 1-child policy, then it changed again after the 2-child policy, but that the change after the three-to-any-number-of-children politics is random. Meaning it’s not a real change. Is that right?
AS: Let’s see. Yes. Your code looks right. And so is your conclusion. I’ll have to watch out for you. You learn fast. You’ll soon have my job. [Laughs]
CF: Would you say a fertility rate below replacement, which I define as one couple breeding at least two more people, ideally who themselves go on to breed, is good or bad?
AS: It would seem to be bad, if the country wants to keep its population up.
CF: So that a fertility rate of 1, as the picture indicates at the end—
AS: Hold up. Remember, we can’t say if that 1 is real or is random. The real number may be anything.
CF: Anything?
AS: Any number that’s possible I mean. That can be 0 to as many children as an average woman can have.
CF: I see. Let me ask something different. The test says that the fertility rate was significantly higher when China had a 1-child policy than when it had a 2-child policy, and that the rate was again significantly higher in the 2-child policy than the 3-child policy (and that the rate following this may be any number). That means that if China wants to increase its fertility rate, it should eliminate its latest open policy and reinstitute the 1-child policy.
AS: Ha ha! No, no. That doesn’t follow at all.
CF: Why not? The test confirmed the changes, and those changes, the information we had about them I mean, were the 1-child policy, 2-child policy and so on. Rates were statistically significantly higher in the 1-child policy, and higher in the 2-child than the 3-child. It follows, by statistical testing, that the 1-child policy produced more children.
AS: It obviously can’t work that way. Women can’t have more children when they aren’t allowed to have more.
CF: So the test is wrong?
AS: No, but it can be wrong in five percent of the cases in which it’s used.
CF: You never said before the test can be wrong.
AS: Well, unfortunately it can. That’s one of its peculiarities. But the test is likely right. It’s that it is wrong in 5% of the cases it’s used in the long run.
CF: I see. How can you tell if the test is wrong?
AS: You can never tell in any instance whether it’s right or wrong, but we do know it can only be wrong in 5% of its cases, but again only in the long run.
CF: The long run?
AS: I mean at the limit.
CF: The limit?
AS: Out at infinity. The math guarantees that 5% at infinity.
CF: I see. What’s this field called?
AS: Time series analysis.
The conversation reportedly ended here. We’ll take it up again in Class, and investigate one of the most misused and misunderstand areas of Science: Time Series.
Video
Here are the various ways to support this work:
Follow-up note - Finding useful patterns remains easy, at least for financial trading firms employing math PhDs and supercomputers plus near-zero trading costs and high leverage at low interest rates. These are trades, often across markets or between securities, with holding times of minutes or seconds.
It is easy until they blow themselves up, as Long-Term Capital Management (LTCM) did in 1998. Led by Nobel laureates and elite Wall Street traders who relied upon unexamined sophomoric mistakes. Fortunately the US Government (Federal Reserve division) bailed out them and their creditors.
A rationally-run nation not corrupted by the financial sector would have a substantial tax on transactions. Then financial markets would cease being casinos and return to socially useful purposes, and math PhDs would find new jobs.
The Emperor's new clothes are exquisite.