Keywords: Data smoothing, noise reduction, blurring;

First of all my gratitude goes to the Director of this research, Professor Francis Bismans who wisely and intelligently guided and encouraged me in my studies. He introduced me to the Universe of Economics and Econometrics so artistically, that I fell in love with this great subject

List of Tables

List of Abbreviations

• ARDL -Auto-regressive distributed lag;

• ARIMA -Autoregressive integrated moving average;

• BB -BB algorithm for cycles dating named after Bry and Boschan;

• BBQ -BB Quarterly algorithm for cycles dating;

• BETA -Bureau d'economie theorique et appliquee;

• CDA -Conditional demand analysis;

• CDD -Cooling degree day EnMF Energy modelling and forecasting;

• CDF -Cumulative Distribution Function;

• CEnF -Commercial energy forecasting;

• CO2 -Carbon dioxide (CO 2 );

• COEF -Centre of Expertise in Forecasting;

• CoP -Conference of Parties;

• CPI -Consumer price index;

• DAElP -Day-ahead electricity prices;

• DLL -Differenced LL model;

• DME -Department of Minerals and Energy;

• DOE US -Department of Energy (US);

• ElC -Electricity consumption;

• ElCF -Electricity consumption forecasting;

• ElLF -Electricity load forecasting;

• ElMA -Electricity market analysis;

• ElMF -Electricity market forecasting;

• ElPF -Electricity prices forecasting;

• EnCF -Energy consumption forecasting;

• EnDF -Energy Demand Forecasting;

• EnMA -Energy Market Analysis;

• EnSA -Energy System Analysis;

• FBM -Fractional (or Fractal) Brownian Motion;

• FE -Full Evaluation;

• FF-ANN -Feed-Forward ANN;

• GARCH -Generalised Autoregressive Conditional Heteroskedasticity (process);

• GCM -General Circulation Model;

• GDP -Gross Domestic Product;

• GHGs -Greenhouse Gases;

• GW -Gigawatt;

• HDD -Heating degree days;

• iid -independent identically distributed;

• IPCC -Intergovernmental Panel on Climate Change;

• IPP -Independent Power Producers;

• JSE -Johannesburg Stock Exchange;

• kW -Kilowatts;

• kWh -Kilowatt Hours;

• LAR -Linear Auto-regression;

• LL -Linear-Linear (Model);

• LPG -Liquid Petroleum Gas;

• LR -Linear Regression;

• LTF -Long-term Forecasting;

• MAPE -Mean Absolute Percentage Error;

• MLS -Method of Least Squares (same as OLS);

• MW -Megawatt;

• NGO -Non-government organisation;

• NPC -Net Present Cost;

• NPGHE -Non-parametric Generalised Hurst Exponent;

• OF -Objective Function;

• OLS -Ordinary Least Squares (same as MLS);

• OT -Optimal Trading;

• PDF -Probability Density Function;

• PGHE -Parametric Generalised Hurst Exponent;

• PLC -Price less commission;

• PV -Photovoltaic;

• RCM -Reference Class Models;

• RCM -Regional Circulation Model;

• RE -Renewable Energy;

• REFIT -Renewable Energy Feed-in Tariffs;

• REnDF -Renewable Energy Demand Forecasting;

• REnSF -Renewable Energy Substitution Forecasting;

• RES -Renewable Energy Systems;

• RMSE -Root Mean Square Error;

• RSA -Republic of South Africa;

• RWP -Random Walk with Persistence;

• SAWS -South Africa Weather Services;

• S&P -Standard and Poor (rating agency);

• SPV -Solar Photovoltaic;

• STAR -Smooth Transition Auto-Regressive (Model);

• TC -Total Change;

• TV -Total Variation;

• UN -United Nations;

• UNFCCC -United Nations Framework Convention on Climate Change;

• VAR -Vector Auto-Regressive (model);

• VER -Variable Energy Resource;

• VLTF -Very Long-term Forecasting;

• WEM -World Energy Modelling;

• WEPS -World Energy Projection System;

• WES -Wind Energy Systems;

• WTI -West Texas Intermediate (grade of crude oil);

• Introduction of the concept of axiomatic definition of persistence;

• The role of the Hurst exponent in analysis of cycles;

• Establishing the link between axiomatic persistence and the Hurst exponent;

• New fast method for estimation of Hurst exponent;

• Hierarchical optimal dating of cycles in time series;

• Hierarchical estimation of time series models, including ANN estimation;

For our research we used both real data related to the South African national economy and simulated data. Wolfram Mathematica was used as the principal research tool.

Keywords: Business Cycle, Economic Cycle, Financial Cycle, Persistence, Cycle Dating, Hurst Exponent, Energy, Climate, Hierarchical methods, Artificial Neural Networks.

Résumé Cycles économiques et financiers en Afrique du Sud

Cette thèse est consacrée à l'étude des cycles. Ces derniers sont partout autour de nous, dans la nature comme dans la société. Pour autant, on s'intéressera ici exclusivement aux cycles économiques, financiers, de la demande d'énergie et même à ceux qui caractérisent le changement climatique.

Certains de ces cycles sont très réguliers et donc facilement prédictibles; d'autres, par contre, sont clairement apériodiques et de ce fait, les prévisions à leur sujet sont empreintes d'une grande incertitude. Les cycles que l'on étudiera dans cette thèse relèvent de cette dernière catégorie.

Le travail est structuré en six chapitres.

Le premier d'entre eux définit la problématique de la thèse. Il brosse, en particulier, un panorama des trois types de fluctuations étudiées en commenant par les cycles économiques. Ceux-ci se caractérisent par leur extrême irrégularité et par leur variété: cycles courts, cycles classiques d'une périodicité variable, mais égale en moyenne à dix années, cycles longs, encore connu sous le nom de Kondratieff, cycles de croissance, d'accélération, etc. Une attention toute particulière est réservée à l'étude du cycle économique classique (classical business cycle), mesuré par la série agrégée du Produit Intérieur Brut réel d'une économie et à sa datation, autrement dit à la détermination des pics et des creux, c'est-à-dire des points de retournement dans l'activité économique.

Les cycles financiers sont ceux qui affectent les marchés du même nom. D'une particulière importance sont les bourses de valeurs, qui permettent à des investisseurs d'acheter des actions de sociétés cotées. L'avantage pour ces dernières de telles transactions réside dans l'apport de capitaux neufs. Classiquement, ces marchés se subdivisent en deux compartiments : le marché primaire et le secondaire.

Enfin sont présentés les cycles dans la demande d'énergie et d'électricité. D'une manière générale, on peut distinguer trois types de fluctuations:

1. Les cycles journaliers: la demande nocturne est inférieure à celle du jour; de plus, on dénombre deux pics intra-jour, l'un au début de la matinée, l'autre en soirée.

2. Les cycles hebdomadaires: la demande est plus basse durant les weekends que lors des journées de travail.

3. Les cycles annuels: la demande d'électricité est évidemment inférieure durant les mois d'été comparativement aux mois d'hiver ; un tel cycle est prééminent dans les pays aux hivers sévères et presque inexistants dans les zones proches de l'équateur.

Le chapitre 2 est essentiellement dévolu à l'étude des modèles mathématiques utilisés pour la prévision. Selon le mot fameux de Georges Box, tous les modèles sont faux, mais certains d'entre eux sont extrêmement utiles. Ils sont faux parce qu'ils représentent une vue simplifiée de la réalité ; ils sont cependant utiles, parce qu'ils écartent les traits inessentiels, non pertinents pour la recherche envisagée.

Les plus populaires de ces constructions mathématiques sont les modèles linéaires, qui se subdivisent eux-mêmes en deux grandes catégories:

• le modèle de régression linéaire (LR);

• le modèle linéaire autorégressif (LAR).

Le chapitre présente quelques traits essentiels de ces modèles et surtout se préoccupe de la question de savoir si des processus linéaires peuvent présenter des cycles. La réponse est affirmative. Le problème de la prévision est également abordé de front en distinguant les prévisions à court terme (quelques périodes postérieures à la dernière observation disponible) de celles de long terme (un horizon prédictif de cinq périodes ou plus). Surtout, le chapitre 2 introduit un nouveau modèle, dit doublement linéaire: le modèle LL (double linear model). Celui-ci est une combinaison du modèle linéaire simple (LR) et du modèle linéaire autorégressif (LAR). Le premier peut être exprimé sous la forme simplifiée:

y 1 (t) = a + bt + e 1 (t), (1)
tandis que le second s'écrit comme suit:

y 2 (t) = y 2 (t) + bt + e 2 (t), y 2 (0) = a. (2)
Dans ces deux cas, e 1 (t) et e 2 (t) sont une suite de variables aléatoires indépendamment et identiquement distribuées, de moyenne nulle et de variance constante.

Pour sa part, le modèle doublement linéaire LL s'exprime de la manière suivante:

y(t) = py 1 (t) + qy 2 (t) (3)
où p + q = 1. Il faut également ajouter la restriction d'absence de corrélation entre les bruits blancs e 1 (t) et e 2 (t). Ce chapitre fournit les estimateurs des différents paramètres du modèle LL et en propose une interprétation.

Enfin, dans une dernière partie, les modèles d'analyse et de prévision de la demande d'énergie sont présentés dans leur diversité. Synthétiquement, ils se rangent en quatre grandes classes:

1. les modèles statistiques dans lesquels la relation fonctionnelle entre les valeurs futures et actuelles des séries utilisées prend une forme analytique précise;

2. les modèles comportementaux sont, comme l'indique leur nom, destinés à reproduire le fonctionnement réel des systèmes analysés;

3. les modèles de classe référentielle sont basés sur une analyse intuitive du problème par des experts, analyse qu'il faut ensuite quantifier;

4. les modèles hybrides qui se sont développés plus récemment, combinent différentes méthodes et de ce fait, ne rentrent nullement dans l'une ou l'autre des trois classes précédentes.

Au total, ce sont plus de cent études qui sont alors présentées en faisant fond sur cette classification.

Le chapitre 3 est entièrement dévolu au problème de la datation optimale des cycles financiers. Supposons de ce point de vue que l'on dispose d'une série chronologique y(t), où t est un entier positif, et d'un entier k. Le point y(t) sera considéré comme un pic si y(t -i) < y(t) > y(t + i), i = 1, ..., k.

En d'autres termes, le point y(t) est le point le plus élevé sur une fenêtre de taille 2k + 1.

Semblablement, mutatis mutandis, le point y(t) est un creux de la série considérée si y(t -i) > y(t) < y(t + i), i = 1, ..., k.

Le point y(t) est donc ici le point le plus petit sur une fenêtre de taille 2k + 1.

Une telle définition recèle cependant un problème majeur pour dater le cycle, car rien ne garantit l'alternance des pics et des creux. Pour résoudre cette difficulté, des règles dites de censure doivent être définies. Elles ont été systématisées par Bry-Boschan (BB) pour des données mensuelles et Harding-Pagan pour des données trimestrielles. Elles ont débouché sur des algorithmes informatiques perfectionnés, tels l'algorithme BB ou encore BBQ (Bry-Boschan Quarterly).

Cette méthode de datation des séries temporelles comporte plusieurs avantages, mais elle présente également une limitation importante : elle ne s'intéresse en effet qu'à la seule dimension temporelle en ignorant celle de y. Ainsi, par exemple, lorsqu'on a affaire à un marché d'actif (action, immobilier, matière première), l'investisseur polarise son attention, essentielleemnt sur la variation du cours de l'actif et conséquemment, s'intéresse à des fenêtres plus courtes, plus réduites.

Le chapitre 3 propose alors une méthode de datation optimale qui surmonte les limitations des algorithmes du type Bry-Boschan. L'essence de cette méthode consiste à déterminer les cycles dans une série temporelle de manière à maximiser une fonction-objectif donnée. Elle comporte sept étapes basées sur la sélection hiérarchique des points pertinents.

La méthode hiérarchique optimale est alors appliquée à 180 séries temporelles représentant les prix des actions des principales sociétés cotées sur la bourse des valeurs de Johannesburg, Afrique du Sud (JSE).

Le chapitre 4 de la thèse porte sur l'étude du phénomène de persistance ou de mémoire longue dans les séries temporelles. Elle est fondée sur la considération de ce qui est appelé l'exposant de Hurst introduit en 1951. Les définitions de cette caractéristique d'une série varient beaucoup dans la littérature et apparaissent même contradictoires.

Dans ce chapitre, une définition précise de l'exposant de Hurst est donnée. Considérons à cet effet la série temporelle x(t), t = 0, 1, .... Les différences premières de cette série, dx(t) = x(t) -x(t -1), permettent de générer la suite des déviations cumulées par rapport à la moyenne, soit

cd N (t) = t i=1
(dx(i) -m N ), t = 1, .., N, où m N est la moyenne des accroissements dx(1), ..., dx(N):

m N = (1/N) N t=1 dx(t).
On calcule ensuite, pour chaque N, l'étendue (range) de cd N (t):

R N = max t cd N (t) -min t cd N (t)
Pour chaque N, on calcule aussi l'écart-type des dx(1), ..., dx(N):

sd N = 1 N N t=1 (dx(t) -m N ) 2 1/2
L'étendue réétalonnée R/S (rescaled range) est alors définie comme le rapport de l'étendue R N et de l'écart-type sd N : R/S(N) = R N /sd N (4)

Finalement, on ajuste à la série R/S(N) la fonction puissance

R/S(N) = aN H (5)
L'exposant de Hurst H est défini comme le meilleur paramètre ajusté dans (5) lorsque N → +∞.

Le chapitre 4 aborde ensuite l'analyse de l'exposant de Hurst pour les processus LR, LAR et LL. Il présente aussi une définition purement axiomatique de la persistance qui repose sur cinq axiomes et ce, de manière à offrir une mesure idéale du phénomène. Il fournit un exemple d'une mesure qui satisfait tous ces axiomes.

Enfin, le chapitre établit la relation entre la nouvelle mesure de la persistance et l'exposant de Hurst pour les processus stochastiques connus sous le nom de "mouvements browniens fractionnaires". Il suggère également un algorithme, à la fois simple et rapide, pour évaluer l'exposant de Hurst.

Le chapitre 5 de la thèse utilise les modèles de réseaux de neurones (Artificial Neural Networks, ANN) pour analyser et prévoir la demande d'électricité en Afrique du Sud. Plus exactement, une forme particulière, mais canonique, de ces modèles est utilisée : le "feed-forward neural network model". Risquons une traduction, pas très jolie du terme : le modèle de réseaux neuronaux alimenté avant. Un tel modèle s'efforce en quelque sorte de mimer le fonctionnement du cerveau humain. Il utilise en conséquence une terminologie spécifique, notamment : entrée (input), sortie (output), couche (layer), fonctions d'activation (activation functions), noeuds (nodes), etc. Un schéma permet d'en prendre une vue heuristique dans le cas où il y a trois entrées dans le système et une seule sortie. Dans cette représentation, les neurones X's constituent la première couche, celle des entrées ou influx ; la deuxième couche (qualifiée de couche cachée) est formée des deux noeuds G 1 et G 2 ; la troisième est celle de l'output.

Le diagramme qui vient d'être donné conduit à l'équation non linéaire suivante:

Y = 2 i=1 G i (γ ′ x) (6)
où G i est une fonction d'activation, x est un vecteur colonne d'inputs et γ un vecteur ligne de paramètres, appelés "poids de connexion" dans la terminologie propre aux modèles ANN.

Il reste à présent à généraliser (6) en lui donnant la forme explicitement économétrique:

y t = α ′ z t + q j=1 β j G(γ ′ z t ) + ǫ t , t = 1, ...T, (7)
où y t est la variable dépendante (= output), z t = (1, y t-1 , ..., y t-p , x 1t , ..., x kt ) ′ est le vecteur des variables explicatives, incluant la constante et les valeurs retardées de y t , γ et α sont des (p + k -1)-vecteurs des paramètres, qest le nombre de neurones et les G's sont les fonctions d'activation. Comme de coutume, ǫ t est un bruit blanc gaussien de moyenne nulle et de variance constante.

Pour ce qui concerne la forme précise des fonctions d'activation, on supposera l'existence d'une relation fonctionnelle logistique, soit:

G(x) = 1 1 + e -x , x ∈ IR (8)
Cette fonction est partout dérivable.

Le modèle défini par ( 7)-( 8) est alors utilisé pour prévoir la demande d'électricité (mensuelle) en Afrique du Sud. Le résultat essentiel de l'exercice se résume en une phrase: l'utilisation d'un modèle ANN permet de réaliser de meilleures prévisions comparativement à celles obtenues sur base d'une régression économétrique linéaire.

Enfin, le chapitre 5 se termine par la mise au point d'un algorithme destiné à estimer les paramètres (nombreux!) qui interviennent dans un modèle ANN. La démarche s'inspire de l'algorithme développé par Halbert White et connu sous le nom de "QuickNet". Son originalité cependant est de se fonder sur une approche hiérarchique de la sélection de q, c'est-à-dire du nombre de neurones. Une première application montre que l'algorithme rend de très bonnes prévisions.

Le sixième et dernier chapitre de la thèse donne les conclusions générales du travail et aborde la question des liens entre climat et énergie. Il esquisse ensuite un programme de recherches "lyapounovien" destiné à étudier et modéliser économétriquement les interrelations entre climat et énergie.

Chapter 1

Cycles: Business, Finance, Energy

Introduction

This study is about cycles. Various cycles are all around us: day-night cycle, winter-springsummer-autumn, moon phases, high/low tides, solar activity cycle, life cycles in plants (seedplant-flower-fruit-seed), life-cycles in animals (e.g. butterfly: egg-larva-pupa-imago), ecosystem cycles, etc. Humans have their cycles too: cardiac (heart-beat and blood circulation) cycle, ovulatory cycles, mood-swings, activity-relaxation, etc. Any form of life would be impossible without cycles, as cycles are essential in the reproduction for both flora and fauna. Some cycles are very regular and therefore easily predictable. In astronomy, one may very accurately calculate positions of celestial bodies many years ahead. Other cycles are irregular (aperiodic) and therefore may be to a large extent unpredictable or very difficult to predict with certainty. Cycles which we observe in Economic studies belong to this category.

Why Cycles?

In many instances studying the cycles is preferred to studying the original data:

• Data compression/reduction: Some data-sets are really huge (especially financial timeseries). However, an investor may not be interested to see all the data. Of real interest to an investor are certain characteristics of cycles (e.g. length, magnitude) and their statistical distributions.

• Data smoothing, noise reduction, blurring: Many time series in Econometrics are quite "noisy". There are various reasons for this: random nature of data, random disturbances (e.g. strikes). Also, certain variables (e.g. economic indicators) cannot be measured directly and are estimated, using complex procedures.

• Analysis of cycles in many instances is more robust. For instance, hidden relations between two time series may be difficult to reveal using standard correlation analysis, but such associations may be revealed using cycle synchronisation techniques [START_REF] Harding | Synchronisation of cycles[END_REF].

• Benchmarking and assessing performance of investor or trader: Retrospective analysis of past cycles can reveal optimal strategies and allow finding maximum possible potential profit. Comparing this optimal profit with true profit gained gives an objective assessment of trader or investor performance.

• Modelling: Through knowledge of peaks and troughs (also called critical points), one may establish a connection between process switching and econometric variables and indicators, revealing situations which may be regarded as such that precede critical points (e.g. in Markov-switching modelling, probit regression, etc.).

• Last, but not least: Time-series data is often measured on different time scales, still one may need to simultaneously study two series recorded at different intervals. Classical analysis (e.g. correlation) requires that both sets be on the same scale and have equal length. This requirement is unnecessary for comparing cycles (e.g. for synchronisation analysis).

Cycles in Economics

Various cycles are studied in Economics:

• Economic or Business Cycle is defined as follows [84]: The business cycle is the fluctuation in economic activity that an economy experiences over a period of time. A business cycle is basically defined in terms of periods of expansion or recession. During expansions, the economy is growing in real terms (i.e. excluding inflation), as evidenced by increases in indicators like employment, industrial production, sales and personal incomes. During recessions, the economy is contracting, as measured by decreases in the above indicators. Expansion is measured from the trough (or bottom) of the previous business cycle to the peak of the current cycle, while recession is measured from the peak to the trough.

• Financial Cycle: Fluctuations in various aspects of financial markets: stocks (shares), interest rates, exchange rates, commodities' prices, real estate prices, etc.

• Stock Cycle [84]: The evolution of a stock's price from an early uptrend to a price high and eventually to a downtrend. The stock cycle is a buy-and-sell cycle that occurs over several years and has four stages: (a) accumulation (b) markup (c) distribution (d) markdown.

• Option Cycle [84]:

The expiration dates that apply to the different series of options. An option cycle is the pattern of months in which options contracts expire.

• Market Cycle [84]: Trends or patterns that may exist in a given market environment, allowing some securities or asset classes to outperform others. The securities themselves may exhibit price patterns in their trading.

• Credit Cycle, etc.

In this study we shall deal with the business cycles, financial cycles and also cycles in energy markets. We consider these cycles as the most important in today's Econometric research, modelling and forecasting. Of course, all three are closely interrelated and we shall further investigate this interrelation in our study.

Business Cycles

Modern Economic Science generally acknowledges the natural character of cycles (expansioncontraction, growth-recession) in the process of economic development. The major challenge is the erratic pattern (irregularity) of the cycles. Some cycles are very long (e.g. economic growth after the Second World War lasted over 30 years), or may be quite short (just a few quarters). A huge variety of factors are believed to influence the patterns of the cycles: political, legal, regulatory, financial policies, credit-related and even human errors. The topic is of huge interest for a wide variety of entities and individuals: governments, international financial institutions, private and corporate investors, researchers, politicians, businessmen and even the general public -everybody is affected. Substantial research is produced continuously. A wide variety of models is suggested and tested: from quite straightforward regression to most complex time-series/stochastic processes structures (e.g. Markov-switching processes [START_REF] Rabah-Rondhane | Studies in business cycles using markov-switching models[END_REF], fractional stochastic processes [START_REF] Mandelbrot | The Misbehavior of Markets: A Fractal View of Financial Turbulence[END_REF], etc.).

As a special case of a business cycle, a boom-bust cycle is also considered. This cycle is a process of fast, explosive economic expansion, followed by contraction that occurs repeatedly. The boom and bust cycle is a key characteristic of todays capitalist economies. During the boom the economy grows, jobs are plentiful and the market yeilds high returns to investors. In the subsequent bust the economy shrinks, people lose their jobs and investors lose money. Boom-bust cycles last for varying lengths of time and they also vary in severity [84].

Special attention is given to the dating of the cycles, that is defining the exact point of a peak (end of expansion, start of contraction) and the exact point of a trough (start of expansion, end of contraction). We should note that two essentially different approaches are possible here: 1. regulatory;

mathematical.

In the first case a government (or state appointed body) analyses a number of economic indicators (GDP, GVA, inflation, unemployment, etc.) and makes an executive decision. For example, in the United States, the National Bureau of Economic Research (NBER) determines the official dates for business cycles.

In the second case only mathematical criteria is used to define the peaks and troughs. The most popular method used in most research is the method introduced by Bry and Boschan [START_REF] Bry | Cyclical analysis of time series: Selected procedures and computer programs[END_REF] (essentially they built an algorithm based on ideas of Burns and Mitchell [START_REF] Burns | Measuring business cycles[END_REF]), and known as the BB-algorithm. Another popular technique stems from the BB-algorithm and is known as BBQ-algorithm [START_REF] Harding | Dissecting the cycle: A methodological investigation[END_REF]. An obvious limitation of the BB and BBQ methods is that they only take into account the length of time when growth/contraction occurs, ignoring the magnitude of the change. This approach may be unacceptable for investors and speculators who purchase assets to sell with maximum profit in the shortest term. We shall discuss this issue in detail in chapter 3.

Data Specifics

It should be noted here that most Economic indicators which are mainly used in business cycle analysis (e.g. GDP, GVA, various inflation indicators, production, unemployment rate, etc.) are estimated monthly or quarterly, and sometimes even annually. This has two implications:

1. data are relatively less volatile; 2. time series is relatively short.

The second item is especially important as many methodologies require substantial data length for accurate estimation. Another important issue is that these indicators are not available at the end of the current time period. It takes time to process the data from surveys. For example, an accurate estimate of a GDP may be available only half a year after the period in question has ended. This creates additional challenges for forecasters, who will need to forecast for the next period using from two to three three periods old data.

Financial cycles

The 2008 global financial crisis has led to increasing debates amongst economists, government policymakers and analysts on reasons why very few were able to predict such an event, and how to prevent such a global-scaled recession from recurring. One of the factors influencing such an event is the interaction between the financial cycle and the business cycle. This relationship has been researched and established in various papers. Governments even use the stock market as a leading indicator for business cycle forecasts [START_REF] Commission | Key indicators for the euro area[END_REF]. Thus it is of utmost importance to be able to model and forecast financial cycles themselves.

Financial markets play an important part in the global economy. They have a central role in economic development and in turn, economic development leads to the formation of new financial markets. They play an important part in the innovation and implementation of new technologies, promotion of growth by allowing resources to be directed to their highest return and encouraging growth after they are formed. Investors will invest in financial markets to change the composition of their savings and to encourage specialisation. An effective allocation of capital, based on an understanding of how financial markets work, will enable investors to realise highest possible returns. The competition between potential providers of financial market services result in efficient markets being formed by market participants [START_REF] Greenwood | Financial markets in development, and the development of financial markets[END_REF].

The importance of financial markets has thus led to a substantial volume of research being undertaken. Therefore, it is understandable that an interest was developed in the relationship between stock and commodity prices and this has become increasingly popular. There has especially been a vast increase in commodity investors over the past few years [START_REF] Lombardi | On the correlation between commodity and equity returns: implications for portfolio allocation[END_REF]. There is evidence that the relationship between commodities and stock has changed over the last decade, with the 2007-2008 financial crises playing a central role. Research was undertaken to look at various commodities and the related stock prices of JSE (Johannesburg Stock Exchange) listed companies and identify the relationship between them. More specifically, what type of synchronisation is present [START_REF] Tarboton | The synchronization between stock prices in jse and related commodities[END_REF]?

Commodities play an important role in financial markets and thus, the global economy. They allow companies to obtain insurance for the future values of their inputs and outputs. The risk of the future price is shifted to investors who receive compensation for bearing it. The continual advances in technology have also led to newer types of commodities being made available, such as cell phone minutes and bandwidth. The selling and buying of commodities is usually carried out by a futures contract which takes place on an exchange. A commodity is standardised by an exchange, as an exchange implements a minimum quality and quantity that the commodity must be traded at. The futures contract allows for a particular commodity to be bought or sold at a predetermined price at a predetermined future date. Some futures contracts may insist on physical delivery of the asset, whilst others may be settled in cash. An investor will benefit from the futures contract if the spot price at maturity is higher than the expected value when they first entered into the contract. An investor will, however, make a loss if the spot price is lower than the anticipated value.

The stock market allows investors to buy shares of listed companies and in exchange companies gain access to capital. It allows the financial achievements of these companies to be shared by investors for bearing the risk of unknown future cash flows. If a company is profitable, the investor will share this profit through dividends and by capital gain. These claims represent the discounted value of the cash flows into the future. The decisions made by corporate managers will influence these cash flows [START_REF] Gorton | Facts and fantasies about commodity futures[END_REF].

The stock market can be divided into two main sections: primary market and secondary market. The primary market is where new issues are first sold through initial public offerings. The secondary market is where investors buy and sell previously traded stock. The exchanges allow stock to be easily traded. Today, most stock is traded electronically and even the stocks themselves are held in electronic form, whereas in the past you would receive a physical certificate. This process is known as de-materialisation.

The exchange is a market place where stock, commodities, derivatives and other financial instruments are allowed to be traded. It provides orderly and fair trading as well as efficient spread of price information for any given security on the exchange. It gives governments and companies the opportunity to sell securities to the public and for the public to invest. Exchanges are located globally. Some examples of exchanges are: New York Stock Exchange, NASDAQ, and Tokyo Stock Exchange. Most trading occurs electronically, which allows fair trading without all members being on the same trading floor. South Africas exchange is called the JSE, the Johannesburg Stock Exchange. It is the largest stock exchange in Africa.

Similarities, differences and relationships between commodities and stock

Fluctuation of prices of both stock and commodities are (to a large extent) unpredictable, thus both may be presented by random variables. Stocks are a claim on a company that is long-lived, whereas commodities are a claim on an asset that is short term. Commodities also tend to have a seasonal component [START_REF] Gorton | Facts and fantasies about commodity futures[END_REF]. The risk associated with stock and commodities completely depend on the asset purchased. For example, in the stock market, there are a number of companies, called blue chip companies, which have demonstrated growth consistently over a number of years. These types of companies are considered conservative companies where there is low risk involved when investing in them. Similarly, certain commodities like gold and wheat are able to hold steady prices, hence they are less risky. Others, such as oil, face unstable periods and are considered to be more risky.

An understanding of the financial markets, which includes commodities and stock, is important to both investors and policy makers. Investors are interested in the relationship between commodity and stock markets as raw materials, together with different stock classes, enter many investment portfolios. Traders simultaneously look at both stock and commodity market changes to find the trend of each market [START_REF] Choi | Volatility behavior of oil,industrial commodity and stock markets in a regime-switching environment[END_REF]. Useful information is then gained about potential substitution strategies between stock and commodities by comparing the volatility of the raw materials and equity prices. Volatility plays an important role for potential hedging possibilities. It impacts the asset allocation for raw materials and their risk-return trade-off. The main concern for an investor is how to re-balance their portfolio and at what time this is optimal. This is done by purchasing "cheap" stocks when the share price is falling, called bearish periods, and selling "expensive"stocks when the share price is rising, called bullish periods. Technical analysts usually make their buying and selling decisions based on graphs. To improve investors' timing of this decision, a more thorough statistical analysis of bear and bull markets can be done [START_REF] Candelon | On measuring synchronisation of bulls and bears: The case of east asia[END_REF].

Policy makers are interested in the relationship between financial markets, as changes potentially have a destabilising effect on the real economy of a country. This raises the issue of how they, the policy makers, should respond. The following is an example of a destabilising effect: "bullish stock markets can induce large amounts of loan collateral -especially in less developed banking systems with poor regulatory frameworks -which then increase demand and goods price inflation. Moreover, when the stock market bulls turn into bears, this can result in a widespread liquidity problem and a 'credit crunch' in the financial system" [START_REF] Candelon | On measuring synchronisation of bulls and bears: The case of east asia[END_REF]. Policy makers pay particular attention to commodity prices and their volatility, given their potential to feed inflation pressures. It is, therefore, of interest to policy makers to monitor and understand the changes in the financial market in order to keep financial stability.

There has recently been a secular boom across all commodities [START_REF] Mcdermott | Is gold a safe haven? international evidence[END_REF]. This initially started in the early 2000s. Prior to the 2000s, commodity prices behaved differently to that of financial assets. The initial start was the result of collapse of the equity market and the widely publicised discovery of a small negative correlation between commodity and equity returns. This led to the belief that commodity futures could be used to reduce portfolio risk [START_REF] Tang | Index investment and financialization of commodities[END_REF]. As part of a diversification strategy, one should invest in assets that have a negative or no correlation across other invested assets in order to minimise portfolio risk. In [START_REF] Gorton | Facts and fantasies about commodity futures[END_REF] was found that there was negative correlation between the S&P 500 and equally weighted commodity futures over the period of July 1959 to December 2004 for long term holding periods.

Fund managers have started encouraging their customers to set aside a share of their portfolios to commodity related products as part of a long-term diversification strategy. The large inflow into commodity related investment products have led many commentators to speculate whether or not commodities are increasingly behaving as an asset class.

The argument is that financial investors have less commodity-specific knowledge and an attitude that is different to that of commercial traders. They will tend to trade based on their overall perception of the macro-economic situation rather than based on market-specific factors. Looking at more recent years, commodity and equity prices appear to be increasingly correlated with each other. This is confirmed in a paper by Bhardwaj, et al. who found there has been an increase in correlation from the beginning of 2005 to the end of 2014. It increased to above 0.5 [START_REF] Bhardwaj | Facts and fantasies about commodity futures ten years later[END_REF]. The increase in correlations could be explained by the fact that equity prices are likely to have been driven by global economic activity shocks, which is probably the reasons behind the increase in correlation for the 2007-2008 financial crises.

There has also been evidence of intra-commodity correlation within the commodity class. Both stock-commodity and intra-commodity correlation appear to be more sensitive to news concerning the global macro-economic environment, rather than distinctive market-specific shocks. Policy makers are currently actively considering tougher regulations to control the destabilising speculation, due to the effect that it is having on price volatility [START_REF] Tang | Index investment and financialization of commodities[END_REF].

Besides speculators, firms use commodities as variable inputs for production. During bad economic times firms tend to quickly cut back on variable costs in order to either avoid bankruptcy, or in response to constrained credit conditions. Firms may regard falling stock prices as an indication that things are getting worse and vice versa, as leading to an increase in the stock commodity correlation [START_REF] Bhardwaj | The business cycle and correlation between stocks and commodities[END_REF].

In the paper by [START_REF] Creti | On the links between stock and commodity markets[END_REF] the stock market was compared with the commodity market. The data they used for commodities was the daily spot price for a sample of 25 different commodities over the period 3 January 2001 to 28 November 2011. They also considered the aggregate commodity price index and the Commodity Research Bureau (CRB) index. With regards to the equity market, they relied on the S&P 500 index, which is one of the main indices for the US stock market. The model they used was the DCC-GARCH (Dynamic Conditional Correlation-Generalised Autoregressive Conditional Heteroscedasticity) model, which was used to investigate the correlation between the commodity and stock markets over time [START_REF] Creti | On the links between stock and commodity markets[END_REF].

In their findings, three common characteristics emerged. Firstly, correlations were highly volatile throughout the studied period. The volatility in correlation increased, for all cases, during and after the 2007-2008 financial crisis.

Secondly, in most of the cases, the largest drop in correlations appeared at the time of the 2008 financial crisis. What we see in the very short run is that when the stock market collapses, it loosens the conditional links between commodity and stock price returns. It is suggested that the decrease in correlation during financial markets distress may be linked to the flight-toquality phenomenon. When markets become risky, the benefit of diversification is appreciated and investors will tend to turn to commodities as refuge instruments. This is especially true for gold.

Thirdly, for almost all of the series, the highest correlation is observed after the financial crises, at the end of the period under study. The highest correlation was usually found when both markets were in an upward direction, which generally happened during episodes of growing worldly demands for industrial commodities. At these times it gives commercial traders, who use commodity futures to hedge their business activities, an important role.

Daily data running over almost the same period, from 3 January 2000 to 31 December 2011, was used in the literature by Mensi, et al. They used the S&P 500 index and the commodity price indices for beverage, wheat, gold and two benchmarks for crude oil: Cushing West Texas Intermediate (WTI) and Europe Brent. A VAR (1) -GARCH (1, 1) (Vector Autoregressive-Generalised Autoregressive Conditional Heteroscedasticity) model was used to explore the joint conditional returns, volatility and correlations between the commodities and S&P 500 [START_REF] Mensi | Correlations and volatility spillovers across commodity and stock markets: Linking energies, food and gold[END_REF].

They found that the highest market reaction to the change in the S&P 500 price was firstly gold, then WTI crude oil and wheat. The effect of a change in the commodity market returns on S&P 500 was significant for all commodities. When analysing volatility interdependence, they found that there was significant volatility spill overs between crude oil and the stock market. A spill over is the effects of economic activity or process on those who are not directly involved in it. The spill overs increase during crisis periods due to the economic uncertainties and financial instability. The constant conditional correlation was positive between the stock market and all commodities. It however, was highest between S&P 500 and gold, then WTI crude oil and wheat. The Gold and WTI crude oil markets have the most volatility transmission from the S&P 500 market. It is evident that the gold and WTI crude oil markets have the most noticeable relationship with the S&P 500 [START_REF] Mensi | Correlations and volatility spillovers across commodity and stock markets: Linking energies, food and gold[END_REF].

Looking at the more specific types of markets, it is suggested that the stock market has less of an effect on agricultural commodities than on industrial commodities, such as base metals and oil [START_REF] Bhardwaj | The business cycle and correlation between stocks and commodities[END_REF]. In both of the above reviewed papers, Creti, et al. and Mensi, et al., oil was found to be the commodity most related to the stock market from the energy group.

The dominance of oil may be due to oil being one of the most important factors for production. During periods of increases in stock prices, the correlation between stock and oil markets increases. During periods of declining stock prices, such as during the 2007-2008 financial crises, the correlation between stock and oil tend to decrease and become negative [START_REF] Creti | On the links between stock and commodity markets[END_REF]. The overall response of the stock market differs greatly depending on the underlying cause to the oil price shock and from this knowledge we can make the appropriate portfolio adjustments. The variability for the returns of U.S. real stock associated with global crude oil shocks will increase in explanatory power as the horizon length is increased [START_REF] Kilian | The impacts of oil price shocks on the u.s. stock market[END_REF].

In the precious metals group, researchers found that the behaviour of gold was different when compared to other commodities. The correlation was predominantly negative and diminished over time during periods of declining stock prices [START_REF] Creti | On the links between stock and commodity markets[END_REF]. This highlights the adverse evolution of the markets and is consistent with the role of gold being a safe haven. It is suggested that gold is a safe haven for most developed countries, but at most, a weak safe haven for emerging markets. Investors suffering losses in emerging markets may simply readjust their portfolios by withdrawing from emerging markets in exchange for developed markets [START_REF] Mcdermott | Is gold a safe haven? international evidence[END_REF]. For other precious metals, it has been found that a similar pattern is followed. After the 2007-2008 financial crisis, there was an increase in the volatility of the correlations.This was followed by a rise in correlation until mid-2010. Other commodities also displaying a specific profile within the US stock market were coffee and cocoa, whereas the correlation for sugar displayed no particular pattern [START_REF] Creti | On the links between stock and commodity markets[END_REF].

From the literature it is confirmed that the relationship between stock and commodity markets has changed significantly since the 2007-2008 financial crisis. There has also been an increase in the correlation of volatility. With respect to the long run trends for correlation, they are most likely to be influenced by industrialisation and financialisation processes, as well as commercial and non-commercial trades [START_REF] Creti | On the links between stock and commodity markets[END_REF].

As we have previously outlined, an understanding of the cyclical movements of financial markets will be valuable to all stake-holders. Due to the increase in financialisation in the world economy, we can understand that there has been an increase in interest regarding the methods used to measure the business cycle. In the field of statistics, and especially the economic field, we often encounter problems where the answer to whether or not a relationship is present can be of great assistance when trying to solve the problem. The problem then extends as to how to measure the relationship or synchronisation between the cyclical series. For example, when countries are trying to form a monetary union, such as the the EU, they would look at whether or not their business cycles are coordinated or synchronised [START_REF] Harding | Synchronisation of cycles[END_REF].

The strength of the relationship between variables can be measured using a variety of statistical coefficients, depending on the type of analysis which will determine the method that should be used. These coefficients are referred to as the 'measures' of association. One method used is finding the correlation coefficient between series of data. There are three well known types of correlation that can be measured: the Pearson correlation, the Kendall rank correlation and the Spearmans rank correlation.

The Pearson correlation is the most widely used statistic. It is best suited for continuous, normally distributed data. It is a very unreliable measure if there is a nonlinear relationship between the two sets of data. The Spearmans rank correlation and Kendall rank correlation are non-parametric measures of the statistical dependence between two ranked sets of data. They assess how well the relationship can be described using a monotonic function. We can also investigate the co-integration between the sets of data. The co-integration between two sets of data indicates the relationship between time series in equilibrium that individually are not in equilibrium. It will allow us to see both the short-term dynamics and deviations from the equilibrium, and the long-run expectations, corrections to the equilibrium. It is only defined when the error term in the regression model is stationary. The above approaches, however, do not correctly indicate the degree of association in a general sense, for fluctuating data, and hence in cycles. Some other methods used to measure the dependencies of variables are: the Hoeffdings Dependence coefficient, distance correlation, mutual information and the maximal information coefficients. These measures are viable, but do not produce a desirable interpretable outcome in all situations. We, therefore, need to search for a measure that is both interpretable and that possesses the desirable qualities we want [START_REF] Clark | A comparison of correlation measures[END_REF].

Harding & Pagan suggest a more robust method that uses existing cycles in a series to measure the degree of association -so called synchronisation. How they achieve this, is by translating the information from a continuous random variable into a binary random variable. From this new binary variable the correlation coefficient can then be calculated. The results are easy to interpret and provide a very good indication of the relationship between the sets of time series data [START_REF] Harding | Synchronisation of cycles[END_REF].

The method suggested by Harding & Pagan overcomes the problem of capturing the cyclical behaviour and finding the measure of association in fluctuating time series data. These methods can be extended to any cyclical series [START_REF] Harding | Synchronisation of cycles[END_REF].

Elliott Wave

The first detailed analysis of self-similar cycles in financial time series was conducted by Ralph Nelson Elliott in 1930s [START_REF] Elliott | elliott's masterworks[END_REF]. Surprisingly, Elliott was neither Economist nor Statistician nor Mathematician. He was an accountant. He explained his theory by social psychology (a.k.a. psychology of crowds) principles.

The Elliott Wave Principle refers to the theory that social behaviour drifts back and forth in recognisable patterns. Using stock market data as his main research tool, Elliott noticed that five-wave and three-wave patterns of movement occur repeatedly in market price data. He named, defined and illustrated those patterns and described how these waves link together to form larger versions of those same patterns; how those in turn link to form identical patterns of the next larger size, and so on.

Frost and Prechter [START_REF] Frost | Elliott wave principle[END_REF] provide a very detailed explanation of the rules and process of identifying these patterns. They say that five-wave patterns are called impulsive waves and follow the trend of the larger degree wave while three-wave patterns are corrective waves and go against the larger degree trend. They also mention several variations in the basic impulsive and corrective wave pattern. Based on these patterns, they can forecast the most probable direction and magnitude of the next wave to occur, thus also predicting the peaks and troughs that arise at the end of those waves. Figure 1.1 illustrates those concepts. • the big 5-3 wave cycle consists or four Λ-shaped cycles;

• each Λ-cycle has two waves; the longer one is known as the motif and the shorter one is known as the correction;

• the first five waves may be seen as a bigger (jagged) motif, while the last three waves constitute a bigger correction

In our view, the following facts related to the Elliott's principle are exceptionally important:

• Elliott instituted foundations for self-similarity in financial cycles: each wave contains four smaller waves and itself is a part of a bigger wave (see figure 1.2);

• The Elliott wave has direct connection to Fibonacci's numbers and Golden Ratio, both widely used in the financial theory;

• The Elliott's principle is often criticised by modern researchers and technical analysts: technologies, governments, economies and social systems have changed, and so has the behaviour of people. These changes have affected the wave patterns R.N. Elliott discovered;

• However, the terminology of Elliott is still in everyday use (e.g. "wave", "motive", "correction"). New publications with suggested upgrades still appear regularly; A stochastic process received by multiple replacements of the Λ-cycles by smaller Elliott waves will be called Elliot Wave Process (EWP). The most remarkable property of this process is the fact that every point of this process is a turning (critical) point. This fact is important for us and will be referred to in the section 4.12.

While the Elliott Wave Principle is still used today by chart analysts to predict peaks and troughs of the stock and commodity markets, it remains a largely subjective method and relies on experience and observatory analysis.

In contrast to the Economic indicators, observations of financial variables are available at much shorter intervals. For instance, the all-share index is available practically continuously and may change several times during a second. As a result, the data-sets may be really huge and exceptionally volatile. Several significant cycles may be observed in a single day or even within an hour. So, if one wants to study two sets simultaneously, say, one recorded daily and another recorder per second, the researcher often has to bring two sets to the same time-scale. The first possibility is to impute observations into a daily data-set, which will mean that most of the data will be artificially generated and therefore not reliable. The second possibility is to enlarge intervals for per-second data. This may be done in a number of ways:

• Take the last/first, "closing"/"opening" value;

• Take highest/lowest value;

• Take some kind of average (e.g. weighted average): several possibilities exist;

• Other possibilities also exist, depending on the purpose of the analysis;

We would recommend, however, that neither of the above is used in the analysis of cycles. The cycles may be dated by means of one of the dating techniques (e.g. blurring) using original datasets. We will demonstrate later how this may be done in practice. When the cycles are defined, the original time intervals are not relevant.

Cycles in Energy/Electricity Demand

Three absolutely obvious cycles in Electricity demand can be observed in practically any part of the world:

1. Daily cycles: demand at night is naturally lower than during the day due to much lower domestic, commercial and industrial activities. There are two peaks in the demandmorning peak and evening peak (see figure 1.3).

2. Weekly cycles: demand during weekends is lower than during working days due to lower commercial and industrial activities on weekends.

3. Annual cycles: demand is lower during summer than during winter, purely due to the need for heating. This cycle is more prominent in countries with severe winters and is almost absent in equatorial areas.

Figure 1.3: Typical Daily Profile for Electricity Demand in RSA (source [START_REF] Mangisa | Statistical analysis of electricity demand profiles[END_REF]) .

In addition to the above, we have to also consider public holidays, when the electricity consumption drops significantly.

Apart from these cycles (which have very well defined periods), there are so-called repeating events. The repeating events take place regularly, but do not have constant period. The best examples of these are religious holidays, such as Easter (in Christian culture) and Ramadan (in Islamic culture). These holidays occur every year but the dates fluctuate. Apart from these other repeating events, which impact on the electricity demand are (for South Africa): election days (municipal and national), school holidays (at all levels up to tertiary), exam periods, etc. One may also consider adding to this list strikes (labour actions), especially in energy consuming industries (in South Africa: smelters, especially aluminium, mining, etc.).

A study was undertaken to investigate to what extent the cycles and repeating events define daily fluctuations in the electricity demand in South Africa [START_REF] Nyuly | Weather neutral models for short-term electricity demand forecasting[END_REF]. We developed a set of so-called weather-neutral models, which are VAR models where all of the independent variables were calendar-related. All of the variables, except for one (time) were categorical. The variables were: month, days of the week, public holiday, Christmas, Easter, day before a public holiday, day after a public holiday, school holidays, university exams and major religious holidays. We purposely excluded from analysis all econometric and weather variables, as our goal was to see how accurate weather-neutral models could be. Our data covered the period from 1 January 1996 to 26 July 2009.

All of the variables were found to be significant. We expected that the above variables would have a serious impact on the electricity demand but the results far exceeded our expectations. The MAPE (Mean Absolute Percentage Error) of the model was 0.9%. This means that by using only a calendar as a source of data one could predict the daily demand for electricity in South Africa with an average error margin of under 1% (see figure 1.4). We also modelled the daily cycles [START_REF] Mangisa | Statistical analysis of electricity demand profiles[END_REF]. Our model was based on two erf-function curves:

demand(t) = b + a 1 f( t -t 1 σ 1 ) + a 2 f( t -t 2 σ 2 ), 0 ≤ t ≤ 24, (1.1)
where:

f(x) = 1 √ 2π e -t 2 /2
. This allowed us to dissect the daily demand into the following seven components (see figure 1 The suggested model showed an exceptionally good fit. In 99% of cases, the coefficient of determination R 2 exceeded 98% (see figure 1

Structure of the Thesis

In this introductory chapter, we discussed general issues related to the topic of the study. Specifically, we emphasised the importance of studying cycles in Economic, Business, Financial and Energy studies. We also discussed other research preceding this one.

In the second chapter, we discuss the general principles of mathematical modelling and give a brief introduction to linear regression and auto-regression. Then we introduce the double linear (LL) model and study its properties and possible applications in modelling econometric data. The chapter concludes with a detailed critical review of the literature on the models used for the forecasting of energy demand.

The third chapter deals with the methods of dating of cycles. The hierarchical method for cycles dating is introduced and motivation is provided for when this methodology is useful and will over-perform other techniques. We show the use of this approach for dating cycles of share prices listed at Johannesburg Stock Exchange (JSE).

The next (fourth) chapter discusses the issue of persistence in time series, which may be seen as an opposite to cycling. We discuss the Hurst exponent as it was and is still widely used for measuring persistence. We introduce methodologies for calculating the Hurst exponent for LR, LAR and LL processes. This allows us to reveal the shortcomings of using Hurst exponent for these linear models. Further, we suggest a new approach based on axiomatic philosophy. Hence, we introduce a new parameter for persistence measurement. We establish the relationship between this new concept and Hurst exponent. This allows us to generalise the Hurst exponent.

In chapter five, we discuss the artificial neural networks (ANN) as a model for forecasting cyclical data. Thereafter we use the hybrid model for analysis and forecasting electricity demand in South Africa. Finally we introduce a new method for estimating ANN models' parameters, which is also based on principles of hierarchy. We discuss the pros and cons of the ANN approach in econometric modelling.

In the concluding chapter the study sets a path for analysis of closed-loop models involving mutual interrelations between weather/climate and energy sector. This is followed by a general conclusions to the thesis.

Chapter 2 Models

Introduction to Scientific Models

All models are wrong, but some of them are extremely useful. This famous saying of George Box [START_REF] Box | Empirical Model-Building and Response Surfaces[END_REF] emphasises two important features of a scientific model:

• A model represents a simplification (in simulacra) of a real object or process. From this perspective any model is "wrong", as it disregards certain properties and features of the reality;

• Models are useful because they are supposed to disregard only irrelevant, unimportant features of an object or process making it much easier to understand, visualise, analyse and simulate the reality. Not to forget that studying a model is normally less dangerous, much cheaper and and holds less risk than studying an object or process itself.

A perfect model comes as a result of a balance between the principles of parsimony and adequacy. A parsimonious model is as small and simple as possible in given circumstances (some scientists also appreciate aesthetic appeal). A model is adequate if it retains all properties of the real object which are essential for a given study. So the art of modelling is in making a perfect separation between the irrelevant, which may be discarded, and the important, which should be retained and analysed. Especially important is preservation of existing causal mechanisms.

There are many approaches in scientific modelling: deterministic and probabilistic, scaling, iconic (pictorial), operational, structural, in vitro, etc. However, in this study, we deal exclusively with mathematical models; that is models which may be presented by numbers, variables, parameters, functions, inequalities and equations (of all kinds) and other mathematical objects.

Linear Models

Among the mathematical models the most popular are linear models, which are models that contain only linear functions and relationships. The reasons are obvious:

• These models are simplest (remember about parsimony!);

• Many non-linear models may be reduced or transformed to linear ones;

• Often linear models represent acceptable approximation of nonlinear ones;

• Linear models make a great departure point. Subsequently, they may be upgraded and generalised to more complex models.

Despite their seeming simplicity, linear models are not yet fully discovered. Later in this chapter we introduce a quite simple new linear model (LL-model or double linear model), which was introduced and analysed only recently [START_REF] Litvine | Ll model -theory and applications[END_REF].

Linear Models in Time Series Regression: Definitions 2.2.1 Linear Regression Model (LR)

Suppose we have a time series (TS) y 1 (t), where t = 0, 1, 2, ... -discrete time. The linear regression model is defined as:

y 1 (t) = a + bt + e 1 (t) (2.1)
where e 1 (t) is a sequence of independent identically distributed (idd) random variables with zero mean and variance σ 2

1 . The e 1 (t) are referred to as random errors of the model.

Linear Auto-Regression Model (LAR)

The linear auto-regression model is defined as:

y 2 (t) = y 2 (t -1) + b + e 2 (t) (2.2)
y 2 (0) = a
where e 2 (t) is a sequence of independent identically distributed (idd) random variables with zero mean and variance σ 2 2 . As before, the e 2 (t) are referred to as random errors. It is also very common to assume that the errors are normally distributed.

Since the above is true for any t, one may easily derive that:

y 2 (t) = a + tb + ǫ(t) (2.3)
where ǫ(t) = (e 2 (1) +... + e 2 (t)). The above may not be regarded as a linear regression model because (a) the errors are dependent and (b) the errors are not identically distributed.

Expected Values

Both series have exactly the same mean at any point t:

E(y 1 (t)) = E(a + bt + e 1 (t)) = E(a) + E(bt) + E(e 1 (t))) = a + bt (2.4) E(y 2 (t)) = E(a + tb + (e 2 (1) + ... + e 2 (t))) = E(a + tb) + (E(e 2 (1)) + ... + E(e 2 (t))) = a + bt (2.5)
Since both expected values represent straight lines, both models are referred to as linear models. The two models have some similar features, but also differ significantly in certain aspects. We shall discuss similarities and dissimilarities in the following sections. We can also see that both models are non-stationary for b = 0 (e.g. since the mean depends on t).

Simulations

The simulations provide a very good visual idea of what we mean when saying that the series are quite different. The two graphs below show two simulations of a regression model: One can see the difference in the behaviour between the two types of the simulations with a naked eye. One clear observation is that the variation in the auto-regression is visibly higher. Let's compare the variances of the two models.

Variances

The variance of y 1 (t) is:

V (y 1 (t)) = V (a + bt + e 1 (t)) = V (e 1 (t)) = s 2 1 (2.6)
The main observation here is that the variance does not depend on the time t.

The variance of y 2 (t) is (we use presentation 1.3):

V (y 2 (t)) = V (a + tb + (e 2 (1) + ...e 2 (t))) = V (e 2 (1) + ...e 2 (t)) = = tV (e 2 (i)) = ts 2 2 (2.7)
The principal difference with the previous result is that the variance is growing with time.

Independence, Covariance and Correlation

It is obvious that y 1 (t) does not depend on y 1 (t -1), and consequently, y 1 (t 1 ) does not depend on y 1 (t 2 ) for any t 1 = t 2 . This is not the case for y 2 . Auto-covariance (at lag 1) for y 2 is:

cov(y 2 (t -1), y 2 (t)) = cov(y 2 (t -1) + b + e 2 (t), y 2 (t -1)) = cov(y 2 (t -1), y 2 (t -1)) + cov(e 2 (t), y 2 (t -1)) = V (y 2 (t -1))
since e 2 (t) and y 2 (t -1) are naturally independent. So, as follows from the above and (1.7):

cov(y 2 (t -1), y 2 (t)) = s 2 2 (t -1) (2.8)
Now we can calculate the correlation coefficient at lag 1:

ρ 2 (y 2 (t -1), y 2 (t)) = cov(y 2 (t -1), y 2 (t)) V (y 2 (t -1))V (y 2 (t)) = s 2 2 (t -1) s 2 √ t -1 s 2 √ t = t -1 t For conciseness we shall denote ρ 2 (t -1, t) = ρ 2 (y 2 (t -1), y 2 (t))
, so:

ρ 2 (t -1, t) = t -1 t (2.9)
One can see that the correlation is growing with t and approaches one as t goes to infinity. Another important observation here is that the correlation does not depend on s 2 2 , a or b.

Higher Lags Correlations

It is easy to show that if t 1 < t 2 , then:

cov(y 2 (t 1 ), y 2 (t 2 )) = V (y 2 (t 1 )) (2.10)
Therefore the correlation coefficient:

ρ(y 2 (t -k), y 2 (t)) = V (y 2 (t -k)) V (y 2 (t -k))V (y 2 (t)) = (t -k)s 2 2 s 2 √ t -k s 2 √ t = t -k t (2.11)

Discussion

We see that the correlation decreases with lag k growing, however, this decline is much slower than in case of stationary models (see section below).

Another inconvenience is that the correlation depends not only on the lag, but also depends on t (which is not the case for stationary processes). This may be of some inconvenience for further analysis. One can try to use other measures of association, e.g. Kendall's τ or Spearmans' ρ.

Conditional Distribution of LAR Process

It is evident that the conditional distribution of y 2 (t 0 + k), given that we know y 2 (t 0 ), is the same as the distribution of y 2 (k) (if we set a = y 2 (t 0 )). From this we can find all conditional characteristics of the process. For example:

E(y 2 (t 0 + k)|y 2 (t 0 )) = y 2 (t 0 ) + bk V (y 2 (t 0 + k)|y 2 (t 0 )) = ks 2 2
(2.12) etc.

Autocorrelation of a Stationary Process

The equation (1.2) may be generalised as follows:

y 3 (t) = φy 3 (t -1) + b + e 3 (t)
(2.13)

y 3 (0) = a
In case of y 2 , φ = 1. However, if |φ| < 1, the process will be stationery. To be exact, the process becomes stationary after initial transition period. In other words, the process is stationary for large t. In this case the mean and variance of the stationary process are:

E(y 3 ) = b 1 -φ V (y 3 ) = s 2 2 1 -φ 2 (2.14)
Both expressions in (2.14) become undefined if φ = 1, like in (1.2). In a sense this agrees with the above, as both mean and variance of y 2 (t) approach infinity if t → ∞

The autocorrelation (at lag k) for the stationary process (2.13) is:

ρ 3 (k) = φ k (2.15)
If φ = 1, this will mean ρ 3 (k) = 1. And this again agrees with the (1.9) as ρ 2 (t -1, t) also approaches one if t is large.

Differencing

Differencing converts series with linear trend into stationary models. Let's look closer.

Differencing LR

∆y 1 (t) = y 1 (t) -y 1 (t -1) = bt + e 2 (t) -b(t -1) -e 1 (t -1) = b + (e 1 (t) -e 1 (t -1)) (2.16)
The above may be interpreted as b plus first order moving average process (MA(1)):

ma(t + 1) = e 1 (t + 1) -e 1 (t)
where t = 1, 2, .... This process is stationary, but not invertible (that is, it cannot be converted into equivalent autoregressive (AR) process).

One can easily evaluate all the parameters of ∆y 1 (t):

E(∆y 1 (t)) = b V (∆y 1 (t)) = 2s 2 1 cov(∆y 1 (t), ∆y 1 (t -1)) = -s 2 1 ρ(∆y 1 (t), ∆y 1 (t -1)) = -0.5 ρ(∆y 1 (t), ∆y 1 (t -k)) = cov(∆y 1 (t), ∆y 1 (t -k)) = 0 (2.17)
where k = 2, 3, ...

The differenced process has questionable possibilities for forecasting purposes as the R 2 is very small (R 2 = (-0.5) 2 = 0.25). That is, only 25% of variation in ∆y 1 (t) may be explained by ∆y 1 (t -1) Also it is not clear how to forecast, as the equivalent AR process does not exist.

Differencing LAR

∆y 2 (t) = y 2 (t) -y 2 (t -1) = b + e 2 (t) (2.18)
In other words, the differenced process is a sequence of independent identically distributed random variables with mean b and variance s 2 2 . As this is not (strictly speaking) a time series, any forecasting better than predicting the mean (b) is not possible.

Estimation

Suppose we have n observations over y 1 (t) and y 2 (t) (t = 1, 2, ...n). The two models may not be estimated from the observations using the same technique.

Estimation for the LR Model y 1

The best estimation of a and b for the linear model y 1 is done using the method of least squares (MLS):

b′ = ĉ ov(y 1 , t) Ŝ2 t , (2.19
)

â′ = ȳ1 -b′ t, (2.20)
where ĉ ov(y 1 , t) is the sample covariance between y 1 (t) and t (t = 1, 2, ..., n), Ŝ2 t is the sample variance of the series 1, 2, ..., n, ȳ1 is the sample mean of y 1 and t -sample mean of t:

Ŝ2 t = n(n + 1) 12 ȳ1 = 1 n n t=1 y 1 (t) t = 1 n n t=1 t = n + 1 2
This method of estimation has a number of optimal properties.

Estimation for the Auto-Regression Model y 2

An attempt to use the same approach for the AR model (y 2 ) may lead to disastrous results. In the figure below we can see an example of the AR process (broken line), the expected value of the true model (straight line in blue), and the estimated regression line (in red). One can see that both the slope b and the intercept a are estimated in essentially the wrong way. The reason for that is the violation of the LR assumptions (i.e. independence of errors).

The correct method for this model is to find b as the mean of the differenced series: The above estimate is:

1. Unbiased. 2. Has variance V ( b) = V ( e 2 (t)/n) = s 2 2 /n. 3. Consistent.
The parameter a may be estimated as:

â′′ = y 2 (1) -b′′ (2.22)
There are some issues with the estimate for a:

1. It is not consistent (the estimate is not converging to a if the sample grows)

2. If observations started not at t = 1, but at some later stage (e.g. t = k), the estimate becomes: â = y 2 (k) -k b. The accuracy of this estimate for large k is very low.

3. A researcher may always move the origin of time to the time of the first observation. Essentially this implies that the estimate for a is not unique, but depends on the researcher's discretion. We will see later that this may not impact on the accuracy of a forecast. For the sake simplicity, in what follows, we will always assume that the first observation was done at t = 1.

Do Linear Processes Have Cycles?

The question in this section has a seemingly easy and natural answer: "No". Linear processes are usually used to model trends, not cycles. However, we shall challenge this concept. If linear processes are disturbed by random impacts, they are preset to have cycles. Below we provide images of two random simulations: LR (figure 2.4) and LAR (figure 2.5) processes. The cycles were dated using the BB-algorithm with k = 4 (see section 3.2 for details of the algorithm). One should remember that the simulated processes are linear by definition and therefore there is no regime switching as such. If we reject naive assumptions (e.g. that the dating technique is invalid), we have to conclude that cycles are hiding literally everywhere and we need to improve our approaches of studying them. In other words, any analysis or time series that ignores presence of cycles is at least incomplete.

Forecasting: Introductory Comments and Basic Concepts

The aim of forecasting is to predict (as accurately as possible), future values of a time series. The forecasting horizon is the time between the moment when a forecast is presented and the moment for which the forecast is made.

In practical forecasting it is common to talk about short term forecast (short horizon), medium term forecast, etc. Depending on the subject area, the horizons may differ significantly. For example, in Economics, a short term forecast may have a horizon of 1-2 quarters (half a year). In Technometrics (when data is measured every second), long term forecast may mean forecasting just few hours ahead.

In the forecasting theory we shall just distinguish between short term forecasting (just few periods ahead of the last available observation) and long term forecasting (anything above 5 periods ahead).

Forecast Accuracy

The accuracy of a forecast may depend on the following principal factors:

1. Adequacy of the model.

Estimation accuracy.

3. Forecasting method.

Random errors.

Adequacy of the Model

The model is regarded as an adequate if it correctly represents principal relationships of the true process. Of course, this definition is not suggesting rigorous methods of adequacy verification, moreover, the true relationships of the real process are usually not known.

Most justified conclusions about adequacy are made at the residuals analysis stage.

Inadequate models sometimes may provide reasonably small errors for the available dataset, however it can lead to large (and growing with time) errors (e.g. see figure 5.11). The issue of the adequacy is very complex and intriguing. We shall repeatedly return to a discussion of this issue in what follows.

Estimation Accuracy

This depends on the estimation method used and the sample size. The estimation methods may be parametric and non-parametric. Generally parametric methods are more accurate, however, they depend on the validity of the assumptions (good example -maximum likelihood method -it requires the knowledge of the statistical distributions of the variables and parameters). Non-parametric methods are more versatile, but less accurate (e.g. least squares method). It is important to use consistent estimation; in this case the accuracy of the estimation may be improved by increasing the sample size (whenever possible).

In the case of linear models, we need to estimate the parameters of the model (a and b) as accurately as possible, to improve the forecasts.

Forecasting Method

The forecasting method is based on the model and the estimated parameters. It should be selected according to the preferred loss function.

Random Error

In our models, the random errors are represented by e 1 (t) and e 2 (t). If a model is adequate, the errors may be reduced only by introducing additional appropriate explanatory variables. However, there is always a limit and this error cannot be reduced below a certain level (e.g. due to non-availability of perfect measuring tools).

In-Sample Error and Out-of-Sample Error

The quality of a model and of an associated forecasting method may be assessed by comparing actual observations to the forecasts. As was discussed before, we need to estimate the parameters of the model before we may produce forecasts. One may use the whole available sample to estimate the unknown parameters. These parameters are used then to generate the time series (often called predicted response), which is then compared with the actual observations. In this case we speak about in-sample error. Alternatively, one may use the estimated parameters to generate future values of the series and compare them with the observations which were not used in the estimation process. In this case one refers to out-of-sample error.

Linear-Linear Model (LL-model)

The mixed model is defined as the following weighted average of y 1 (t) and y 2 (t):

y(t) = py 1 (t) + qy 2 (t), (2.23)
where p+ q = 1. We also assume that all errors e 1 (t 1 ) and e 2 (t 2 ) are independent for all t 1 , t 2 .

One can easily evaluate all the characteristics of y(t):

E(y(t)) = a + bt V (y(t)) = p 2 s 2 1 + tq 2 s 2 2 cov(y(t), y(t -1)) = q 2 s 2 2 (t -1) ρ(y(t), y(t -1)) = qs 2 2 (t -1) (s 2 1 p 2 + s 2 2 q 2 t)(s 2 1 p 2 + s 2 2 q 2 (t -1)) (2.24)

Alternative Presentations of the LL-model

The LL model may be presented in several possible ways. Depending on the circumstances one or another presentation may be useful.

Pseudo Linear Regression

y(t) = p(a + bt + e 1 (t)) + q(a + bt + ǫ(t)) = a + bt + pe 1 (t) + qǫ(t) (2.25)
where ǫ(t) = (e 2 (1) + ... + e 2 (t)).

The errors in the above equations are independent, but not identically distributed (the variance of the error is growing with time). Therefore we call this presentation pseudo regression.

Pseudo Auto-regression

y(t) = p(a + bt + e 1 (t)) + q(y 2 (t -1) + b + e 2 (t)) = y(t -1) + b + p(e 1 (t) -e 1 (t -1)) + qe 2 (t) (2.26)
The above is a non-stationery AR process with dependent errors.

Independent Identically Distributed Errors

y(t) = p(a + bt) + q(y 2 (t -1) + b) + (pe 1 (t) + qe 2 (t)) (2.27)
In this equation, the errors (pe 1 (t) + qe 2 (t)) are independent and identically distributed. The variance of the error is:

V (pe 1 (t) + qe 2 (t)) = p 2 s 2 1 + q 2 s 2 2
(2.28)

If s 1 = s 2 = s we have:

V (pe 1 (t) + qe 2 (t)) = (p 2 + q 2 )s 2 ≤ s 2 (2.29)
where equality is only possible if p = 0 or p = 1.

Important also to note that the process y 2 (t) is not observed.

Differenced LL Model (DLL)

Naturally, from (2.26):

∆y(t) = y(t) -y(t -1) = b + p(e 1 (t) -e 1 (t -1)) + qe 2 (t) (2.30)
The above process is stationary. It would be a stationary MA process, if not the last term (qe 2 (t)).

Parameters of the DLL

We skip the boring details of evaluation of the parameters of the differenced series and present the results:

E(∆y(t)) = b V (∆y(t)) = 2p 2 s 2 1 + q 2 s 2 2 cov(∆y(t), ∆y(t -1)) = -p 2 s 2 1 ρ(∆y(t), ∆y(t -1)) = - p 2 s 2 1 2p 2 s 2 1 + q 2 s 2 2 ρ(∆y(t), ∆y(t -k)) = 0 (k ≥ 2) (2.31)
It is worth noting that:

0 ≥ - p 2 s 2 1 2p 2 s 2 1 + q 2 s 2 2 ≥ - p 2 s 2 1 2p 2 s 2 1 = -0.5 (2.

32)

So:

-0.5 ≤ ρ(∆y(t), ∆y(t -1)) ≤ 0 (2.33)

2.16 Estimation of the Double Linear Model

Estimating p and q

Let's assume that s 1 = s 2 . Essentially it means that all random factors impacting on the process have equal magnitude. Since the DLL is a stationary process, we can estimate both the variance and the autocorrelation coefficient (lag 1) from a single time series. Suppose the estimated variance is σ2 and the estimated autocorrelation is ρ. So, we have:

σ2 ≈ 2p 2 s 2 1 + q 2 s 2 2 ρ ≈ - p 2 s 2 1 2p 2 s 2 1 + q 2 s 2 2 (2.34)
Since s 1 = s 2 = s and q = 1 -p, we have a system of two equations with 2 unknowns (p and s 2 ). Solving the system we get the estimates:

σ2 ≈ 2p 2 s 2 + q 2 s 2 ρ ≈ - p 2 s 2 2p 2 s 2 + q 2 s 2 (2.35)
Solving the above, we get:

p = ρ ± √ -ρ -2ρ 2 1 + 3ρ s 2 = σ2 2p 2 + q 2 (2.36)
Since p should be between zero and one, one root for p should be dropped and we finally have:

p = ρ + √ -ρ -2ρ 2 1 + 3ρ s 2 = σ2 2p 2 + q 2 = σ2 (1 + ρ + 2 -ρ -2ρ 2 )) (2.37)
It is not difficult to see that if -0.5 ≤ ρ ≤ 0, then 0 ≤ p ≤ 1 (see the graph below). Some other important things to note:

• One may estimate p from ρ (no need to know σ2 ).

• The following relationship between ρ and p may be of use (e.g. for making tables):

ρ = - p 2 1 -2p + 3p 2
(2.38)

• It may happen in practice that ρ is outside of the interval [-0.5, 0]. If one can confirm (by hypothesis testing), that the true autocorrelation is outside of this interval, we have to reject the hypothesis, that we deal with the LL process.

• If one may confirm (by hypothesis testing) that ρ = 0.5, we should conclude that p = 1 and we deal with a linear regression process.

• If one may confirm (by hypothesis testing) that ρ = 0, we should conclude that p = 0 and we deal with a linear auto-regression process.

• When ρ = -1/3, the estimate for p is 0.5 and two processes (y 1 and y 2 ) have equal weight in the mixed process.

• Estimate for s 2 is always positive (and therefore always exists if -0.5 ≤ ρ ≤ 0).

Estimating the Slope b

Decomposition

If we can accurately estimate the slope parameter of the model (b), then we can decompose the data into two "ingredients": linear regression process and linear auto-regression process. This is very important for the purpose of forecasting (this will be discussed later in the relevant section).

In this section we consider the following four estimation procedures:

1. LR Estimation.

2. LAR Estimation.

3. Mixed Estimation.

. All the above estimation methods provide unbiased and consistent estimates. So, the only concern is efficiency. In many applications we have limited size datasets and we cannot increase the sample size in principle (e.g. in most of Economic applications). We tested all the suggested methods by simulation, trying to identify which approach secures the minimum variance of the estimate.

LR Estimation

This estimations uses formulae (2.19 -2.20). Our simulations show that this method is providing estimates which may compete with other methods only if p ≈ 1, e.g. 0.9 < p < 1.

LAR Estimation

If p ≤ 0.5, LAR method (as given by equations (2.21 -2.22)) should be used.

Mixed Estimation

Since the LL process is a weighted average of two processes, one may hypothesise that a good estimate of b may be received as a weighted average of the estimates the LAR and LR estimate (2.19) where

w 1 + w 2 = 1.
Clearly, this method covers the previous two (if we set w 1 = 1 we get LR estimation, if w 1 = 0, we get LR estimation).

We found that a task of finding the optimal weights (u i and w i ) is very challenging. At this stage we have a hypothesis that:

• If p ≤ 0.5 the optimal weights are w 1 = 0 and w 2 = 1, that is LAR estimate is the most efficient.

• If 0.5 < p ≤ 1,

w 1 = - 5 6 + 3 2 p + 1 3 p 2 w 2 = 1 -w 1 = 11 6 - 3 2 p - 1 3 p 2 = q 3 (13/2 -q) (2.40)
By the way, it implies that the LR estimation is the best for p approaching 1.

The graph in figure 2.7 shows the variances of the estimates received by the mixed method using various w 1 and w 2 for p = 0.5. As we can see, the minimal variance is at w 1 = 0.

The graph in the figure 2.8 shows the variances of the estimates received by the mixed method using various w 1 and w 2 for p = 0.7. One can roughly see that the minimum variance is around w 1 ∈ (0.37, 0.39)

LR model

Suppose an object moves along a straight rail. However, the connection between the object and the rail is not rigid, but allows for some flexibility. Random disturbances will make the movement of the object jagged, and the trajectory will be described by the LR model.

LAR model

Again we think of an object moving along a straight rail. But now the random disturbances do not impact the object, but they move the rail (in such a way that the slope/angle of the rail does not change). In this case, the trajectory will be described by the LAR model.

LL model

In this case, the random disturbances affect both the object and the rail. Then the LL model will be right to describe the movement of the object.

Econometric Interpretation

Very often (even in free market economies) a part of a market is controlled (e.g. by the government, central bank, other regulators, etc.). The rest of the market is developing according to "free market" principles, that is, per classical demand-supply theory.

For example, in South Africa:

• The wholesale price of electricity (from ESKOM) will grow at 8% a year (as set by the regulator -NERSA);

• Social housing pricing is also controlled by the national regulator (SHRA);

• Price of petrol is also defined by the government;

• The National Reserve Bank is monitoring and controls the inflation, etc.

We may view the regulated prices as a LR process and the free market price as a LAR process. Surprisingly, we can find a "regulated" component in the markets which are not supposed to be regulated (see the next section).

Another possible interpretation may be given in terms of the price of an asset, e.g. a bond. The dirty price of a bond consists of two components: (a) clean (market) price -defined by the demand-supply principles and (b) the accrued interest. The market price may be regarded as a LAR process. Accrued interest may be regarded as a LR process (highly predictable). Keeping in mind that both components in this case are not linear in time, one may decide to apply the model to the logarithm of the data.

When We May Try the LL Model

George Box said that all models are wrong, but some of them are useful. I strongly support this philosophy and believe that the most appropriate model should be used for a given dataset.

In this section, we suggest a few simple criteria which allow to rule out the LL model in the early stages of the data analysis. Of course the final decision on the validity of the model may be taken only on the basis of the analysis of the residuals, but the simple rules below may save lots of time. It is worthwhile to mention that while this is a double linear model, the data may not look linear at all. 1. The original series is not stationary. The principal features of the LL process are (a) presence of the drift b and (b) growing variance. The inspection of this criteria may be done both visually or with a formal test. Please note, that certain softwares (e.g. eviews) de-trend the data before applying the unit root test. In this case other tests are recommended.

2. The differenced series should have autocorrelation (at lag 1) which is between -0.5 and 0. This is a very strong filtering rule, which discards a great deal of datasets which cannot be described with the LL model.

3.

The differenced series should have zero autocorrelations at lag 2 or higher. This criteria may be easily tested using various standard statistical tests. For longer series, we recommend utilisation of the tests for practical significance.

We found the above rules very useful in practical applications of the model.

Example of an Application

We used our algorithms to fit the LL model to gold price data (530 months, price in dollars).

A simple check-up shows that the criteria (see section 2.17.3) are met. The estimate for autocorrelation of the differenced series was ρ = -0.134, therefore p = 0.2996, meaning that 30% percent of the series is AR process and 70% is LAR process.

Surprisingly, we find that a "regulated" component is as big as 30% of the price.

The Figure 2.9 gives the original data and the two linear components of it.

Figure 2.9: Gold Price (in USDollars): orange -data, green and red -decomposed series)

Models in Energy Demand: Introduction

The choice of approach in energy forecasting impacts the smooth operation of the energy markets through the ability of participating companies to foresee the developments in the situation and adjust accordingly. A great deal of energy forecasting techniques and models are employed today by researchers from different countries. One can observe a great deal of publications suggesting improvements of existing models and creation of new ones. However, there is still no uniform system of the models classification. In this review, we offer our own system of classification of the models used for energy forecasting and energy market analysis. This facilitates the understanding of these by a wide range of users (economists, sociologists, ecologists etc.).

This review covers over 100 publications on models and methods which are used for energy forecasting and energy market analysis. The proposed classification covers methods for short, medium and long-term forecasts. Both traditional and renewable energy consumption and energy substitution models are included in this section. Special attention is given to approaches that predict demand and consumption, load and price fluctuations for electrical energy. This is due to the fact that the expected growth of electricity consumption is highest in comparison to other forms of energy.

According to the forecast made by Exxon Mobil [119] specialists, in 2040 the global demand for electricity will be 80% higher than today. This demand growth distributes for consumer sectors as follows: manufacturing and industry -45%; households -30%, commercial -20%. The demand for the transport sector will also increase, but not as much. In this case, coal will be used less and less for electricity production. It will be replaced with cleaner energy such as natural gas, nuclear and renewable sources. Meanwhile, thanks to more efficient production of electricity, the demand for primary energy (for electricity production) will grow only by about 45%.

The two main causes of increase in energy consumption are population growth and economic growth. However, in the case of electricity, there is an additional factor: the transition to electricity in place of other forms of energy (oil or biomass for lighting and heating in residential sector and coal in the industrial sector). Also, in our classification we included models allowing a set degree and direction of the causal relationship between the factors impacting electricity consumption. Often, causality is the foundation for more accurate forecasting.

The organisation of the paper from here onwards, is as follows: the next section provides notations and acronyms that were used in this study. It is followed by a section that presents classification of models and methods used in the energy forecasting and energy market analysis, as well as the criteria that formed the basis of this classification. The last section provides concluding remarks.

Model Classification

Energy forecasting involves three main areas of application: energy production, energy demand and energy price. The first two are subject to long and relatively uniform trends that can be predicted in the long-term. Contrary to that, the price is highly volatile and is much more difficult to predict, especially for longer horizons projections [START_REF] Ismagilov | Methods for solving the problem of energy forecasting[END_REF].

This review focuses on the following seven parameters that are included in Tables 2.1 -2.5: Energy models, Method, Horizon, Variables, Purpose, Country (where data was collected) and Year of publication.

It should be noted that there is no clearly defined classification of models used today in energy forecasting and energy market analysis. Different authors employ various approaches for classification and different naming of the classes [START_REF] Banos | Optimization methods applied to renewable and sustainable energy: A review[END_REF], [START_REF] Bhattacharyya | Modelling energy demand of developing countries: Are the specific features adequately captured?[END_REF], [START_REF] Connolly | A review of computer tools for analyzing the integration of renewable energy into various energy systems[END_REF], [START_REF] Enders | Applied Econometric Time Series[END_REF], [START_REF] Makluev | Analysis and planning of electricity consumption[END_REF], [START_REF] Suganthi | Energy models for demand forecasting -a review[END_REF], [START_REF] Ismagilov | Methods for solving the problem of energy forecasting[END_REF], [START_REF] Yu | A PSOGA optimal model to estimate primary energy demand of China[END_REF], [START_REF] Zhou | A trigonometric grey prediction approach to forecasting electricity demand[END_REF]. The classification system that we suggest is based on previous studies [START_REF] Bhattacharyya | Modelling energy demand of developing countries: Are the specific features adequately captured?[END_REF], [START_REF] Enders | Applied Econometric Time Series[END_REF], [START_REF] Suganthi | Energy models for demand forecasting -a review[END_REF], and [START_REF] Yu | A PSOGA optimal model to estimate primary energy demand of China[END_REF]. This is done to facilitate the comprehension by the wide range of users (economists, sociologists, ecologists etc.) of the forecasting methods and models that are used by the specialists in statistics and econometrics.

If this approach is used as a basis of the classification, we can identify four groups of models: (i) Statistical, (ii) Behavioural, (iii) Reference Class and (iv) Hybrid. Their corresponding classification is given in the tables below. It is important to note that quite often the name of the model coincides with the name of the method on which they are based.

In Statistical models 2, the functional relationship between the future and the actual values of the time series, as well as external factors is specified analytically. Currently, Statistical models are dominant. Such models are based on statistical methods ranging from simple to quite sophisticated [START_REF] Bhattacharyya | Modelling energy demand of developing countries: Are the specific features adequately captured?[END_REF]. Simple models rely on a single indicator and the forecast is defined by the estimated changes in the indicator for the forecasting period. These approaches are applied for both traditional and renewable energies and can be used for all areas. Examples can be found in [START_REF] Amarawickrama | Electricity demand for Sri Lanka: A time series analysis[END_REF], [START_REF] Berrah | Sustainable energy in China: The closing window of opportunity[END_REF], [START_REF] Keyno | Forecasting electricity consumption by clustering data in order to decline the periodic variables affects and simplification the pattern[END_REF]. However, their main disadvantage is the absence of feedback and they do not consider technological specifics. In constructing sophisticated models, authors use different types of econometric modelling techniques [START_REF] Bianco | Electricity consumption forecasting in Italy using linear regression models[END_REF], [START_REF] Haas | Residential energy demand in OECD-countries and the role of irreversible efficiency improvements[END_REF], [START_REF] Liang | Multi-regional input output model for regional energy requirements and CO2 emissions in China[END_REF], [START_REF] Tan | Day-ahead electricity price forecasting using wavelet transform combined with ARIMA and GARCH models[END_REF].

The second group contains models based on Behavioural approaches, Table 2.3. This definition is applied on the basis of research by Polderman [START_REF] Polderman | Introduction to Mathematical Systems Theory: A Behavioral Approach[END_REF] and Bock [START_REF] Bock | Three kinds of behavior model[END_REF], [START_REF] Bock | A more complete model of relations and their implementation, part iv: Aggregation[END_REF] and implies that the Behavioural model reproduces the required behaviour of the original analysed system.

The essence of the Reference Class Models (RCM), Table 2.4, is in providing intuitive analysis of the problem by experts and its subsequent quantification of judgements and formal treatment of the results. Typically, an RCM is based on the Expert System and is issued as part of integrated forecasting. For example, in a World Energy Projection System (WEPS) this approach is used by the US Department of Energy (DOE) [START_REF] Glagolev | Long-term forecasting of the gas market: the experience of screenwriting software[END_REF].

In the last decade, researchers have been actively experimenting in an attempt to find new The energy forecasting area is vast. It includes comprehensive forecasts and projections for the related sectors as well as the analysis of energy markets. Our task was to explore techniques for different purposes of forecasting: different types of energy demand, including renewable energy, electricity demand, consumption, price and load, energy and electricity market analysis, etc. Not all publications clearly indicate the exact meaning of the time interval involved in the proposed models. We will distinguish two terms: "horizon" and "grain". Horizon is defined as the interval between the moment of time when the forecast is made and the time for which the forecast is made. The grain is defined as the length of the time interval for which the forecast With regards to the electric-load forecasting, the methods are offered for short-term (a few minutes to 24 hours) forecasting and very rarely for the long-term (1 10 years) or intermediateterm (a few days to several months) load forecasting [START_REF] Al-Hamadi | Long-term/mid-term electric load forecasting based on short-term correlation and annual growth[END_REF]. Forecasts with various horizons require different approaches. For long-term and medium-term, it is necessary to analyse the overall economic situation, industry trends etc. Short-term forecasts need to account for weather factors, nature of the day (working or weekend) and technical condition of the energy system in the near future. Thus, the tables include a column for "Variables". The column for "Country" gives the country (or countries) where one or another model was applied. This column provides an opportunity to assess regional prevalence of this or that technique and how intensive the forecasting research is there. Also it explains the specifics of the study and determines the choice of variables. The use of a lunar monthly date that has been applied as a variable [START_REF] Barakat | Modeling of no-stationary time-series data. part ii: Dynamic periodic trends[END_REF], is conditioned by the traditional use of the lunar calendar, along with an official in some countries. The year of publication of the articles "Year" reflects the structural changes in the techniques that were used for the energy forecasting and energy market analysis as well as increasing interest in this area.

For this review, we selected more than one hundred scientific papers that best represent the existing model range and allow for its classification. All levels of energy forecasting were covered for a significant number of the world regions. As a result, the following conclusions may be drawn:

• There is no single best approach to classification of methods and models for energy forecasting and energy market analysis and this leads to inaccuracies and ambiguities in relevant research;

• Application of the proposed classification allows obtaining of a detailed understanding of the methods used for energy forecasting and aids the selection of energy forecasting tools;

• This classification can be complemented and expanded as further research emerges.

Comparative Analysis of the Models

Energy markets are at the core of world economy and politics. That is why the predictions of fluctuations in the energy markets are of great importance for all economic agents. These changes could be in the context of pricing, energy production, energy consumption, socio-economic elements, interactions between these, and so forth. Among important questions that need to be asked in given contexts are: to which extent are changes predictable? What are the best tools to predict the changes? What methodologies are used worldwide? Which of these are most promising? What are the challenges with these methodologies? Many scientists from different countries are attempting to provide answers to these questions. A brief expose, specifically regarding energy forecasting, of their research is presented in this review.

The forecasting of the energy market elements forms the basis for integrated energy management, which in turn, is critically important for sustainable development of any country and the worlds economy as a whole. To be more specific, since the mid 70s of the last century, forecasting production and consumption of energy is a leading question in the global development strategy.

With the advent of industrialization and globalization, the demand for energy has increased significantly. This has led to aggravation of environmental issues and restructuring of primary energy balances. Thus, the integrated approach in energy forecasting and energy market analysis becomes of key importance. However, some situations are quite contradictory in the usage of existing methods and approaches for forecasting. This is due to the fact that developed models and software often do not provide for the following conditions:

• Creation, storage and processing of multi-data archives which are needed to conduct comprehensive analysis of the patterns and trends of energy consumption;

• Availability of various techniques to develop applicable models;

• Accounting for various influencing factors, including climate change;

• High accuracy required from the models utilised for forecasting.

Accuracy of forecast calculations is determined by mathematical models used to describe the fluctuations of energy consumption. In general, consumption fluctuations are complex nonstationary random processes, including cyclical/repeating fluctuations [START_REF] Hang | Short-term electricity demand and gas price forecasts using wavelet transforms and adaptive models[END_REF].

The main objective of this study is to review existing models and techniques that are used for energy forecasting and energy market analysis and their classification in a form that is accessible to a wide range of users.

The organisation of the section from here onwards, is as follows: firstly we overviews and summarise the advantages and disadvantages regarding the application of different techniques included in this review. This is followed by concluding remarks.

In this review, comparative analysis of models is made according to the classification that was proposed in the previous sections (Models in Energy Demand and Models Classification). These are the next groups: Statistical models, Behavioural models, Reference Class models and Hybrid models.

The main task of researchers is to achieve maximum efficiency and accuracy of forecasting models and the relative availability of their use. Some of the publications pay special attention to the selection, comparative analysis and testing of the accuracy of the models. Akkurt, Demirel and Zaim [5] compare four different time-series models for forecasting natural gas consumption. They revealed that in the yearly data set, double exponential smoothing model yield better results than the other alternative forecasting models. From the other side, in terms of monthly data sets, the SARIMA model provides better results than the others. Arsenault, Bernard et al focused on the forecasting accuracy [START_REF] Arsenault | A total energy demand model of Qubec: Forecasting properties[END_REF]. Others prefer to base their forecasts on a preliminary study of relationships and dependencies between the variables. This section provides a survey of the results of such studies.

The choice of forecasting methods depends on several factors such as the area and horizon of the forecast, selected variables, accessibility and availability of information databases and previous experience in such techniques. However, the most prevalent approaches are based on statistical methods, especially those using econometric time-series. This point of view is based on the fact that the models officially utilized in different countries for the energy forecasting and energy market analysis employ mainly mathematical and statistical approaches. Thus, Connolly, Lund et al [START_REF] Connolly | A review of computer tools for analyzing the integration of renewable energy into various energy systems[END_REF] performed detailed analysis of the most popular software which are used for research on integration of renewable energy into various energy systems and which are based on Statistical forecasting. Some of them like H2RES, INFORSE-Europe, LEAP, MARKAL/TIMES, MESSAGE, NEMS are also applied to energy forecasting and energy market analysis. These sophisticated models are classified as bottom-up and top-down models. These also include the Times G5 model [START_REF] Rout | Energy and emissions forecast of China over a long-time horizon[END_REF] and the Energystat model [START_REF] Makluev | Analysis and planning of electricity consumption[END_REF].

Bhattacharyya and Timilsina [START_REF] Bhattacharyya | Modelling energy demand of developing countries: Are the specific features adequately captured?[END_REF] compared a number of models that have been used in developing countries. They consider two accounting type (bottom-up) models -LEAP and MEDEE/MAED -and two global/hybrid models -POLES and WEN. One of the results of their research was the assertion that only LEAP and MEDEE/MAED have the generic capabilities for use in a wider context.

Their study reveals, however, an extremely high demand for MARKAL models [START_REF] Rout | Energy and emissions forecast of China over a long-time horizon[END_REF], [START_REF] Seebregts | Energy/environmental modelling with the MARKAL family of models[END_REF], [START_REF] Sharma | Demand for commercial energy in the state of Kerala, India: an econometric analysis with medium-range projections[END_REF], [START_REF] Strachan | Final report on DTIDEFRA scenarios and sensitivities using the UK MARKAL and MARKAL-macro energy system models[END_REF], [START_REF] Taylor | Energy technology perspectives 2008:scenarios and strategies to 2050[END_REF]. MARKAL family models are the set of techniques aimed at addressind various problems of energy planning and related environmental impacts. Thus, Bianco, Manca et al conducted a comparative analysis between regression models, based on co-integrated or stationary data, and Italian national forecasts, based on complex econometric models, such as MARKAL-TIMES. As a result, it was shown that the developed regressions deviate from the official projections from 1% to 11% [START_REF] Bianco | Analysis and forecasting of non-residential electricity consumption in Romania[END_REF].

Published in 1978, the famous work of Box and Jenkins opened a new generation of fore-casting tools. This is known as ARIMA technique. Today, this method is widely used in energy forecasting and energy market analysis. Also, it is the basis for other techniques, such as Bayesian vector autoregressive methodology (BVAR) which was used by the Francis, Moseley et al for an energy consumption outlook in selected Caribbean countries [START_REF] Francis | Energy consumption and projected growth in selected Caribbean countries[END_REF].

Erdogdu used co-integration and ARIMA modelling for electricity demand analysis and forecasting in Turkey and compared the results with official projections [START_REF] Erdogdu | Electricity demand analysis using cointegration and ARIMA modelling: A case study of Turkey[END_REF]. In another study, he applied this approach to natural gas demand forecasting [START_REF] Erdogdu | Natural gas demand in Turkey[END_REF]. Particularly, ARIMA models can be used for day-ahead electricity prices prediction as it was undertaken by Conejo, Contrras et al [START_REF] Conejo | Forecasting electricity prices for a day-ahead poolbased electric energy market[END_REF]. However, they tested different forecasting techniques to predict the 24 market-clearing prices of a day-ahead electric energy market and concluded that time series techniques show themselves as more efficacious than wavelet-transform or neural network methods. One reason for this is that both wavelet and neural network techniques break the time series of marketclearing prices in some manner. Among time series techniques, the dynamic regression and transfer function algorithms are more effective than ARIMA models. Wavelet models behave similarly to ARIMA models, while the neural network procedures normally do not show good performance. The Box-Jenkins method is also used for electricity load forecasting [START_REF] Badri | EDSSF: A decision support system (dss) for electricity peak-load forecasting[END_REF], [START_REF] Fan | Machine learning based switching model for electricity load forecasting[END_REF].

Important advantages of regression and auto-regression models are their simplicity and transparency of the simulation. Another advantage is the consistency of analysis and design. However, there are two major flaws in the regression and "causality" models: (i) absence of feedback [START_REF] Glagolev | Long-term forecasting of the gas market: the experience of screenwriting software[END_REF];

(ii) quality forecasting requires that the data on the independent variables should be very accurate and reliable [START_REF] Zhou | A trigonometric grey prediction approach to forecasting electricity demand[END_REF]. To account for these issues, some researchers exploit the relationship between dependent variables and independent variables as it is assumed that the variation in dependent variables can be explained by independent variables. So, Amarawickrama and Hunt estimate the influence of different variables on the electricity demand using five co-integration methods and achieve different results [START_REF] Amarawickrama | Electricity demand for Sri Lanka: A time series analysis[END_REF]. This underlines the importance of using of the causal econometric relationship when attempting to forecast electricity demand or construct various scenarios. Haas and Schiper [START_REF] Haas | Residential energy demand in OECD-countries and the role of irreversible efficiency improvements[END_REF] draw attention to the fact that household energy demand did not rebound in times of declining energy prices as might have been expected. They tested different models for the role of irreversible efficiency improvements in the residential energy demand and concluded that price elasticity is not the same for rising and falling prices. Hence, elasticity approaches may lead to a far too high overestimation of energy demand.

Nyulu and Litvine [START_REF]Weather neutral models for short-term electricity demand forecasting[END_REF] showed that short-term forecasting (regression and auto-regression models) of the national demand in South Africa may be very accurate if only calendar data is used (e.g. season, day of week, public holidays, election days, and school holidays, etc.) Egelioglu, Mohamed and Guven investigated the relationship between economic variables and annual electricity consumption in Northern Cyprus. The results indicate that the model using the number of customers, the number of tourists and the electricity prices as regressors has very strong predictive ability and can be used to forecast future annual electricity consumption in this area [START_REF] Egelioglu | Economic variables and electricity consumption in Northern Cyprus[END_REF].

Other authors used different methods to determine the relationship between the variables to build further forecasts: Bessec and Fouquau investigated the nonlinear link between electricity consumption and temperature in Europe [START_REF] Bessec | The non-linear link between electricity consumption and temperature in Europe: A threshold panel approach energy[END_REF]. Al-Ghandoor et al found that industrial production outputs and capacity utilization are the two most important variables that affect demand for electrical power and that the multivariate linear regression model can be used adequately to simulate industrial electricity consumption with very high coefficient of determination [6]. Bianco, Manca and Nardini [START_REF] Bianco | Electricity consumption forecasting in Italy using linear regression models[END_REF] analysed the price elasticity and concluded that there was no need to consider electricity price as explaining variable in forecasting models for Italian electricity consumption.

Mohamed and Bodger [START_REF] Bodger | Forecasting electricity consumption in New Zealand using economic and demographic variables[END_REF] used economic and demographic variables in multiple linear regression models. They than compared these models with the national forecasts available in New Zealand. The comparison revealed that the forecasts made by the regression models are very comparable with the national forecasts. The accuracy of the forecasts of these models strongly depended on the accuracy of forecasts made for the explaining variables.

Sari and Soytas examined how much of the variance in national income growth could be explained by the growth of different sources of energy consumption and employment in Turkey [START_REF] Rout | Energy and emissions forecast of China over a long-time horizon[END_REF]. Similar studies were performed by a number of other researchers [START_REF] Achao | Decomposition analysis of the variations in residential electricity consumption in Brazil for the 19802007 period: Measuring the activity, intensity and structure[END_REF], [START_REF] Al-Iriani | Energy-GDP relationship revisited: An example from GCC countries using panel causality[END_REF], [START_REF] Bianco | Electricity consumption forecasting in Italy using linear regression models[END_REF], [START_REF] Bildirici | Economic growth and electricity consumption in former Soviet Republics[END_REF], [START_REF] Erdogdu | Electricity demand analysis using cointegration and ARIMA modelling: A case study of Turkey[END_REF], [START_REF] Inglesi | Aggregate electricity demand in South Africa: Conditional forecasts to 2030[END_REF], [START_REF] Iniyan | Energy models for commercial energy prediction and substitution of renewable energy sources[END_REF], [START_REF] Lee | Dynamic modeling of energy consumption, capital stock, and real income in g-7 countries[END_REF], [START_REF] Liang | Multi-regional input output model for regional energy requirements and CO2 emissions in China[END_REF], [START_REF] Mu | Input-output table of electricity demand and its application[END_REF], [START_REF] Odhiambo | Energy consumption and economic growth nexus in Tanzania: An ARDL bounds testing approach[END_REF], [START_REF] Pao | Forecast of electricity consumption and economic growth in Taiwan by state space modeling[END_REF], [START_REF] Shengfeng | The relationship between electricity consumption and economic growth in China[END_REF], [START_REF] Suganthi | Renewable energy in India a modelling study for 2020-2021[END_REF], [START_REF] Wei | A scenario analysis of energy requirements and energy intensity for Chinas rapidly developing society in the year 2020[END_REF], [START_REF] Wilting | Trends in Dutch energy intensities for the period 1969-1988[END_REF], [START_REF] Wolde-Rufael | Energy consumption and economic growth: The experience of African countries revisited[END_REF], [START_REF] Ziramba | Disaggregate energy consumption and industrial production in South Africa[END_REF], [START_REF] Ziramba | Price and income elasticities of crude oil import demand in South Africa: A cointegration analysis[END_REF]. The main objective of these studies was to establish the relationship and the interdependence between the economic, social, demographic, environmental, social, financial variables and the energy supply and demand, with the purpose to optimize the forecasting models and budgeting energy investments. Decomposition, Unit-root and Co-integration models formed the basis of these investigations.

Researchers are often faced with a shortage of data or insufficient sample size for predicting. It is a serious obstacle in case of the time-series methods. The Grey forecasting models have been developed for such cases. They are based on Grey theory 1 and models can be constructed even for four data points or ambiguous data. The Grey systems theory, established by Julong Dengin 1982, is a new methodology that focuses on the study of problems involving small samples and poor information. It deals with uncertain systems with partially known information through generating, excavating, and extracting useful information from what is available. So, systems' operational behaviours and their laws of evolution can be correctly described and effectively monitored. In the natural world, uncertain systems with small samples and poor information exist commonly. That fact determines the wide range of applicability of Grey systems theory [START_REF] Shengfeng | The relationship between electricity consumption and economic growth in China[END_REF].

Kumar and Jain [START_REF] Kumar | Time series models (grey-markov model with rolling mechanism and singular spectrum analysis) to forecast energy consumption in india[END_REF] use several models based on Grey techniques to predict crude-petroleum, coal, electricity (in utilities) and natural gas consumption in India. Lee and Tong [START_REF] Lee | Forecasting energy consumption using a grey model improved by incorporating genetic programming[END_REF] applied improved Grey forecasting model that integrates Genetic programming and Grey theory in case of Chinese energy consumption projections. In both cases, the use of Grey models showed equally high accuracy in comparison with official forecasts. At the same time, Akay and Atak [START_REF] Akay | Grey prediction with rolling mechanism for electricity demand forecasting of Turkey[END_REF] used Grey prediction with the rolling mechanism approach to forecast Turkeys total and industrial electricity consumption. Results show that the proposed approach results in higher accuracy effects than the results of official forecasts which were established on the basis of the

Model of Analysis of Energy Demand (MAED).

Given the shortcomings of Statistical models, alternative forecasting methods have been developed. These are based on the Behavioral approach and include Artificial Intelligence techniques (AIT)2. Behavioral models are based on behavioral equations of the original system that take into account many variables, including auxiliary and latent, and indicate feedback [START_REF] Bock | A more complete model of relations and their implementation, part iv: Aggregation[END_REF]. Understanding of the AIT is based on the definition introduced by John McCarthy [START_REF] Carthy | What is artificial intelligence?[END_REF]: "Artificial Intelligence is the computational part of the ability to achieve goals in the world. Varying kinds and degrees of intelligence occur in people, many animals and some machines." At present, Artificial Neural Networks (ANN) models have been widely diffused into such applications. First of all, this is associated with their good precision and relatively simple access. Hamzacebi [START_REF] Hamzaebi | Forecasting of Turkeys net electricity energy consumption on sectoral bases[END_REF] and Kavaklioglu, Ceylan et al [START_REF] Kavaklioglu | Modeling and prediction of Turkeys electricity consumption using artificial neural networks[END_REF] used ANN for forecasting Turkeys electricity consumption. Application of the ANN technique yielded better results than the official projections. The official forecasting for Turkey electricity consumption was made by using the MAED simulation technique.

Aydinalp-Koksal and Ugursal [START_REF] Aydinalp-Koksal | Comparison of neural network, conditional demand analysis, and engineering approaches for modeling end-use energy consumption in the residential sector[END_REF] compared ANN, conditional demand analysis (CDA), and engineering approaches for modelling end-use energy consumption in the residential sector of Canada. Results indicated that a CDA model is unable to estimate the effects of some socio-economic factors, because the number of variables is limited in this model. None of the socio-economic factors could be evaluated by the engineering model.

Nevertheless, the ANN model is able to evaluate the effects of different variables on end-use energy consumption. Thus, from the perspective of assessing the impact of socio-economic factors, the ANN model is superior to both the CDA and the engineering models.

Yokoyama, Wakuiand and Satake [START_REF] Yokoyama | Prediction of energy demands using neural network with model identification by global optimization[END_REF] used neural network with model identification by global optimization for prediction of the energy demand in Japan. The main contribution of their research in the application of ANN models was in improving the efficiency of the neural network technique through the use of the global optimisation model. [START_REF] Al-Hamadi | Long-term/mid-term electric load forecasting based on short-term correlation and annual growth[END_REF] proposed the PSO method for forecasting. This forecast was compared with results obtained using the LES method. "From a total error point of view, it is found that the PSO method produced better estimates than the LES method. It gives the background that the PSO approach is quite promising and deserves serious attention as a new tool for parameter estimation". Toksari [START_REF] Toksari | Estimating the net electricity energy generation and demand using the ant colony optimization approach: case of Turkey[END_REF] applied the ant colony optimisation electricity energy estimation (ACOEEE) forecasting model for Turkeys energy market, which is based on the ACO approach. ACO is a multi-agent system in which the behaviour of each ant is inspired by the foraging behaviour of real ants to solve the optimisation problem.

Al-Hamadi and Soliman

In recent years there has been a tendency to use other behavioural integrated models based on Artificial Intelligence (here, we accept that the basis of the integrated model is a combination of methods within the same approach). Hybrid models are based on the use of various techniques/approaches. Thus, Yu, Wei and Wang [START_REF] Yu | A PSOGA optimal model to estimate primary energy demand of China[END_REF] analysed Chinese energy demand based on GDP, population, proportion of industry in GDP, urbanisation rate and share of coal energy and used the MPSORBF estimation model. As a result, they concluded that the proposed model has fewer hidden nodes and smaller estimated errors when compared with other ANN-based estimation models. Kiran, Ozceylan et al [START_REF] Kiran | A novel hybrid approach based on particle swarm optimization and ant colony algorithm to forecast energy demand of Turkey[END_REF] proposed a new integrated method which combined PSO and ACO for estimating the energy demand of Turkey under different scenarios.

The main advantages of AIT are ability to establish non-linear dependence between the future and the actual process values as well as their adaptability, scalability (the ANN parallel structure accelerates calculations) and the uniformity of their analysis and design [START_REF] Haykin | Neural Networks: A Comprehensive Foundation Second Edition[END_REF]. On the other hand, the shortcomings of AIE include: the lack of transparency of modelling, the difficulty of choosing architecture, high demand for consistency of training sample, complexity of choice of a learning algorithm and resources for their learning [START_REF] Mazengia | Forecasting spot electricity market prices using time series models[END_REF].

The genetic algorithm (GA) was developed and is often applied for solving optimisation, as well as search problems. However, some modifications of GA can solve the problem of energy forecasting. For example Ceylan and Ozturk [START_REF] Ceylan | Estimating energy demand of Turkey based on economic indicators using genetic algorithm approach[END_REF] presented the genetic algorithm energy demand model (GAEDM) for estimating energy demand of Turkey. As a result the GAEDM model showed a lower estimation error than official projection. Hence, this model can be applied for forecasting the energy demand in the future by optimising the parameter values using the past data. Similar approaches have been used in the following studies [START_REF] Canyurt | Estimating the Turkish residential commercial energy output based on genetic algorithm (ga) approaches[END_REF], [START_REF] Forouzanfar | Modeling and estimation of the natural gas consumption for residential and commercial sectors in iran[END_REF], [START_REF] Ozturk | Electricity estimation using genetic algorithm approach: a case study of Turkey[END_REF].

Fuzzy techniques are used in energy forecasting mostly for short-term electricity load forecasting [START_REF] Bakirtzis | Short term load forecasting using fuzzy neural networks[END_REF], [START_REF] Kucukali | Turkeys short-term gross annual electricity demand forecast by fuzzy logic approach[END_REF], [106], [START_REF] Miranda | Fuzzy inference in spatial load forecasting[END_REF], [START_REF] Rain | Short term load forecasting using fuzzy adaptive inference and similarity[END_REF] and [START_REF] Song | Short-term load forecasting for the holidays using fuzzy linear regression method[END_REF] and also as part of the methodology of Hybrid models.

Wavelet transform models were used for forecasting day-ahead electricity and natural gas prices [START_REF] Conejo | Forecasting electricity prices for a day-ahead poolbased electric energy market[END_REF], [START_REF] Hang | Short-term electricity demand and gas price forecasts using wavelet transforms and adaptive models[END_REF]. This prediction technique is based on wavelets and makes it possible to take into account the time dependency of prices and the correlations with other relevant series.

Due to application of Reference Class Models (RCM) one can solve a number of the following forecasting tasks:

• To identify dynamic relationships between energy and economy;

• To assess the impact of global political, social and mental events which are difficult to describe by mathematical calculations;

• Step-by-step simulation and adjustment of the interaction between the behaviour of the model and the real world;

• To choose the best option among the available solutions.

Solving of these problems, while using respective analytical models, affects the quality of the forecasting. Examples of RCM-based expert systems were researched by Tao [START_REF] Tao | Scenarios of Chinas oil consumption per capita (OCPC) using a hybrid factor decomposition system dynamics (SD) simulation[END_REF] and Kandil [START_REF] Kandil | The implementation of long-term forecasting strategies using a knowledge-based expert system: Part ii[END_REF]. In the former, the System Dynamics (SD) modelling was used to forecast Chinas oil consumption. In the latter, the Knowledge-based Expert System was applied for electricity market analysis.

One of the most popular current trends in the field of energy forecasting is to create hybrid models and methods. This approach accounts for the shortcomings of individual models and is aimed at improving the accuracy of forecasting as one of the key performance measures of the model.

In our review, the available Hybrid models are based on a combination of the following techniques:

• ANN + regression [START_REF] Azadeh | Integration of genetic algorithm, computer simulation and design of experiments for forecasting electrical energy consumption[END_REF], [START_REF] Pao | Comparing linear and nonlinear forecasts for Taiwans electricity consumption[END_REF];

• ANN + genetic algorithm [START_REF] Yu | A PSOGA optimal model to estimate primary energy demand of China[END_REF];

• ANN + auto-regression [START_REF] Pao | Forecast of electricity consumption and economic growth in Taiwan by state space modeling[END_REF], [START_REF] Tan | Day-ahead electricity price forecasting using wavelet transform combined with ARIMA and GARCH models[END_REF];

• ANN + Fuzzy logic + regression [START_REF] Azadeh | A neuro-fuzzy-stochastic frontier analysis approach for long-term natural gas consumption forecasting and behavior analysis: The cases of Bahrain, Saudi Arabia, Syria, and UAE[END_REF]. Also, according to Alfares and Zazeeruddin [START_REF] Alfares | Electric load forecasting: literature survey and classification of methods[END_REF], a combination of Fuzzy logic and regression can be used for electricity load forecasting. The main disadvantage of the Hybrid models is the complexity and resource intensity of their development [START_REF] Chuchueva | Time series forecasting model based on a sample maximum likelihood[END_REF]. They also require high qualification skills of forecasters.

Conclusions

In this chapter, we discussed the models which may be used for econometric forecasting. Starting with linear models (linear regression (LR) and linear auto-regression (LAR)) we suggested a new double linear model (LL-model) and derived and discussed its non-trivial properties. We showed that even the simplest LR and LAR models have cycles. We shall continue studying linear models in chapter 4, where we shall evaluate the Hurst exponent for these models.

Secondly, we discussed a number of other models which are used for forecasting electricity demand. A thorough and comprehensive analysis of existing literature was done. For this review, we selected more than one hundred scientific papers that best represent the existing model range and allow for its classification. All levels of the energy forecasting were covered for a significant number of the world regions. As a result, the following conclusions may be drawn:

• There is no single best approach to the classification of methods and models for energy forecasting and energy market analysis and this leads to inaccuracies and ambiguities in relevant research;

• Forecast accuracy depends not only on the model chosen, but also on some other objective factors (data availability, technical equipment, etc.) and subjective factors (experience and level of knowledge of the investigator, etc.);

• The most common today are the statistical models; this is due to their relative ease of access, transparency of modelling and extensive experience in the use of energy forecasting;

• Neural network techniques are believed to be best in forecasting at local level and not tested enough to assess national and global trends;

• Hybrid models are of high performance for modern long-term energy forecasting. However, when creating hybrid models one needs to combine individual models to compensate for the shortcomings of each of them and do not lose their benefits.

Precision should also supplement accuracy. The element of precision is often overlooked in the pursuance of an accurate forecast. Accuracies, at times, neglect the issue of variance, as long as the results from the model aggregate around the expectation. Precisions account for the variance but, at times, disregard the bias away from the expectation. A good forecaster is to achieve both: accuracy and precision. The impact of this feat is not trivial in the energy business, it affects planning in various aspects: e.g. building of a new power plant and deciding on the type of the new power plant. For instance, if the forecasts are indicative of more "peak" demand, then rather than electing a base-load power plant (such as a coal fired station), a pump-storage or a hydro-reservoir might be a better option. On the other hand, if the longer term forecasts indicate that relatively shortly after the peaks increase the demand fluctuates back on an increased steady pattern, then a coal fired station might be a better option, or maybe both, depending on the size of the increase or the cyclic behaviour of the peaks (i.e. how often do they re-occur after the increased steady state). It is therefore imperative that models be tested thoroughly for robustness before taking decisions on forecasts; for instance, a classic example is the paper produced by Inglesi [START_REF] Inglesi | Aggregate electricity demand in South Africa: Conditional forecasts to 2030[END_REF] where energy forecasts produced for 2012 (under 2 scenarios) would be close to the same consumption as in 1993/94; therefore concluding a no-need for capacity expansion and questioning the need of funding a new power plant. If any decision maker in the Electrical Energy utility business, in South Africa, would have believed that, it would have had disastrous consequences as it was completely contrary to predictions of the Utility forecasters (whose actual had an accuracy and precision of less than 2% against an inaccuracy and imprecision of more than 50%).

Chapter 3

Optimal Dating of Financial Cycles

Introduction

Fundamentally, in global economy there exist two big kinds of fluctuations: business cycles and financial cycles. The first is concerned with the variations in the Gross Domestic Product (GDP) or in a set of macro-economic variables. The second deals with the movement of the stocks, commodities and property prices, while a number of other definitions is possible.

Dating cycles is an important question, mainly for analytical purposes, but also for the forecast of turning points of economic or financial activity. Furthermore, the dating is crucial to the policy viewpoint.

Four main methods allow us to date cycles. The first approach consists basically in the extraction of the cycle out of relevant series. Persons [START_REF] Persons | Indices of business conditions[END_REF] seems to be the first economist to have proposed to split time series into four components: the trend, the cycle, seasonality and a purely accidental hazard. Today, numerous filters have been developed to isolate the cyclical component: Hodrick-Prescott, Baxter-King, various band-pass filters, etc. However, there are as many different cycles as there are filters and even more, if one varies the specific parameters of each of these filters. This is the big criticism which can be levied against this type of approach.

The second way to determine the cycle is based on the use of a parametric model: the so-called Markov-switching model, popularised in Economics by Hamilton [START_REF] Hamilton | A new approach to the economic analysis of nonstationary time series and the business cycle[END_REF]. The early construct distinguished two basic regimes corresponding to expansion and contraction phases. The simplest probabilistic specification is to assume that the trajectory of the regime variable is governed by a first order Markov chain with two states and transition probabilities from one state to another state. Of course, the obtained chronology is directly related to the adopted modelling: it is as good as the model and varies depending on your model. It is the main weakness of the method. Other criticism is the non-reproducibility of the method.

The third approach is model-free. It consists in isolating the turning points in the relevant series. The pioneering work relevant to this approach was that of Bry-Boschan [START_REF] Bry | Cyclical analysis of time series: Selected procedures and computer programs[END_REF]. These authors implemented a computational algorithm which yields the peaks and troughs of the analysed series. This procedure has been adapted to quarterly data by Harding and Pagan [START_REF] Harding | Dissecting the cycle: A methodological investigation[END_REF], giving the BBQ algorithm (BBQ for Bry-Boschan Quarterly).

Finally, the fourth method may be generally called blurring or smoothing. Various methodologies are available in this category, such as: simple moving average and exponential smoothing, Fourier transform and wavelets transforms, polynomial approximation, Gaussian blurring, etc.

Each approach has strengths and weaknesses. A whole big book may be written on this subject. In the following section, we shall give some details of only one approach, which is the BB-algorithm. There are several reasons for that. Firstly, it is most common in dating business and financial cycles [START_REF] Harding | Dissecting the cycle: A methodological investigation[END_REF]; secondly, the principal advantages of such an approach is ease of implementation and finally, its great reproducibility. For these reasons, this technique was selected for comparison with the new methodology which we are suggesting here.

BB-algorithm for Dating Econometric Time-Series Cycles

Suppose y(t) is a time-series (t -positive integer), and k is a positive integer. Point y(t) is regarded as a peak if:

y(t -i) < y(t) > y(t -i), i = 1, ..., k (3.1)
That is, the point y(t) is the largest point in the window of size 2k + 1, k points to the left of t and k points to the right of t. Similarly y(t) is a trough if:

y(t -i) > y(t) < y(t -i), i = 1, ..., k (3.2)
That is, the point y(t) is the smallest point in the window of size 2k + 1, k points to the left of t and k points to the right of t.

However, this definition has a major problem in terms of defining cycles. It may happen that a peak is followed by another peak. Equally, it may happen that two troughs do not have a peak in-between. Therefore, certain censoring rules are applied after all peaks and troughs are found. For example, if a peak follows another peak, only the highest peak is honoured, and the other one is discarded. Similarly, if a trough is followed by another trough, only the lowest trough is considered and the other one is ignored.

Other censoring rules may also apply, for instance only cycles exceeding certain threshold lengths are considered. However, we will not go into further details on this matter. This methodology has many positive features, which we have mentioned before. However, it also has certain limitations. For example, the method looks only at the time dimension, totally ignoring the y-dimension. This may be critical for investment analysis: if y(t) is a price of an asset (stock, real estate, etc.), investors mainly look at the change in the value of the price and are interested in shorter windows.

The methodology presented below does not have this limitation. In contrast to all other approaches, the suggested technique allows flexible choice of the objective function. So an analyst or a researcher may decide on the criteria for the optimal dating.

Optimal Dating

Suppose we have a series (time-series) of real numbers y(i), i = 1, ..., n. Suppose we need to find cycles in this data in such a way that the given objective function (OF) is maximised. The method consists of the following steps.

1. Find such i 1 and i 2 , that:

• 1 < i 1 < i 2 < n;
• OF (y(1), y(i 1 ), y(i 2 ), y(n)) is maximised over all possible i 1 and i 2 ;

2. The points i 1 and i 2 divide the range (1, n) into 3 subintervals (we shall call them areas):

(1, i 1 ), (i 1 , i 2 ) and (i 2 , n);

3. Of the available areas select one which has the highest potential to have cycles (the examples of such selection techniques will be discussed later). We shall call this area a pivotal area.

4. The first point of this area will now be considered as point one, and the last point is the point number k;

5. Find such i 1 and i 2 , that:

• 1 < i 1 < i 2 < k;
• OF (y(1), y(i 1 ), y(i 2 ), y(k)) is maximised over all possible i 1 and i 2 .

6. Now the pivotal area is divided into 3 subareas (by points i 1 and i 2 found at the present step) and the total number of areas in the series increase by 2.

7. Check the stopping criteria. If the stopping criteria is true, stop the algorithm. Repeat from step 3, while the stopping criteria is false.

Note, that with every iteration of the above algorithm, the number of areas (and consequently the number of dividing points) will increase by two.

In the next sections we shall present the following:

Optimal Trading Objective Function

Let's assume that the series represents certain asset's price traded at a stock exchange. Suppose we have an ideal trader who always purchases and sells the asset at the best possible times. Let i 1 , ..., i m be the times at which the trades occur (that is i 1 , i 3 , i 5 , ..., i m-1 are times of buys and i 2 , i 4 , i 6 , ..., i m be the times of sales and m is even).

The profit that is achieved under the given i 1 , ..., i m , may be represented as:

OT = (y(i 2 ) + y(i 4 ) + y(i 6 ) + ... + y(i m )) -(y(i 1 ) + y(i 3 ) + y(i 5 ) + ... + y(i m -1)) m/2 j=1 (y(i 2j ) -y(i 2j-1 )) (3.3)
That is, this OF equals the total value of all sales less the total value of all buys. The ideal trader will maximize this function (OT) over all possible i 1 , ..., i m and m:

max i 1 ,...,im,m OT (3.4)
The above represents the maximum profit that potentially may be derived from trading the given asset over the defined period of time (the transaction costs are ignored at this stage).

Full Evaluation

To find the exact (and guaranteed) optimum of a function which is defined on a discrete range (if nothing is assumed about the objective function), we need to evaluate the function in every point of the range. Since we do not make any assumptions (at least at this stage) about the series y(i), the exact optimum may be found only in this fashion. We shall call this method Full Evaluation or FE in what follows. Note that apart from checking all possible i 1 , ..., i m we also need to do this for all possible even m (m ≤ n).

For the given length of the series (n) we will need to make N(n, m) substitutions for each m, where:

N(n, m) = n m = n! m!(n -m)! (3.5)
This may result in extremely lengthy calculations (and the time required will grow very fast with the growth of n).

The hierarchical method, however, requires to find only two points (at a time), so the number of substitutions are:

N(n, 2) = n 2 = n! 2!(n -2)! = n(n -1) 2 . (3.6)
While the computing time will grow with n 2 , this is much less than in the FE case (algorithm's complexity 2 n ). In what follows we shall refer to Full Evaluation for fixed m as FEm and, subsequently for m = 2 as FE2 method.

Local Extremes

A local peak is defined as a point in the series which is higher than the two neighbouring points. Similarly, a local trough is defined as a point which is lower than the two immediate neighbours. It may be easily proved (e.g. by contradiction) that the optimal solution (in terms of the OT objective function) may include only local peaks. That is, while applying FE or FEm methodologies, we can try only local extremes. This may considerably reduce the computational time (depending of course on jaggedness of the series, for the reduction will be higher for less jagged series). For example, in the worst case scenario, when every point in the series (say, of length m = 2k) is either peak or trough, the number of pairs that needs to be tried in FE2 will be:

k(k + 1) 2 (3.7) compared to: 2k(2k -1) 2 = k(2k -1) (3.8)
So, the time for calculations is nearly 4 times less (for large k). However, the complexity of the algorithm remains the same:

O(m 2 ).
In what follows we shall always assume that the full evaluations (FE and FEm) are limited to trying local troughs for buy points (y(i 2j )) and trying local peaks for sell points (y(i 2j-1 )).

Area Selection

In each iteration, we need to decide in which sub-area we shall seek new pair of peak and trough. Different approaches were tested, and finally the following two showed the best performance.

TV less TC Method

Total Variation in the series y(1), ..., y(n) is defined as:

T V = n-1 i=1 |y(i + 1) -y(i)| (3.9)
That is we add up the absolute values of all the increments.

The Total Change is defined as:

T C = |y(1) -y(n)| (3.10)
That is the absolute value of the difference between the first and the last value in the series.

In the TV less TC method we calculate:

T V -T C (3.11)
for each current sub-area. The area with the highest T V -T C is selected to be pivotal in the next iteration.

TV over TC Method

In this method, we calculate the ratio between the T V and T C. The current sub-area with highest T V/T C is selected as pivotal in the next iteration.

PLC Stopping Criterion

Firstly, the stopping criterion should take into account the calculated value of the selection criteria. If T V -T C = 0 (or T V/T C = 1) for all subareas, the algorithm should stop, because no more peaks or troughs exist. Secondly, even if T V -T C is greater than zero, but is small, we should not continue because:

1. The possible cycle is likely to be a result of a random noise.

2. In the context of our interpretation (maximising the profit of a trader) the cycle may be too small to have a profit after paying charges for the trade. Therefore we name this criterion Profit Less Charges (PLC). However, some other considerations may also be taken into account, for example that the profit should account for the inflation, discounting rate, etc.

Mathematically this criterion may be defined as follows. Stop the iterations if:

|y(i 2 ) -y(i 1 )| -r 1 y(i 1 ) -r 2 y(i 2 ) > 0 (3.12)
where r 1 is the commission (as percentage) for selling and r 2 is the commission (as percentage) for buying an asset.

Application

The hierarchical dating method was applied to 180 time-series representing daily share prices of major companies listed on the Johannesburg Stock Exchange. Generally the series represented ten year records (except for the companies which were listed more recently). The method showed excellent performance, both in terms of the computational time, and the attained value of the objective function. The values of the charges (r 1 and r 2 ) were chosen equal to 1% (purely for demonstration purposes). The "TV less TC" method was superior to the "TV over TC" method in all instances (in terms of the final value of the objective function).

The calculations were performed using own codes (we used Wolfram Mathematica). The computational time for all 180 series on a relatively slow laptop (Intel R Core TM i5-3317U CPU @1.70GHz) took 2 hours and 34 minutes. The full evaluation method would not have finished this task in billions of years even if performed on the fastest super computer.

Conclusions

The proposed methodology showed excellent performance on real data both in terms of required computing time and optimality of dating. In future, one may consider introduction of other optimality criteria, possibly including the time value for money considerations.

The technique may be used for a wide range of practical and theoretical purposes like synchronisation analysis [START_REF] Harding | Dissecting the cycle: A methodological investigation[END_REF], benchmarking traders' performance, investment analysis, etc.

When applying the methodology to real data, namely daily closing prices of shares listed on JSE, we found incredible opportunities for speculation. It will be very intriguing to compare these results with speculative opportunities at other stock exchanges worldwide.

Chapter 4

Persistence in Time Series

Hurst Exponent: Introduction

The characteristic of a time series which is known today as the Hurst exponent was introduced by Harold Edwin Hurst in 1951. Since then, it was studied by numerous scientists (e.g. see Mandelbrot). Nowadays this parameter is used in various applications: from hydrology to financial markets.

So, what is measured by the Hurst exponent? In various publications one may find the following:

• Measure of long memory in time series;

• Measure of persistence;

• Measure of "jaggedness";

• Measure of risk and volatility;

• Measure of predictability;

• Fractal dimension;

• Parameter in the Fractal Brownian Motion stochastic process.

While I was studying various publications on the subject, I found many inconsistencies and even contradictions between authors. Below is my personal summary of the matter based on my own experience, studies and simulations.

Definition

Suppose we have a time-series x(t), t = 0, 1, .... The increments in this time series dx(t) = x(t) -x(t -1) are used to generate the sequence of cumulative deviations from the mean:

cd N (t) = t i=1 (dx(t) -m N ) (4.1)
where t = 1, 2, ...N and m N is the average of the dx(1), ..., dx(N):

m N = 1 N N t=1 dx(t) (4.2)
Then, for each N we calculate the range of cd N (t):

R N = max t=1,...N cd N (t) -min t=1,...N cd N (t) (4.3)
Also, for each N we calculate the standard deviation of the dx(1), ..., dx(N):

sd N = 1 N N i=1 (dx(t) -m N ) 2 (4.4) or sd N = 1 N N i=1 (dx(t) -m N ) 2 0.5 (4.5)
The rescaled range R/S is defined then as the ratio of the range R N and the standard deviation sd N :

R/S(N) = R N sd N (4.6)
The series R/S(N) is then fitted to the power function:

R/S(N) = aN H (4.7)
In fact, very often H depends on N in a sense that it is different for shorter and longer series. Therefore, it make sense to define the Hurst exponent H as a best fit parameter in (4.7) for N → ∞. In practical calculations this may be achieved by dropping some initial portion of R/S before fitting the power curve. On the other hand, some authors argue that for a wide range of processes, the Hurst exponent converges to 0.5 when only large values of N are considered.

Estimation Issues

The above method works well only if we can generate or observe a substantial number of realisations of the process under study, in order to generate a sequence of independent R/S(N). However, in practice we often have a single series (e.g. in most econometric applications). Therefore, there are a number of publications suggesting alternative, more accurate computational techniques. I found that the most accurate methodology is suggested in http://keldysh.ru/papers/2013/prep2013 11.pdf (NB: this paper is available in Russian only). Other popular techniques may be found in http://mosaic.mpi-cbg.de/docs/Racine2011.pdf.

On the other hand the above method may be modified as follows. First, we substitute dx(t) = x(t) -x(t -1) into 4.2:

m N = 1 N N t=1 dx(t) = 1 N (x(1) -x(0) + x(2) -x(1) + x(3) -x(2) + ... + x(N) -x(N -1)) = 1 N (x(N) -x(0)) (4.8)
That is, the n N depends only on the initial and the last value of the process, and does not depend on all intermediate values. Then we substitute the above and dx(t) = x(t) -x(t -1) into (4.1):

cd N (t) = (x(1) -x(0) + x(2) -x(1) + x(3) -x(2) + ... + x(t) -x(t -1)) - t N (x(N) -x(0)) = x(t) -x(0) - t N (x(N) -x(0)) (4.9)
So, the equation (4.3) becomes:

R N = max t=1,...N (x(t) -x(0) + tm N ) -min t=1,...N (x(t) -x(0) + tm N ) = max t=1,...N (x(t) + tm N ) -min t=1,...N (x(t) + tm N ) (4.10)

Hurst Exponent of LR, LAR and LL Processes 4.3.1 Distribution of a Range

The following result from the Non-parametric Statistics is very useful. If X has CDF F (x), the CDF of the sample range R = max(X i ) -min(X i ) is given by:

F R (x) = P (R < x) = n +∞ -∞ (F (u + x) -F (u)) n-1 dF (u) (4.11)
Moreover, if the population is continuous, the PDF of the range is:

f R (x) = n(n -1) +∞ -∞ (F (u + x) -F (u)) n-2 f(u)f(u + x)du (x ≥ 0) (4.12)
However, the exact evaluation of the above integrals is not possible for certain common distributions (e.g. Normal distribution).

If (for a certain distribution) the evaluation of the expression 4.12 is too complex, the following approximation may be used. It is known that the CDF of the smallest (F 1 (x)) and the largest (F n (x)) order statistics are given by:

F 1 (x) = 1 -(1 -F (x)) n (4.13) F n (x) = F n (x) (4.14)
Strictly speaking, x (1) = min(X i ) and x (n) = max(X i ) are dependent variables, however for large n this dependency is so weak that it may be ignored. Since the time-series studied in applied research are reasonably long (e.g. at least 50 observations), we may assume that the smallest and the highest order statistics are independent.

The PDF of the highest order statistics:

f n (x) = nF n-1 (x)f(x) (4.15)
If the population distribution is symmetric (that is, f(x) = f(-x) and F (x) = 1 -F (-x)), we have:

F 1 (x) = 1 -F n (-x) (4.16)
and

f 1 (x) = nF n-1 (-x)f(-x) = f n (-x) (4.17)
The distribution of the range R may be calculated using the convolution formula:

f R (r) = +∞ -∞ f n (x)f 1 (x -r)dx (4.18)
where f 1 and f n are the PDF's's of the smallest and the highest order statistic. In case of symmetric population distribution:

f R (r) = +∞ -∞ f n (x)f n (r -x)dx (4.19)

Hurst Exponent for LR Process

Suppose we have a time-series y 1 (t) = a + bt + e 1 (t), t = 0, 1, ... (LR model). The increments in this time series are dy 1 (t) = y 1 (t)-y 1 (t-1) = b+e 1 (t)-e 1 (t-1). Naturally, (since E(e 1 (t)) = 0) the average m N of the dx(1), ..., dx(N) is:

m N = E(b + e 1 (t) -e 1 (t -1)) = b (4.20)
Strictly speaking (if m N is estimated from a sample):

m N = b + e 1 (N) -e 1 (0) N (4.21)
But the second term in the above equation quickly diminishes with N → ∞.

Therefore, the cumulative deviation is:

cd N (t) = t i=1 (e 1 (i) -e 1 (i -1)) = e 1 (t) -e 1 (0) (4.22)
The first observation one can make is that all cd N (t) are identically distributed (for all t). Then, the range of cd N (t): are the lowest and the highest order statistics of the sample e 1 (1), ..., e 1 (N). In other words, the range R N of the cd N (t) is the same as the range of the e 1 (1), ..., e 1 (N). And we can use the equations (4.11) and (4.12) to understand the behaviour of R N .

R N = max
Let's find the distribution of the range of the sample from a Normal distribution with zero mean and variance s 2 . Then

F N (x) = F N (x) = Φ N (x/s) (4.24)
where Φ(x) is the CDF of the Standard Normal distribution N(0, 1). So, the PDF of the largest order statistics is:

f N (x) = NF N -1 (x)F ′ (x) = NF N -1 (x)f(x) (4.25)
where f(x) is the PDF of the N(0, s 2 ) distribution.

Let's find the mode of this distribution. To this end we shall find the derivative of the f N (x) and set it to zero. So:

df N (x) dx = N (N -1)F N -2 (x)f 2 (x) + F N -1 (x)f ′ (x) = N(N -1) 1 2πs 2 e -x 2 /s 2 F N -2 (x) + N -x √ 2πs 3 e -x 2 /2s 2 F N -1 (x) (4.26)
Equating the above to zero, after simplifications we get:

(N -1) 1 2πs 2 e -x 2 /s 2 - x √ 2πs 3 e -x 2 /2s 2 Φ(x/s) = 0 (4.27) or (N -1) 1 2πs 2 e -x 2 /s 2 - x √ 2πs 3 e -x 2 /2s 2 Φ(x/s) = 0 (4.28) or (x/s)e x 2 /2s 2 Φ(x/s) = N -1 (4.29)
or

ze z 2 /2 Φ(z) = N -1 (4.30)
where z = x/s or x = zs.

It is evident that the above equation will always have a single root (the left hand side is negative for z < 0, increasing with z and goes to infinity when z → ∞). Naturally, this equation may be solved only numerically. Suppose z 0 (N) is a root of the equation (4.30) (obviously z 0 depends only on N). Then the x 0 = z 0 (N)s is the Maximum likelihood estimate of the highest order statistics of the Normal idd N(0, s 2 1 ) sample. Because of the symmetry, the -x 0 = -z 0 (N)s is the Maximum Likelihood estimate of the highest order statistics of the Normal idd N(0, s 2 1 ) sample. Then the range may be estimated as (keeping in mind the assumption that for large N the highest and smallest order statistics may be regarded as independent):

R N = 2x 0 = 2z 0 (N)s 1 (4.31)
Now we need to calculate the standard deviation of the sample dy 1 (1), ..., dy 1 (N). For the purpose of this calculation we can assume that dy 1 (t) = e 1 (t) -e 1 (t -1) (a constant may not change a standard deviation).

sd N = V N V N = 1 N N i=1 (e 1 (t) -e 1 (t -1)) 2 (4.32)
Obviously, V N and sd N are random variables. We shall calculate the expected value and the variance of V N to show that the expected value of V N converges to a constant and the variance of V N converges to zero (as N → ∞), and therefore the V N converges in probability to a constant. So sd N will converge to the square root of the constant.

Let's define two vectors d1 = (dy 1 (1), ..., dy 1 (N)) T (N-dimensional vector-column) and ē1 = (e 1 (0), e 1 (1), ..., e 1 (N)) T ((N + 1)-dimensional vector-column). Then:

d1 = A ē1 (4.33)
where A is N × (N + 1) -matrix:

A =       -1 1 0 • • • 0 0 0 -1 1 • • • 0 0 . . . . . . . . . . . . . . . . . . 0 0 • • • • • • -1 1       (4.34)
Therefore the vector d1 will have the covariance matrix Σ d :

Σ d = AΣA T (4.35)
where Σ is the covariance matrix of ē1 :

Σ =       s 2 1 0 0 • • • 0 0 0 s 2 1 0 • • • 0 0 . . . . . . . . . . . . . . . . . . 0 0 • • • • • • 0 s 2 1       (4.36)
Therefore we have:

Σ d =          2s 2 1 -s 2 1 0 • • • 0 0 -s 2 1 2s 2 1 0 • • • . . . 0 . . . . . . . . . . . . . . . . . . 0 0 • • • • • • 2s 2 1 -s 2 1 0 0 • • • • • • -s 2 1 2s 2 1          (4.37)
Since the above matrix Σ d is symmetric and positive definite, we can always find such orthogonal matrix U with columns of zero length, that:

UΣ d U T = Λ (4.38)
where Λ is a diagonal matrix with eigenvalues (λ 1 , ..., λ N ) of the matrix Σ d on the diagonal.

The sample standard deviation of the sample dy 1 (1), ..., dy 1 (N) is:

V N = 1 N N i=1 (dy 1 (i)) 2 = 1 N d T 1 d 1 = 1 N d T 1 U T Ud 1 = 1 N (Ud 1 ) T (Ud 1 ) (4.39)
The vector Ud 1 = (u 1 , ...u N ) will have multivariate Normal distribution with zero expectation (for each component) and covariance matrix UΣ d U T = Λ. Since Λ is a diagonal matrix, the elements of the vector Ud 1 will be independent. So, we may write:

E(V N ) = E 1 N N i=1 u 2 i = 1 N N i=1 E(u 2 i ) = 1 N N i=1 λ i = 1 N trace(Σ d ) = 2s 2 1 (4.40)
and

V (sd N ) = V ( 1 N N i=1 u 2 i ) = 1 N 2 N i=1 V (u 2 i ) = 2 N 2 N i=1 λ 2 i = 2 N 2 trace(Σ 2 d ) (4.41)
It is not difficult to see that the diagonal elements of Σ 2 d are all the same and equal 2 2 s 4 1 + 1 2 s 4 1 = 5s 4 1 . Therefore:

V (sd N ) = 2 N 2 trace(Σ 2 d ) = 2 N 2 5Ns 4 1 = 10s 4 1 N → 0 (4.42)
And we may conclude that the V N converges in probability to the own mean 2s 1 . Consequently, sd N converges in probability to √ 2s 1 .

sd N → 2V (e 1 ) (4.43)
It is important to note here that the above result will be true for other distributions of e 1 for as long as the distribution has zero mean, is symmetric and has finite variance s 2 1 . The only change will be that the vector Ud 1 will have uncorrelated components having zero expectations (in case of Normal distributions, the components are independent). However, this does not change further proof. Now we can conclude that the rescaled range R/S for large N may be written as:

R/S(N) = R N sd N = 2z 0 (N)s 1 √ 2s 1 = √ 2s 1 z 0 (N) (4.44)
The series R/S(N) needs to be fitted to the power function:

R/S(N) = cN H (4.45)
Alternatively, the (4.45) may be written as:

N = R/S(N) c 1/H (4.46)
or

N = √ 2s 1 z 0 (N) c 1/H = c 1 (z 0 (N)) H ′ (4.47)
where H ′ = 1/H and c 1 is a constant:

c 1 = √ 2s 1 c 1/H (4.48)
At this stage we can make an important conclusion: the Hurst exponent for the LR process does not depend on s 1 (as well as other parameters of the process, a and b). In other words, all LR processes have the same Hurst exponent! In what follows, we shall use notation L as the Hurst exponent of the LR process.

To estimate the value of the Hurst exponent L for the LR process we need to find such c 1 and L = 1/L ′ that provide the best fit for the following (combine the equations (4.30) and (4.47)):

ze z 2 /2 Φ(z) ≈ c 1 z L ′ -1 (4.49)
The numerical simulations show that for small z (and respectively, small N, e.g. N < 30) the fit is quite poor. However, for large N (and z) the fit is very good. The figure 4.1 shows both curves for 30 ≤ N ≤ 100. We performed numerical estimation of the Hurst Exponent for using interval 100 ≤ N ≤ 500. The estimate we received is: L = 0.10107 (or L′ = 9.89412). In 1000 direct simulations (that is, we simulated LR sequences and estimated H using standard techniques) we had min L = 0.075 and max L = 0.155. This fully complies with the above estimate.

LR Process: Uniform Errors

It is common now to criticise the use of Normal distribution for errors in Financial Time Series. It will be interesting to see the value of the Hurst exponent for the processes having non-normal errors. In this section, we shall look at the Uniform distribution (which has no tails and the theoretical range is finite). In the next section, we shall look at the Laplace-distributed errors which is known to have heavy tails. So, we shall now evaluate the Hurst exponent for the LR process with e 1 (t) ∼ Uniform(-√ 3s 1 , √ 3s 1 ). The choice of the interval is dictated by the goal of having E(e 1 (t)) = 0 and V (e 1 (t)) = s 2 1 .

As before, we need to estimate the range

R N = e (N ) 1 -e (1)
1 . The CDF and the PDF of the highest order statistic e (N ) 1 are:

F N (x) = x + √ 3s 1 2 √ 3s 1 N (4.50) f N (x) = (x + √ 3s 1 ) N -1 (2 √ 3s 1 ) N (4.51)
Obviously, the f N (x) is an increasing function and the mode is √ 3s 1 , which is independent of N. So, if we use the ML method for the estimation of range we will have that R N is a constant. Since sd N is also a constant we will have H = 0. Therefore, we may try to use the mean of the distribution. The mean of the distribution (4.51) is:

E(e N (t)) = √ 3s 1 - √ 3s 1 x (x + √ 3s 1 ) N -1 (2 √ 3s 1 ) N dx = √ 3s 1 N -1 N + 1 ≈ √ 3s 1 (4.52)
for large N. And again we arrive to the conclusion that H = 0.

We can also use the equation (4.12). Skipping some boring details the PDF of the range may be written as:

f r (x) = 2 -N 3 -N 2 (N -1)N 2 √ 3s -x x s 1 N x 2 (4.53)
Taking the derivative and equating it to zero will give us the mode (the most likely value of the range). Ignoring the root x = 0 we get:

RN = 2 √ 3(N -2)s N -1 ≈ 2 √ 3s 1 (4.54)
for large N. Essentially this is the same result as in the equation (4.52) (double the expected value of the highest order statistic).

Computer simulation confirms this result (for 100 ≤ N ≤ 500 we received H = 0.0087). As it was for the Normal errors, the value of the Hurst exponent does not depend on s 1 . So, we can see that in both cases (Normal and Uniform errors) the Hurst exponent is very small. We can assume that H = 0 for the Uniform errors (for any practical purpose). However, in the case of Normally distributed errors, the Hurst exponent is small but is not equal to zero!

LR Process: Laplace Errors

It is common now in Financial Time Series to believe that the errors have heavy-tailed distributions. It will be interesting to see the value of the Hurst exponent for one of such LR processes. The Laplace distribution is the one of simplest distributions with heavy tails. So, we shall now evaluate the Hurst exponent for the LR process with e 1 (t) ∼ Laplace(0, s 1 / √ 2). The choice of the parameters is dictated by the goal of having E(e 1 (t)) = 0 and V (e

1 (t)) = s 2 1 .
As before, we need to estimate the range

R N = e (N ) 1 -e (1)
1 . The PDF of the highest order statistic e (N ) 1 is:

f N (x) = √ 2n s 1 1 - 1 2 exp(- √ 2x/s) N -1 + 2 exp( √ 2x/s) -1 (4.55)
It is not difficult to check that the f N (x) has a single mode which may be found by equating the derivative to zero. Skipping some details we get the following equation for the mode:

1 - 1 2 exp(- √ 2x/s 1 ) N 2 exp( √ 2x/s 1 ) -n = 0 (4.56)
The above equation has a single root:

x = s 1 √ 2 (log N -log 2) (4.57)
So, our estimate for the range R N is:

R N = √ 2s 1 (log N -log 2) (4.58) or R N sd N = log N -log 2 = log(N/2) (4.59)
for large N.

The log(N/2) is well approximated by the power law (cN H ). A computerised fit on the interval 100 ≤ N ≤ 500 shows solid fit with c = 1.527 and H = 0.208 (see the figure 2.2). As it was shown before, the value of the Hurst exponent does not depend on s 1 . The Student distribution (t-distribution) is also a heavy tailed distribution. The parameter of the distribution -number of degrees of freedom ν -defines the "heaviness" of the tails.

Unlike the Laplace distribution this distribution is not quite as easy for analysis. So, we are skipping all the multi-line formulae and present the summary of results only. The PDF of the highest order statistic has a single mode and this mode may be very accurately found numerically using modern software. The fit of the rescaled range to the power law is also quite good.

For ν = 1 (Cauchy distribution) full RS analysis is not possible, because this distribution does not have finite variance. However, if we estimate the Hurst exponent on the range alone, this estimate is Ĥ = 0.988. Essentially, the power law in this case is a straight line. This distribution has the heaviest tail.

The value of H quickly decreases as ν increases and the tails become lighter. For example, if ν = 2, Ĥ = 0.54 (also based on the range alone) and ν = 4, Ĥ = 0.36.

When ν = 6, the Student distribution has the same kurtosis as the Laplace distribution (six). The value of the Hurst exponent is very close to that of the Laplace case: Ĥ = 0.22

Hurst Exponent for LAR Process

It is not difficult to show that the sd N for LAR process also converges to a constant when N → ∞. Indeed:

m N = b + 1 N (e 2 (1) + ... + e 2 (N)) ∼ N(b, s 2 2 /N) (4.60)
where N(b, s 2 2 /N) is a Normal distribution with mean µ and variance s 2 2 /N (due to the Central Limit Theorem). Therefore, m N ≈ b for large N. So, we have:

sd N = e 2 2 (1) + ... + e 2 2 (N) N ∼ N(s 2 2 , 2s 4 2 /N) (4.61)
where N(s 2 2 , 2s 4 2 /N) is a Normal distribution with mean s 2 2 and variance 2s 4 2 /N. Therefore, sd N → s 2 2 when N goes to infinity.

So, we may conclude that the Hurst exponent for the LAR process (as it was for the LR process) is defined from the behaviour of the range R N of the cd N (t). It may be shown that the range R N follows the power law (for large N):

R N = cN 0.5 (4.62)
and therefore the Hurst exponent for the LAR process will always be 0.5 (regardless of the distribution of the errors e 2 (t)). The same may be found in [START_REF] Mandelbrot | The Misbehavior of Markets: A Fractal View of Financial Turbulence[END_REF].

Axiomatic Definition of Persistence: Introduction

Persistence in a time series is the ability of the series to keep the direction of change (in other words if last increments were positive, the next increment is likely to be positive as well; it is the same in the case of a negative increment). The persistence in a series is also referred to as a memory: stronger persistence means longer memory in a series. The first mentioning of a persistence was in publications of Hurst. Hurst studied discrete-time series with applications in hydrology (Nile water inflows). Mandelbrot later introduced Fractal Brownian Motion (FBM)a continuous-time stochastic process which is believed to be a continuous analogy of the discrete processes simulated by Hurst. The FBM is apparently a generalisation of a Brownian Motion Process (BMP). While the BMP has two parameters (drift and volatility) the FBM, in addition, has a parameter known as Hurst exponent h, which serves as a measure of persistence. If 0 < h < 0.5, the FBM is a mean reverting stationary series. If h = 0.5, the series is a random walk. For 0.5 < h < 1, the series exhibits a persistent behaviour (bigger h stands for bigger persistence). Interestingly a series with h close to one, shows a strong trend even for series with zero drift (however different realizations of the process may either grow or decline for exactly the same process parameters). Today, the Hurst exponent is widely used as a measure of persistence with or without reference to FBM. The applications include Finance, Medical Sciences, Signal Processing, etc. A wide variety of techniques are available to calculate the exponent. These include: Rescaled Range (aka R/S) analysis, Detrended Fluctuation Analysis (DFA), wavelet methods, etc.

Issues with Hurst Exponent

Ideally, the persistence in a series should not depend on:

• drift;

• volatility;

• type of the distribution of the increments. and be constant (at least approximately) for the whole series.

Unfortunately, not all of the above is always true for the Hurst exponent. While the above is true for an FBM process (and similar), it is not so for other common models. For example, if a series is a simple linear regression process, the Hurst exponent (if calculated wit R/S method) does depend on the type of the distribution ( [START_REF] Litvine | Ll model -theory and applications[END_REF]). Also, in practice the Hurst exponent may vary considerably. For example, it was shown that for well known financial indicators (e.g. Nikkei, GSTPSV, etc.) the value of Hurst exponent (computed by DFA algorithm for windows in data) fluctuates between 0.3 and 0.9, and the value depends not only on where the window was positioned, but also on the size of the window.

Another problem is that certain non-parametric methods for estimating the Hurst exponent (e.g. Detrended Fluctuation Analysis aka DFA) may give values of ĥ outside of the (0, 1) interval.

The idea of this paper is to define persistence (for discrete-time series) as a measure satisfying certain natural properties and to suggest some simple indicators, satisfying these properties. Applications to simulated and true series will be attempted.

Axioms and Properties

Suppose we have a time-series of real numbers y(i), i = 1, ..., n. The following axioms refer to the random process itself. The statements we call "properties" refer to realizations ŷ(i), i = 1, ..., n of the process y(.). Also l refers to the true persistence parameter of the series, while l refers to the persistence estimated from a realisation.

We shall keep in mind that informally, a persistence is understood as an ability to preserve (to a given extent) the direction of change. A parameter l will be regarded as a persistence of the series if it satisfies the following axioms: Axiom 1. (Sign Transform). The persistence of the series y(i) is the same as persistence of -y(i). This is a very obvious yet a very important axiom. Clearly, the persistence is not supposed to depend on the direction of growth (upwards or downwards). Property 1. (Sign Transform). The persistence of the series ŷ(i) is the same as persistence of -ŷ(i).

Axiom 2. The persistence is not sensitive to a linear transformation: z(i) = a + by(i), b = 0. In other words, the two series y(i) and z(i) have identical persistence.

Let's denote g(y) = a + by, b > 0. The motivation for this axiom is the following: P (g(y(i + 1)) -g(y(i)) > 0|g(y(i)), g(y(i -1)), ..., g(y(1))) = P (g(y(i + 1)) > g(y(i))|g(y(i)), g(y(i -1)), ..., g(y(1))) = P (y(i + 1) > y(i)|g(y(i), g(y(i -1)), ..., g(y(1))) = (4.63)

Since g is monotone, it is one-to-one function and conditions g(y i ), g(y(i -1)), ..., g(y(1)) may be replaced with y(i), y(i -1), ..., y(1): P (g(y(i + 1)) -g(y(i)) > 0|g(y(i)), y(i -1)), ..., g(y(1))) = P (y(i + 1) > y(i)|y(i), y(i -1), ..., y(1)) = P (y(i + 1) -y(i) > 0|y(i), y(i -1), ..., y(1))

A much stronger statement can be made in case of realisations.

Property 2. (Monotone Transform). Let g(y) be any strictly monotone non-random function (either strictly increasing or strictly decreasing). Then the series g(y(i)) will have the same persistence l as the series y(.).

Consider the case of increasing g(.):

if y(i + 1)) -y(i) > 0, then g(y(i + 1)) > g(y(i)) (4.65)
Corollary 1. Clearly, if the persistence is not sensitive to any monotone transformation, it will not be sensitive to a linear transformation:

z(i) = a + by(i), b = 0.
Corollary 2. The persistence parameter (both l and l) does not depend on the drift and intercept of a series. Moreover, the l does not depend on any non-linear trend for as long as this trend is monotone. Axiom 3. (Multiplicative Transform). Let a i be a sequence of positive numbers. Then the series z(i + 1) = z(i) + a i (y(i + 1) -y(i)) will have the same persistence l as the series y(.).

Property 3. (Multiplicative Transform). The same property is true for realisations.

Again, it is easy to show that the sign of both increments will be always equal: sign(z(i + 1) -z(i)) = sign(y(i + 1) -y(i)). Corollary 3. Let a i be a sequence of negative random numbers. Then the series z(i + 1) = z(i -1) + a i (y(i + 1) -y(i)) will have the same persistence as the series y(.). This corollary follows from the last axiom and the axiom 1.

Corollary 4. A series with persistence l ( l) and volatility σ may be transformed into series with the same persistence and any volatility σ 1 > 0. Axiom 4. (Scaling). The persistence measure is between zero and one: 0 ≤ l ≤ 1, moreover,

• The persistence measure is zero if and only if:

l = 1 ⇐⇒ P (y(i + 1) -y(i))(y(i) -y(i -1)) < 0) = 1 (4.66)
In other words, the series always moves in the direction opposite to the previous move.

• The persistence measure is one, if and only if:

l = 1 ⇐⇒ P (y(i + 1) -y(i))(y(i) -y(i -1)) > 0) = 1 (4.67)
In other words, the series has all increments of the same sign with probability one.

• The persistence measure is half, if and only if: l = 0.5 ⇐⇒ P (y(i + 1) > y(i)|y(i), ..., y(1)) = 0.5 (4.68) Property 4. (Scaling). The persistence measure is always between zero and one: 0 ≤ l ≤ 1, moreover,

• The persistence measure is zero if and only if:

l = 1 ⇐⇒ (ŷ(i + 1) -ŷ(i))(ŷ(i) -ŷ(i -1)) < 0 for all i = 2, ..., n -1 (4.69)
In other words, the series always moves in the direction opposite to the previous move.

• The persistence measure is one, if and only if:

l = 1 ⇐⇒ (ŷ(i + 1) -ŷ(i))(ŷ(i) -ŷ(i -1)) > 0 for all i = 2, ..., n -1 (4.70)
In other words, the series has all increments of the same sign.

• The persistence measure is half, if and only if the inequality (ŷ(i+1

)-ŷ(i))(ŷ(i)-ŷ(i-1)) < 0 is true for exactly (n -2)/2 points i(i = 2, ..., n -1).
In other words, out of n -2 possible points the series changes the direction of move in half of the cases.

Note that the Axiom 4 (and property 4) almost exactly mirror the properties of the Hurst exponent. We say "almost" since the Hurst exponent cannot be equal zero nor one (however may be infinitely close to these values). Property 5. (Time Reversing). The persistence of series ẑ(t) = ŷ(n -t -1) is exactly the same as the persistence of series ŷ(t). Axiom 5. (Time Reversing). Suppose we have a model of a random process with persistence parameter l. Then this parameter has unbiased estimate l that does not change value if the series is reversed:

l(y 1 , ..., y n ) = l(y n , ..., y 1 ) (4.71)
This last axiom (and the property) look the most controversial, specifically, if one relates the persistence to "memory". Essentially this property means that a series with hight persistence will "remember" not only the past, but also the future.

Example: Random Walk with Persistence

To show that the above set of the axioms is consistent (in other words we need to show it is not self-contradictory), we need give at least one example of persistence parameter that satisfies all the axioms. Consider the following discrete-time autoregressive random process:

x(0) = x 0 , x(1) = x 1 , where x 0 and x 1 are any constants.

x(i + 1) = x(i) + v(i)(u i + l)sign(x(i) -x(i -1)) (4.72)
where u i ∼ Uniform(-1, 0), iid random variables, v(i) > 0 -sequence of positive numbers. In what follows we shall refer to the above process as a random walk with persistence (RWP).

We can show that the model parameter l satisfies all axioms. All the proofs are quite obvious, so we shall only do the proof for axiom 3.

Let series x(.) satisfy the equation 4.72, a i be a series of positive numbers and series z(.) satisfies:

z(i + 1) = z(i) + a i (x(i + 1) -x(i)) (4.73)
Firstly we note that z(i + 1) -z(i) = a i (x(i + 1) -x(i)) and the increments of x(.) and z(.) will be always of the same sign. Then we have:

z(i + 1) = z(i) + a i (x(i + 1) -x(i)) = z(i) + a i (x(i) + v(i)(u i + l)sign(x(i) -x(i -1)) -x(i)) = z(i) + a i v(i)(u i + l)sign(z(i) -z(i -1)) = z(i) + v 1 (i)(u i + l)sign(z(i) -z(i -1)) (4.74)
So, z i is also a RWP process with the same persistence parameter l.

Example: Persistence Estimator

Now we shall prove the consistency of the properties 1-5, giving a non-parametric (i.e. modelfree) estimator of the persistence in observed series.

Definition 1. If for certain i we have ŷ(i -1) < ŷ(i) > ŷ(i + 1), then we say that the series y(.) has a local peak in this point i. Definition 2. If for certain i we have ŷ(i -1) > ŷ(i) < ŷ(i + 1), then we say that the series y(.) has a local trough in this point i.

Note, that a series has either peak or a trough if and only if (ŷ(i+1)-ŷ(i))(ŷ(i)-ŷ(i-1)) < 0. Definition 3. The empirical persistence (EP) in a series ŷ(.) will be defined as:

l = 1 - n 1 + n 2 n -2 (4.75)
where n 1 is the number of local peaks in the series, n 2 is the number of local troughs and the n is the series length. Note that the denominator in the above equation is n -2 since the first and the last points cannot be peaks nor troughs.

It is obvious that the EP satisfies all the above properties. Indeed, the sign change turns peaks into troughs and troughs into peaks, while the monotone transform, the multiplicative transform and time reversing keep the peaks and troughs in their positions. The scaling property is also obvious.

It is also important to note that this estimator gives an unbiased estimate of the parameter l in the process (4.72).

Simulations

In this section, we shall compare the Hurst exponent H with our EP estimator l for a range of simulated processes. All calculations were done in Wolfram Mathematica R .

Fractional Brownian Motion

We simulated a number of series (each series had 10 000 points) for H ranging from 0.01 to 0.99 (with step 0.01), several hundreds observations per each H. The following graph (fig. 1) represents the relationship between the pre-defined H and the average of l as a function of H. Separately, we generated 10000 observations for H = 0.0001, H = 0.9999 and H = 0.5. From these simulations we conclude that l → 1/3 when H → 0, l → 1 when H → 1 and l = 1/2 when H = 1/2.

The conclusions one can make from the graph (fig. 4

.3) are as follows:

• There is a one-to-one (monotone) correspondence between 0 < H < 1 and 1/3 < E( l) < 1;

• This relationship is non-linear;

• Knowing H one may estimate l;

• Knowing l an estimate of H may be found (given 1/3 < l < 1).

Using the "try-and-error" method, we arrived to the following model that we suggest for relating H and l: The most remarkable fact is that the above algorithm has complexity O(n) (linear time). All other known algorithms for estimation of H have complexity O(n 2 ) or more. On the negative side, we note that the equation (4.76) cannot be solved for H analytically and one will need to solve it numerically (alternatively one may use pre-calculated tables or estimate H from graph on figure 4.3).

l = ab H d (1-H) c

Random Walk With Persistence

We simulated a number of series (each series had 1 000 points) for l ranging from 0.01 to 0.99 (with step 0.01), one hundred observations per each l (the length of series and number of repetitions was dictated by relatively long time required for the DFA (De-Detrended Fluctuation Analysis) algorithm, which was used for the estimation of H. The following graph (fig. 4.5) represents the relationship between the pre-defined l and the average of Ĥ as a function of l (dotted image). For reference, we also show on the same graph the relationship between l and H from figure 4.3 (continuous line). • There is a monotone relationship between l and Ĥ (also confirmed by Spearman's correlation test);

• Both lines intersect at the point H = 0.5, l = 0.5. This means that both Ĥ and l may be equally used to distinguish between persistent and anti-persistent series;

• Ĥ for RWP process ranges from approximately 0.4 to 1.2. This contradicts the theory which suggests that the Hurst exponent should be between zero and one. In our view this is the result of two factors (a) the series is not a Fractional Brownian Motion; and (b) the DFA method may give non-feasible estimates for persistence parameter (if given series is not an FBM).

Persistence of a Random Sequence

Persistence of a random sequence may be regarded as an important benchmark. So, assume that y(1), ..., y(n) is a sequence of iid random numbers. If we consider it to be a time series, what is the value of l?

Consider the expected value of l, as defined in equation (4.75):

E(1 - n 1 + n 2 n -2 ) = 1 -E( n 1 + n 2 n -2 ) =
1 -P (y(i -1) < y(i) > y(i + 1) or y(i -1) > y(i) < y(i + 1)) = P (y(i -1) < y(i) < y(i + 1) or y(i -1) > y(i) > y(i + 1)), where 2 ≤ i ≤ n -1. In other words, we may define l as a probability of a randomly chosen point not be neither peak nor trough. We have: l = P (y(i -1) < y(i) < y(i + 1)) + P (y(i -1) > y(i) > y(i + 1))

There are six possible ways to order values y(i -1), y(i) and y(i + 1). All six possibilities are equally likely (as y's are independent and identically distributed). So, each possibility has probability 1/6, and, therefore l = 1/3. Note that if we formally apply other methods for estimation of Hurst exponent to the random sequence we will receive different estimates. For example, if we utilise the R/S analysis we will see that Ĥ depends on the distribution of y(i) and may vary in a very wide range.

Another interesting observation is that random sequence has the same value of l as the FBM process for H → 0.

Generalised Hurst Exponent

Definition 4. Let ŷ(i), i = 1, ..., n be a realization of a random process., l be defined by formula (4.75). Then Non-parametric Generalised Hurst Exponent (NGHE) Ĥg is defined as a solution of the equation (4.76) for H. Definition 5. Let y(i), i = 1, ..., n be a random process with persistence parameter l. Then Parametric Generalised Hurst Exponent (PGHE) H g is defined as a solution for H of the equation (4.76), where l is replaced with l.

It is obvious that the Ĥg will be approximating the Hurst exponent Ĥ if the process is the FBM process. However, (for other processes, e.g for RWP process) the values of Ĥg will be between -∞ and 1.

The values of the Ĥg should be interpreted as follows:

1. If Ĥg = -∞ ( l = 0), positive and negative increments will alternate. We shall refer to this case as perfect negative predictability (one can predict the direction of change with certainty -it will be in opposite direction from the previous change).

2. If Ĥg = 0 ( l = 1/3), the series has same persistence as a iid sequence of random numbers. We shall refer to this case as perfect unpredictability of type 1.

3. If -∞ < Ĥg < 0 (0 < l < 1/3), we say that the series is partially negatively predictable (smaller Ĥg will mean stronger negative predictability).

4. If Ĥg = 1/2 ( l = 1/2), the series has same persistence as a random walk -the probability of positive increment will be the same as the probability of a negative increment (i.e. both probabilities will be 0.5) regardless of the past history. We shall refer to this case as perfect unpredictability of type 2.

5. If 0 < Ĥg < 1/2 (1/3 < l < 1/2), the persistence of a series will be between persistence of a sequence of random numbers and persistence of a random walk. We shall refer to this case as anti-persistence or perfect unpredictability of type 3.

6. If Ĥg = 1 ( l = 1), all increments will be of the same sign. We shall refer to this case as a perfect positive predictability (one can predict the direction of change with certainty -it will be always in the same direction).

7. If 1/2 < Ĥg < 1 (1/2 < l < 1), positive increments will be likely to be followed with positive increments and negative increments will be likely to be followed with negative increments. We shall refer to this case as partial predictability (larger Ĥg means greater predictability) or persistence.

One may question the value of the cases of negative Ĥg for research. However we may recall here that every point of the Elliott wave process (EWP) is either peak or though, that is, the generalised Hurst exponent for EWP is always negative infinity Ĥg = -∞ ( l = 0).

Conclusions

In this chapter, we studied issues related to persistence in time-series models. Persistence may be seen as opposite to cycling, therefore it is important to measure persistence correctly. In the beginning, we discussed and analysed the Hurst exponent which is the most common tool for measuring persistence. We derived the Hurst exponent for LR and LAR processes and showed that in LR process the Hurst exponent depends on the distribution of the errors, namely it depends on the peakedness (kurtosis) of the distribution and may vary from 0.1 for Normal distribution to 0.998 for Cauchy distribution. To the contrary, the Hurst exponent is fixed (H = 0.5) for the LAR process.

The above causes doubts as to whether the Hurst exponent is the best tool to measure persistence and leads to the idea of investigating what properties an ideal persistent measure would possess. Thereafter, we introduce a concept of axiomatic definition of persistence and give an example of a measure which satisfies all the axioms. We established the link between the new persistence measure and the Hurst exponent for the FBM process. This made it possible to suggest a simple and very fast algorithm for evaluation of the Hurst exponent.

Further, we suggested a generalisation of the Hurst exponent for the negative values of H. As an example of a process with negative H we offered EWP (Elliott Wave Process).

It seems that we have made a real breakthrough in the issue of persistence which can be extended further in future studies.

Chapter 5

Forecasting Cycles with ANN Models

Introduction

In this chapter, we use so-called feed-forward artificial neural network model FF-ANN for modelling econometric cycles. We consider the FF-ANN from a statistical and an econometric viewpoint. It was shown how this model can be estimated by maximum likelihood. Finally, we apply the ANN methodology to model the demand for electricity in South Africa. The comparison of forecasts based on a linear and FF-ANN model respectively shows the usefulness of the latter.

Artificial neural networks (ANN) subsume a set of models which have been developed in the cognitive sciences to understand functioning of the human brain. These models originated in the publications of McCulloch and Pitts [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF] and in the study of perceptron by Rosenblatt [START_REF] Rosenblatt | The perceptron: A probabilistic model for information storage and organisation in the brain[END_REF]. However, at that time, the capabilities of computing were very limited, so these early models were too simple to explain the complexities of the actual operation of the brain.

Consequently, with the growth of computing powers, more complex ANN structures and network learning methods were designed, peculiarly in the investigations of Rummelhart et al. [START_REF] Rummelhart | Parallel Distributed Processing: Explorations in the Microstructure of Cognition[END_REF] and McClelland et al. [START_REF] Mcclelland | Parallel Distributed Processing: Explorations in the Microstructure of Cognition[END_REF]. Since then, the research in neural networks has grown progressively.

In this study, we are interested in the application of the econometric approach to ANN models. From this viewpoint, the inspiring and path breaking contribution is that of Kuan and White [START_REF] Kuan | Artificial neuronal networks: An econometric perspective[END_REF]. More recently, Kuan [START_REF] Kuan | Artificial Neural Networks, The New Palgrave Dictionary of Economics[END_REF] gave a review of the matter from an econometric perspective. In brief, ANN models for an econometrician constitute a specific set of non-linear models and "learning" is understood as an estimation of model parameters.

There have been numerous applications of econometric ANN models in the field of market finance. For a presentation of the main results, see e.g. [START_REF] Franses | Non-linear Time Series Models in Empirical Finance[END_REF], chapter 5. Forecasting, especially macro-economic, was also an area to prospect well, as evidenced by the studies of Swanson and White [START_REF] Swanson | A model selection approach to real-time macroeconomic forecasting using linear models and artificial neural networks[END_REF] , McMenamin [START_REF] Mcmenamin | What not pi? a primer on neural networks for forecasting[END_REF], Zhang, G. et al. [START_REF] Zhang | Forecasting with artificial neural networks: The state of the art[END_REF], Rech [START_REF] Rech | Forecasting with artificial neural network models[END_REF], White [START_REF] White | Approximate Nonlinear Forecasting Methods[END_REF], Medeiros et al. [START_REF] Medeiros | Building neural networks models for time series: A statistical approach[END_REF], and Ozdemir et al. [START_REF] Ozdemir | Ann forecasting models for ise national100 index[END_REF], which is a short list in the vast literature on the subject.

The structure of this chapter is as follows. The next section presents the standard ANN model from a twofold viewpoint heuristic and econometric-statistical. The following section deals with the estimation and forecasting method of the considered model. Then we develop an application focused on the electricity production in South Africa. Finally, conclusions and research prospects are discussed in the next section.

The Model

A first heuristic approach

Neural networks models mimics the brain operation. They use therefore a specific terminology such as layer, nodes, output, input and so one. For example consider the simple "feed-forward neural network". The output is denoted by Y and the three inputs are X 1 , X 2 and X 3 . Nodes G 1 and G 2 represent so-called hidden layer. The basic idea is that the inputs feed into the nodes and in a second step they feed onward to the output layer. Thus there is no feedback.

The following diagram depicts a representation of the links between output and inputs structured in three layers. Evidently, in the reality, i.e. in biological neural systems, the number of processing units is enormous: they amount to billions. Such numbers are usually not considered in the economic models.

The ANN(k,q) model

Translating the previous diagram into a well-defined statistical model yields the following nonlinear equation:

Y = 2 i=1 G i (γ ′ x) (5.1)
where G i are activation functions, x is a column vector of inputs and the γ's are the parameters of the model.

Naturally, for practical purposes, the structure of equation (5.1) must be more complex. Therefore, from an econometric perspective, the single hidden-layer feed-forward model will be written as follows:

y t = α ′ z t + q j=1 , β j G(γ ′ j z t ) + ǫ t , t = 1, ...T, (5.2)
where y t is the dependent variable (output), z t = (1, y t-1 , ...y t-p , x 1t , ..., x kt ) ′ is the vector of the explanatory variables, including the constant and the delayed values of y t , γ and α are vectors of parameters and the G i are the activation functions. Consequently, the relationships between y t and x t are nonlinear (if G i are nonlinear). Moreover, as usual, ǫ t is a Gaussian iid random errors (noise) with null mathematical expectation and constant variance. Some discussion on activation functions is now required. These functions are usually restricted to those which have values between zero and one. From this viewpoint, the logistic function:

G(x) = 1 1 + e -x , -∞ < x < +∞ (5.3)
seems a reasonable choice. Mathematically, this kind of function is differentiable everywhere and its derivative is easily computed:

dG(x) dx = G(x)(1 -G(x)). (5.4)
However, many other choices are possible, such as smooth cumulative distribution functions, sine and cosine functions, hyperbolic tangent functions, etc. See [START_REF] Kuan | Artificial Neural Networks, The New Palgrave Dictionary of Economics[END_REF]. Furthermore, McMenamin [START_REF] Mcmenamin | What not pi? a primer on neural networks for forecasting[END_REF] has even proposed to use the π-based activation function, say π x , for which the derivative is given by: dπ x dx = π x ln(π) = 1.145 ln(π).

(5.5)

Nevertheless, all these mappings must also be bounded and should be asymptotically constant.

In total, equation (5.2) and the relevant specific activation functions constitute the ANN (k, q) model. Fundamentally, this model belongs to the class of nonlinear models and it subsumes many other models, well known in the econometric literature see e.g. Franses and van Dijk [START_REF] Franses | Non-linear Time Series Models in Empirical Finance[END_REF] such as the switching-regression model or the Smooth Transition Autoregressive (STAR) model. Better, as noted by Kock and Terasvirta [START_REF] Clements | Forecasting with Nonlinear Time Series Models, The Oxford Handbook of Economic Forecasting[END_REF], the ANN-model is also a so-called "universal approximator", just like the Kolmogorov-Gabor polynomial, all used to determine unknown nonlinear functional forms.

Estimation and Forecasting Method

The problem is drastically simplified if a ANN(k, 1) model is considered:

y t = α ′ z t + βG(γ ′ z t ) + ǫ t , t = 1, ..., T, (5.6)
where ǫ t ∼ N(0, σ 2 ) and G is the logistic function (5.3).

Estimation

The estimation strategy consists of two steps. In the first step we shall estimate the linear part of the model, obtaining the vector of parameters α by OLS. The second step of estimation yields the estimates of the vector of parameters γ. Of course, in this latter case, the estimation procedure is more intricate due to the non-linearity of the model. Practically, by using estimated parameters in the first step, the equation (5.6) can be written:

w t = y t -α′ z t = βG(γ ′ z t ) + ǫ t , ǫ t ∼ N(0, σ 2 ). (5.7)
Consequently, the likelihood function is given:

L c (β, γ) = T t=1 2 σ √ 2π e 1 2σ 2 (wt-βG(γ ′ zt)) 2 .
(5.8)

Finally, the log-likelihood function is:

ln L c (β, γ) = - T 2 ln(2π) -- T 2 ln(σ 2 ) - 1 2σ 2 T t=1 (w t -βG(γ ′ z t )) 2 .
(5.9)

So it remains to maximise this log-likelihood (5) using a usual algorithm such as that of Newton-Raphson or better, of Broyden-Fletcher-Goldfarb-Shannon (BFGS). For a discussion of the more specific algorithms relevant to neural feed-forward networks, see [START_REF] Fine | Feedforward Neural Network Methodology[END_REF], chapter 5. In our study, we used the built-in optimisation algorithm of Wolfram Mathematica.

Forecasting Methodology

Here the procedure involves two stages as well:

1. In a first step, the linear dynamic explicative model

y t = R r=1 δ r y t-r + K k=0 P k p=0 α ′ x k,t-p + u t (5.10)
is estimated using (T -h) observations and we retain h data points for out-of-sample prediction purposes. Forecasting can be then realized for these h future periods.

2. The second step, based on the estimation of the complete ANN model 5.6, yields the forecasts for the same forward periods and allows a direct comparison with the set of predicted values in the precedent stage.

Note, this prediction trial is a first attempt and our main objective is merely to show the utility of the ANN models.

Real Data Forecasting Exercise

Data

The series to be "explained" and to be used for forecasting, (denoted Elec), is that of the monthly electricity demand in South Africa for the period 2002-M1 to 2010-M6. It furnishes the production GWh. Three explicative variables were defined as:

1. The consumer prices of services index (denoted CPI), which gives a picture of the prices variation for all urban areas (2012-M12 = 100);

2. The total volume of manufactured production (denoted Prod), which constitutes an index with 2010 = 100;

3. A binary variable (denoted Rec), which takes the value one during the recessions and the value zero during the expansion phases of the economy. This dummy series was constructed based on the dating of the South African business cycle by Bismans and Majetti [START_REF] Bismans | Dating the south african business cycle[END_REF] and Bismans and Le Roux [START_REF] Bismans | Dating the business cycle in south africa by using a markov-switching model[END_REF].

All the series are monthly and have been down-loaded from the data base of the South African Reserve Bank website. The comparison of point predictions depicts that globally, the ANN-model is a better tool for forecasting in comparison to the linear model. To refine the analysis, two additional evaluation indicators are computed: the Root Mean Square Error (RMSE) and the Mean Absolute Percentage Error (MAPE). Finally, the Theils coefficient a statistic which compares the forecasts of the proposed model with a naive prediction given by a random walk process is equally implemented.

All the statistics deliver the same information: the predictive performance of the neural network model is superior to that of its linear counterpart. However, it is possible to push the interpretation further by considering the coefficient of Theil. From this viewpoint, both models used -linear and feed-forward neural network -do less well than a simple random walk, for which the best prediction of a variable in t is the observed value of this variable during the immediately preceding period.

Explaining this apparent paradox is not difficult: it follows from this that the strong seasonality in the series was not taken into consideration. Nevertheless, one must recall that the only objective of this contribution was to compare linear and ANN models from a forecasting perspective.

Hierarchical ANN Estimation

Definitions

Suppose we have the following non-linear model:

y(x 1 , ..., x n ) = α 0 + α ′ x + q i=1 β i Ψ(γ ′ i x + γ i0 ) + ǫ (5.12)
where x = (x 1 , ..., x n ) is an n-dimensional vector of explanatory variables, α j , (j = 0, 1, ..., n), β i , (i = 1, ..., q) and γ i = (γ i1 , ..., γ in ) (i = 0, 1, ..., q) are parameters, Ψ(x) = (1 + e -x ) -1 is the sigmoid function (aka activation function) and ǫ is the error term. All vectors are considered to be columns and prime represents transposition of a vector (or a matrix). This model may be explained in terms of the Artificial Neural Networks (ANN) theory. In this case the integer q represents the number of neurons in the hidden layer. Quite often the linear term α ′ x is not included in the model, which is generally not important, however, is essential for the approach which we are going to discuss.

Alternatively, the model may be presented as:

y(x 1 , ..., x n ) = α 0 + α ′ x + q i=1 β i Ψ(γ ′ i x) + ǫ (5.13)
where x and all γ i have an extra element: x = (1, x 1 , ..., x n ), γ i = (γ i0 , γ i1 , ..., γ in ) (i = 0, 1, ..., q). However, we shall use equation 5.12. The total number of parameters which have to be estimated is 1 + n + q + q(n + 1) = q(n + 2) + n + 1, hence a considerable number of observations are required to have an accurate estimation.

The most serious obstacle in the application of ANN modelling is that the computational process (often called training of a network) takes considerable time. Therefore, a number of attempts to suggest faster algorithms were conducted. Among these attempts one may find a number of randomised algorithms. Our technique also belongs to this category.

In this approach we shall assume that the explanatory variables are contained in certain finite intervals a i ≤ x i ≤ b i , which is often the case (e.g. in econometric and technometric applications).

Sigmoid

Firstly, consider the sigmoid function. The graph of the function is given in figure 5.3. At this point it may be mentioned that the sigmoid may be seen as a cumulative distribution function (CDF) of the logistic distribution with parameter λ = 1. The standard deviation of the distribution is σ = π/ √ 3 ≈ 1.8. One may notice that on the interval (-σ, σ), the sigmoid is practically linear (correlation coefficient is 0.998). Now it becomes clear why the inclusion of the linear term is optional -selection of sufficiently small components of γ i (which may be compensated by large β i ) secures that Ψ(γ ′ x) ≈ 1/2 + γ ′ x/4.

Figure 5.3: Sigmoid

One should notice that the sigmoid has quite strong "linearity" even on a wider range. For example, the correlation coefficient of the sigmoid on the interval (-10, 10) equals 0.936.

Basics of Hierarchical Estimation

In this section, we shall introduce the hierarchical estimation of time series models using a simple example of polynomial estimation.

Firstly we shall note that many models for time series have the following general form:

y(x; a) = q i=1 f i (x; a i ) + ǫ (5.14)
where x -is an independent variable (or a vector of independent variables), a = (a 1 , ...a q )vector of unknown parameters (where a i may be also a vector), ǫ -is a random error, f i (x; a i ), (i = 1, ..., q) -is a set of similar functions of certain kind. One may see that the ANN model is also from this category of models with

f i (x; a i ) = β i Ψ(γ ′ i x + γ i0 ) (5.15)
It may be noted here that a huge number of popular models fall into this class, e.g. multivariate linear regression, generalised linear regression, polynomial regression, Fourier series, and many others.

To demonstrate the hierarchical approach we shall fist consider it on polynomial regression. That is when:

f i (x; a i ) = a i x i , i = 0, ..., q (5.16)
Also, we shall pretend (for reasons which will be explained later), that we do not know anything about Ordinary Least Squares (OLS) technique, moreover, we shall rely only on common sense and from Statistics we shall use ONLY the concept of the sample mean (average of the observations).

Firstly, we simulate a series:

y x = a 0 + a 1 x + a 2 x 2 + ǫ (5. 17
)
where a 0 = 1, a 1 = 3, a 2 = -2, ǫ ∼ N(0.0.03) -independent, Normally distributed random errors (see figure 5.4). Independent variable x runs from 0 to 1 at 1/1000 intervals. Then we start our hierarchical procedure with fitting the data to f 0 (x, a) = a 0 . Purely from common sense considerations, we conclude that we can estimate a 0 as ȳ = (y 0 + ... + y n )/n = 1.82286 (see figure 5.5). This will be our first approximation for the model. This is quite a rough approximation, however, it has one good property: the sum of the residuals ǫ 1 (x) = y(x) -ȳ is zero. In the coming steps of the approximation, we shall keep this property. We would like to emphasise the importance of appropriate interpretation of the current section:

• We do not advocate the use of the hierarchical method for the polynomial regression;

• Moreover, we do recommend to use the OLS methodology whenever it works, as it has many optimal properties;

• We used the simple scenario of the polynomial fit merely to illustrate the use of the hierarchical technique on an easy and straightforward example;

• In the next section, we shall attempt to apply this technique to fitting ANN models, where OLS normally shows extremely poor performance.

Hierarchical Technique for ANN Estimation

The ANN models are quite different to classic statistical models, because of the following:

• Relatively high number of unknown parameters which should be estimated. As was shown earlier, the number of parameters is q(n + 2) + n + 1. For example, if a model has four inputs (independent variables) and 3 cells in the hidden layer (n = 4, q = 3), then the number of parameters will be 23.

• From this prospective the ANN models cannot be regarded as parsimonious;

• Due to the same reason the number of observations should be quite high;

• Numerical methods based on OLS normally fail in ANN estimation due to two reasons:

-The substantial number of unknown parameters makes optimisation slow and unstable and -The objective function normally has multiple local minimums and one may not be sure if the answer represents the global optimum.

In this section, we shall use the suggested approach to estimate parameters of the ANN models. The approach here was inspired by the QuickNet algorithm suggested by Halbert White (Approximate Nonlinear Forecasting Methods). Again, we shall do it by means of an example. The example we shall use is a classical example of ANN modelling. We shall fit the ANN network to the sinusoid function y(x) = sin(x) where 0 ≤ x ≤ 2π (x is a single input and y is the output. It is known that this function may be very accurately approximated with the ANN network with 3 hidden neurons:

y(x) = sin(x) ≈ 3 i=1 β i Ψ(γ 1i x + γ 0i ) (5.22)
White suggests that at each step we select γ 1i and γ 0i in such a way that the absolute value correlation between the function and the Ψ(γ 1 ix + γ 0 i) is the highest. It is not difficult to show that in the first step there is a single maximum. Now it is not difficult to find the best β 1 and the result is shown in figure 5.8. This is a reasonably good fit and the model looks quite adequate. However, we know that even better fit is possible. So, we upgrade the hierarchical process with another step: we use the current model as an initial approximation and perform OLS optimisation starting from this point in 10-dimensional space. The result is shown in figure 5.11 (the two lines overlap, so we extended the range to see that the curves split after x = 2π). The table of optimal parameters is given in the figure 5.12. One may see that all parameters are significant. Figure 5.12: Optimal fit parameters One important observation may be emphasised here. If we calculate the absolute correlation between sin(x) and the first summand of the model (in optimal fit), we shall get ρ ≈ 0.85, while the correlation at the first step of the hierarchical method was ρ ≈ 0.93. This means that the hierarchical method was leading away from the optimal model starting from step 1 and that the last step (we can call it "fine-tuning") was absolutely necessary.

It makes sense to suggest that using the absolute correlation coefficient as a tool for selecting best γ i,j in hierarchical process is not appropriate and some other measure of goodness should be suggested.

Conclusions

Undoubtedly, the study has proved empirically that the ANN-model demonstrated superior predictive properties over the linear one. However, two limitations of the canonical ANN benchmark structure will be exceeded in the future: on one hand, considering one node in the hidden layer is a simplification, at best temporary, to be abandoned. On the other hand, the seasonality should be modelled explicitly. So, we do notice space for improvement. ANN models represent a real challenge for the estimation of parameters. Therefore, we made a quite successful attempt at introducing a new approach to estimate parameters. The method was applied to fit sinusoid and showed an exceptionally great fit in terms of accuracy.

On the other hand, one should note that while ANN models are quite accurate, they lack two important properties that a forecasting model ideally should have: they are neither adequate nor parsimonious. without fixed periods.

In such cases, it seems feasible to use techniques related to the concept of persistence (which is in a sense opposite to cycling). Persistent series have less cycles, while anti-persistent series may change direction or change even at every point of series.

Moving in this direction, we introduced the concept of axiomatic definition of persistence and suggested a parameter that measures this persistence. We revealed the relationship of this parameter with the Hurst exponent, which is currently widely used for assessing persistence. As a side product, we suggested the fastest amongst all known methods of estimation of the Hurst exponent for Fractional Brownian Motion processes.

We also suggested an innovative method of dating irregular cycles. We called this method hierarchical, as it first finds biggest cycles, then breaking them into smaller ones. The clear advantage of this method is that it does not require censoring and delivers dating which is optimal as per objective function defined by an analyst/researcher. The use of this method for the buy-sell problem reveals huge potential for investing and/or speculating at the Johannesburg Stock Exchange, the biggest exchange in Africa.

We also looked at the use of non-linear models, specifically ANN models for analysis and forecasting of the South African electricity demand and the connection between the demand and major national economic indicators. We showed that one of the critically important variables in this model is the recession indicator -this is another confirmation of the importance of business cycles in such research.

ANN models are known to take considerable time for parameter estimation. We attempted to overcome this obstacle by suggesting a new approach (also based on the hierarchical principle). We demonstrated the use of this technique by modelling the sinusoidal cycle.

Finally, further into this chapter we suggested future research on the mutual connections between energy production and the climate and weather. Many researchers in this field also believe in the importance of the cycles (e.g. solar activity cycles).

Energy and Climate

On September 25, 2015, the United Nations defined the following seventeen goals of sustainable development: no poverty; zero hunger; good health and well being; quality education; gender equality; clean water and sanitation; affordable and clean energy; decent work and economic growth; industry innovation and infrastructure; reduced inequalities; sustainable cities and communities; responsible consumption and production; climate action; life below water; life on land; peace, justice and strong institutions and partnerships for the goals [123].

This study deals directly with the following two goals:

1. Affordable and clean energy; 2. Climate action.

Indirectly, these two have strong impact on all others.

Access to affordable, reliable, sustainable and modern energy for all

This is the seventh goal of sustainable development. The UN web-site goes: "Be it for jobs, security, climate change, food production or increasing incomes, access to energy for all is essential. Sustainable energy is opportunity -it transforms lives, economies and the planet".

The following facts are highlighted:

• One in five people still lacks access to modern electricity;

• Three billion people rely on wood, coal, charcoal or animal waste for cooking and heating;

• Energy is the dominant contributor to climate change, accounting for around 60 per cent of total global greenhouse gas emissions;

• Reducing the carbon intensity of energy is a key objective in long-term climate goals.

Urgent action to combat climate change and its impacts

The thirteenth goal of sustainable development is motivated as follows: "People are experiencing the significant impacts of climate change, which include changing weather patterns, rising sea level, and more extreme weather events. The greenhouse gas emissions from human activities are driving climate change and continue to rise. They are now at their highest levels in history. Without action, the worlds average surface temperature is projected to rise over the 21 st century and is likely to surpass 3 degrees Celsius this century -with some areas of the world expected to warm even more".

It is important to note that:

• From 1880 to 2012, the average global temperature increased by 0.85 • C;

• Oceans have warmed, the amounts of snow and ice have diminished and the sea level has risen. From 1901 to 2010, the global average sea level rose by 19cm as oceans expanded due to warming and ice melted;

• Given the current concentrations and on-going emissions of greenhouse gases, it is likely that by the end of this century, the increase in global temperature will exceed 1.5 • C compared to the period 1850 to 1900 for all but one scenario. The worlds oceans will warm and melting of ice will continue. The average sea level rise is predicted as 24 -30cm by 2065 and 40 -63cm by 2100;

• Global emissions of carbon dioxide (CO2) have increased by almost 50 percent since 1990.

Goals interrelations

One may see that it is obvious that climate and energy are tightly interlinked. Furthermore, they both are strongly connected to other goals. For example, climate warming leads to reduction in grain (and other) crops. It is estimated that 1 • C increase in the temperature reduces grain crops by 5%. This has immediate impact on goals two (zero hunger) and eight (economic growth).

Another example is the increasing use of renewable energy which is linked to the goal nine (industry innovation and infrastructure), eleven (sustainable cities and communities) and twelve (responsible consumption and production).

The interrelations between these global challenges should be thoroughly studied, analysing them in isolation does not make much sense, as the feedback mechanisms are strong and causality is complex and often bidirectional. As a result, no progress on a goal is achievable without making advances on the others.

The risks (and even dangers) are that certain causality mechanisms are hidden and not well investigated. Certain obvious assumptions may be found irrelevant while others, seemingly inappropriate, may expand to critical proportions. For example, the UN says nothing about the risks that climate change may have on the renewable energy industry. This is one of the issues that we shall discuss further in more detail.

Renewable Energy

Renewable energy (RE) has become synonymous with change mitigation, adaptation and clean energy (e.g see [START_REF]Renewable energy sources in Turkey for climate change mitigation and energy sustainability[END_REF], [START_REF] Iii | Climate change[END_REF], and [START_REF]Making Climate Change Work for Us: European Perspectives on Adaptation and Mitigation Strategies[END_REF]). Article 1 of the United Nations Framework Convention on Climate Change (UNFCCC) defines climate change as a change of climate which is attributed directly or indirectly to human activity that alters the composition of the global atmosphere. This change in the climate occurs in addition to natural climate variability observed over comparable time periods. It also defines emissions as the release of greenhouse gases and/or their precursors into the atmosphere over a specified area and period of time, while greenhouse gases (GHGs) are defined as those gaseous constituents of the atmosphere, both natural and anthropogenic, that absorb and re-emit infrared radiation.

The climate change stakeholders are defined as governments, private sector entities, financial institutions, non-governmental organisations (NGOs) and research and/or education institutions. Interactions and barriers between the stakeholders vary according to sector, type of technology and country and trends in international financial flows that drive technology transfer by altering the relative capacities and roles of different stakeholders.

Renewable Energy is regarded among stakeholders, policy makers and implementers to play an important role in change mitigation, adaptation and clean energy [START_REF]Renewable energy sources in Turkey for climate change mitigation and energy sustainability[END_REF][START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF]. Indeed RE is the key to solving energy-related challenges and it will play an important role in the worlds future [START_REF]Renewable energy sources in Turkey for climate change mitigation and energy sustainability[END_REF].

The official documents put the emphasis on the positive sides of the renewable energy, however we need to clearly understand the problems and challenges associated with the generation of electricity from renewable sources.

• Solar and wind energy (and hydro energy to some extent) are known as Variable Energy

Resource (VER). The variability may be subdivided into two types: deterministic variability (e.g. due to day-night solar cycle) and random variability (e.g. variability in wind generation due to varying wind speeds).

• As a result, the following two issues become critically important:

-Storage: when the generation exceeds the demand, the energy needs to be stored for use in periods of lower generation;

-Alternative source: in case of insufficient generation and limited storage, it is necessary to have a back up source (e.g. grid, diesel generation, or controllable renewable source (e.g. hydro, pump-storage, etc.);

-Forecasting and optimisation: this should involve thorough analysis of generation profiles, weather patterns in the area and demand profiles.

• Transmission: quite often the sites with the best energy generation potential are far from consumers and from existing transmission lines, therefore the cost of transmission should be included in the considerations.

Impact of the Energy Sector on Weather and Climate

The United Nations (UN) and Intergovernmental Panel on Climate Change (IPCC) basic documents define only major factors impacting on the climate change (e.g. greenhouse gases emis-sions). However, the impact of the energy sector is in fact much wider. We can name the following:

• Many traditional power producing technologies (e.g. coal-fired power generation, diesel/natural gas/LPG generation) emit substantial heat.

• The use of coal for power generation results in coal dust pollution of the surroundings of the stations. This changes the reflecting properties of the Earth's surface and has a negative effect on the vegetation. This factor may be regarded as negligible in itself, however, in combination with other factors (increased CO2 content, emission of heat, etc.) it results in visible change of the micro-climate in the vicinity of the power stations. In South Africa, these power stations are relatively close to each other and occupy large areas spanning across provinces (mainly in Mpumalanga and Limpopo). Therefore, the special micro-climate area has considerable dimensions.

• Water use: the traditional energy sector (not only electrical) requires considerable amounts of water (see for example [START_REF] Macknick | Operational water consumption and withdrawal factors for electricity generating technologies: a review of existing literature[END_REF]):

-Electricity Generation: normally power plants need cooling. These types of power plants use steam for generating electricity.

-Fuel Extraction and Production: Water is a critical resource for the drilling and mining of natural gas, coal, oil, and uranium. In many cases, fuel extraction also produces wastewater, as with natural gas and oil wells and coal slurry ponds.

-Fuel Refining and Processing: oil, uranium, and natural gas all require refining before they can be used as fuels a process that uses substantial amounts of water.

-Fuel Transportation: water is used to transport coal through slurries pipelines of finely ground coal mixed with water and to test energy pipelines for leaks [127].

-The Kouberg power station in South Africa uses cold water from the Atlantic Ocean for cooling and returns hot water to the ocean. This creates visible change in the aquatic micro-climate. Other nuclear stations around the world use similar technologies.

• Even electricity generation from renewable sources may have an impact on the weather and climate. Huge wind farms and photo-voltaic solar farms change wind turbulence patterns in the area. Apart from that, the solar farms change the reflecting properties of the surface.

Impact of the Weather and Climate on the Energy Sectors

On the other hand, the weather and climate impact on the energy sector. Here we may mention the following:

• The ambient temperature is one of the major factors driving the demand for electricity and other energy. Low temperatures require considerable use of energy for heating of premises/dwellings. However, the industry also increases the use of heat as many technological processes operate at prescribed temperatures (various refineries, smelters, greenhouses, etc.).

• Higher temperatures also increase the use of the energy, for example for air conditioning of premises/dwellings, and for cold storing/refrigeration. Certain industrial processes also require cooling or keeping constant temperature (e.g. gas liquefaction, many chemical and biochemical processes).

• Temperature variations have different effects on energy demand (for all kinds of energy: electricity, oil products, gas, coal) in different countries and also vary according to the season [START_REF] Cian | The impact of temperature change on energy demand: A dynamic panel analysis[END_REF].

• Higher temperatures make power plants less efficient. Up to 3% of efficiency may be lost due to cooling problems in thermal stations [START_REF] Prevention | Reference document on the application of best available, techniques to industrial cooling systems[END_REF]. Considerable loss in efficiency is also observed in photo-voltaic electricity generation [START_REF] Dubey | Temperature dependent photovoltaic (pv) efficiency and its effect on pv production in the world a review[END_REF].

• Climate change may have an impact on the availability of natural resources needed to generate renewable energy. For instance, changes in weather patterns such as precipitation, humidity, wind speed and cloudiness may result in changes to quantity and timing of energy generated from renewable resources.

• Hydro-power stations depend on the water inflows which hugely depend on the rainfall in the catchment areas. Here not only average inflows are important, but also the variability of the inflows. Significant inflows may result in so-called spills and waste of water.

• The energy contained in the wind is proportional to the cube of wind speed, which means that alterations in the latter can have significant impacts on the former [START_REF] Pryor | Climate change impacts on wind energy: A review[END_REF]. The decadal and multi-decadal variability in wind speed statistics introduces an element of risk into the decision process for siting new wind power generation facilities [START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF]. There is a risk that winds speeds will reduce by 1.0 -3.2% in the next fifty years and by 1.4 -4.5% over the next one hundred years.

• Climate change can increase the probability, recurrence and distribution of natural disasters (heavy precipitation, high temperatures, strong winds, storms, hurricanes, floods and tsunamis), which may contribute to initiating unfavourable geodynamic processes [START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF].

Questions to be Answered

The problem is not whether the potential risks related to climate change and/or global warming will occur, but the ability and readiness to reduce and contain them [START_REF] Richardson | Climate Change: Global Risks, Challenges and Decisions[END_REF]. Some authors (e.g. see [START_REF] Mills | Weighing the risks of climate change mitigation strategies[END_REF]) acknowledge the risks presented by climate change and claim that many of these mitigation strategies suggested are themselves susceptible to climate impacts, as well as to a host of geopolitical factors.

It is therefore imperative to investigate the impact of the energy sector on climate together with the impact of climate change on renewable energy potential.

Political/Legal Constraints and Role of Governments

Another issue which is equally important, both for energy research and climate/weather research, is the need to account for national legislation and understanding the role of governments. We can give some examples of these for France and the RSA:

• In France, all solar PV installations at private dwellings and at industrial sites must be connected to the grid. In South Africa, the government encourages the installation of isolated PV systems, both for private and business use;

• In France, the producers of renewable electricity are paid according to a quite sophisticated tariff system, which accounts for time of day, season, availability of alternative supply, etc. In South Africa, the payments to independent power producers (IPP's) are governed by the so-called REFIT (Renewable Energy Feed-In Tariffs) legislation. Basically, the IPP's who qualify for the REFIT programme are guaranteed to receive a certain amount per kilowatt-hour for a period of 20 years (regardless how much and when the electricity was generated and whether it was needed or not).

While the private sector has the capacity to mobilise the funds required to address pressing energy needs, but the incentives to do so, as well as legislation to create a stable investment climate, need to be provided by governments [START_REF]Technology transfer through climate change: Setting a sustainable energy pattern[END_REF]. This is particularly important in developing countries where the perceived risks of technology investments are generally higher and investments are more difficult to attract.

Tackling the problem of pollution and that of the increasing demand for electricity largely depends on the speed with which we can massively expand the contribution of renewable energy to our overall energy needs. However, the key factor is still the political will displayed by individual governments to encourage potential investors through incentives. The increasing of the share of RE in the energy mix require policies to stimulate the changes in the energy system [START_REF]Renewable energy sources in Turkey for climate change mitigation and energy sustainability[END_REF].

Models in Use

The most popular models for the climate change analysis are so-called, General Circulation Models (also known as global climate models, both abbreviated as GCMs). They are used to represent physical processes in the atmosphere, ocean, cryosphere and land surface, and are the most advanced tools currently available for simulating the response of the global climate system to increasing greenhouse gas concentrations [START_REF] Iii | Climate change[END_REF]. The GCM is defined as a global, threedimensional computer model of the climate system which can be used to simulate human-induced climate change. GCMs are highly complex and they represent the effects of such factors as reflective and absorptive properties of atmospheric water vapour, greenhouse gas concentrations, clouds, annual and daily solar heating, ocean temperatures and ice boundaries. The most recent GCMs include global representations of the atmosphere, oceans, and land surface. Needless to say, these models require the use of powerful (super)computers.

GCMs were first developed in the late 1950s and early 1960s to study the interaction between the atmosphere and the environment and to improve weather forecasting [START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF]. GCMs capture the global-scale climate phenomena well at an approximate resolution of about 100km. However, regional-scale climate phenomena occur at scales that make it difficult, if not impossible, for GCMs to capture [START_REF] Henderson-Sellers | The future of world's climate[END_REF]. Several models, sometimes referred to as regional circulation models (RCMs), have been used to downscale the coarse information provided by the GCMs and provide high resolution information for a particular region of interest [START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF].

There are two main approaches to downscaling GCM results that are currently used i.e. dynamical downscaling and statistical downscaling. Dynamical downscaling involves driving an RCM at a high resolution using the output of the GCM for the region interest. Examples of dynamical downscaling models that are common in literature are the Hadley Centre GCM from the UK Meteorological Office and the Max Planck Institutes European Center Hamburg Models (ECHAM). Statistical downscaling, also referred to as empirical downscaling, involves establishing a mathematical relationship between a large-scale climate field or variable and a local-scale observation [START_REF] Henderson-Sellers | The future of world's climate[END_REF].

Climate models are run for different future scenarios of GHG concentrations. These are derived from story lines developed by IPCC working group III, that explain how changes in technology, demography and socio-economic activity may unfold globally. From the results obtained on the effect of climate change on climatology such as wind patterns, irradiation and precipitation, we were able to deduce the impact on renewable energy.

Although it is generally agreed that GCMs offer the best tools for analysing climate change with respect to GHGs, their output suffers in both resolution and accuracy [START_REF]Vulnerability of wind power resources to climate change in the continental united states[END_REF]. For regional consideration, down-scaling of the GCMs is required which implies an additional work load and increased computational capacity. As with GCMs, as the spatial resolution increases, dynamical down-scaling also requires longer computational time and larger computational capacity and storage. RCMs produce grid-average values, while most climate observations are point-based and measured at particular meteorological stations, making them difficult to compare. Some researchers concur that whereas GCMs often capture global-scale climate phenomena well (with spatial resolution of about 100km), is too coarse to capture regional and local phenomena that are important for the regional climate [START_REF] Henderson-Sellers | The future of world's climate[END_REF].

Lyapunovian Modelling Program

We should now be convinced that the energy sector has an impact on the climate and vise-versa, that the climate impacts the energy sector. While we acknowledge the interaction of both the energy sector and climate with other factors, we will consider a model which includes only vari-ables related to these two components. • CO2 content in the atmosphere;

• ambient temperature;

• heating degree-days;

• cooling degree-days;

• wind speed;

• humidity;

• etc.

For the Energy component, we may include (for instance) the following:

• installed capacity of CO2 emitting power stations;

• installed capacity of clean power stations;

• electricity price;

• crude oil price;

• natural gas price;

• electricity demand;

• etc. Apart from the above, the model may also include derivatives (of any order) of the above variables.

The natural way of studying such a system is to check if the system is stable (in the understanding of Lyapunov [START_REF] Lasalle | Stability by Lyapunov's Second Method with Applications[END_REF]). In case of a stable system, such system will remain close to a certain equilibrium even if small disturbances are present. In case of instability, the trajectory will "shoot" into a certain direction even if the disturbances are relatively small. Mathematically we can write a system either in a form of differential equations or difference (regression/autoregression) equations. In case of differential equations:

x ′ (t) = f(x(t)), x(0) = x 0 (6.1)

where x(t) = (x 1 (t), ...x n (t)) is the vector of the variables, f(.) vector-function which depends on all x i .

The above system will be stable if the Jacobian of the f at point t 1 (present time) is a stable matrix, that is a matrix with eigenvalues which all have only strictly negative real parts.

In case of discrete time, we may write the following model (assuming linearity):

x i = Ax i-1 , x 0 = x0 (6.2)
where A is an n × n square matrix. In this case, the system will be stable if the modulus of all eigenvalues of A is smaller than one.

In case of matrices with random elements, we can calculate the probability that the matrix is stable [START_REF] Girko | Stochastic lyapunov problem for a system of stationary linear differential equations[END_REF].

Consider a simple (in fact oversimplified) example. Suppose x 1 is the percentage of CO2 content in the atmosphere above country X, x 2 is the average monthly temperature (deseasonalised), x 3 is the installed capacity of all power stations which emit CO2 (traditional generation), x 4 is the installed capacity of all country's power stations which do not emit CO2. Assuming linearity of the model, we have:

x ′ i (t) = 4 j=1
a ij x j (t), x(0) = x 0 (6.3)

In this case, the Jacobian is simply A = a ij . Suppose we have:

• a 12 = 0.001, CO2 content is growing proportionally to temperature;

• a 13 = 0.001, CO2 content is growing due to traditional generation;

• a 21 = 0.001, temperature is growing due to CO2;

• a 23 = 0.01, temperature is growing due to traditional generation;

• a 33 = 0.001, traditional generation is growing;

• a 44 = 0.01, clean generation is growing;

• all other a ij =0.

Calculating real parts of the eigenvalues yields: 0.01, -0.001, 0.001, 0.001. Such system is unstable (three eigenvalues have positive real parts).

Another example:

• a 12 = 0.001, CO2 content is growing proportionally to temperature;

• a 13 = 0.05, CO2 content is growing due to traditional generation;

• a 21 = 0.05, temperature is growing due to CO2;

• a 23 = 0.05, temperature is growing due to traditional generation;

• a 31 = -0.1, traditional generation is declining with proportionally to CO2 content;

• a 32 = -0.1, traditional generation is declining with proportionally to temperature;

• a 33 = -0.1, traditional generation is declining;

• a 34 = -0.1, traditional generation is declining with proportionally to clean generation;

• a 41 = 0.01, clean generation is growing proportionally to CO2 content;

• a 42 = 0.01, clean generation is growing proportionally to temperature; generation;

• all other a ij =0.

Calculating the real parts of the eigenvalues yields: -0.03, -0.03, -0.03, -0.005. All real parts are negative and such system is stable. Note that the above example is totally far-fetched, the coefficients have nothing do with reality. We provided this example for illustration purposes only. However, it is possible to estimate (using econometric methods) the coefficients of the model to obtain a fully dynamic system. In other words, we propose to estimate and test the vector autoregressive (VAR) model. This is the perspective of our research.

Conclusions

The clean energy and climate action are not only very important goals defined by the humankind as top priorities in world development, they are also closely interlinked. We suggest that in econometric modelling, these two components of national growth be studied together as a closed-loop feedback system. The Lyapunov stability theory is one of the relatively simple but efficient tools that may be used for such modelling and analysis.

Figure 1 :

 1 Figure 1: Simple three-layer feed-forward ANN model

Figure 1 . 1 :

 11 Figure 1.1: Elliott's 5-3 Cycle

Figure 1 . 2 :

 12 Figure 1.2: Self-similarity of Elliott Wave

Figure 1 . 4 :

 14 Figure 1.4: Actual Demand (blue) and Forecast by a Weather-neutral Model (source [124])

Figure 1 . 5 :

 15 Figure 1.5: Daily Demand: The Model (source [108])

  .6).

Figure 1 . 6 :

 16 Figure 1.6: Daily Demand: Comparing the Actual (red) and the Model (green) (source [108])

Figure 2 . 1 :

 21 Figure 2.1: Two simulations of regression model (200 observations, a = 2, b = 0.2, s 1 = 1)

Figure 2 . 2 :

 22 Figure 2.2: Two simulations of auto-regression model (200 observations, a = 2, b = 0.2, s 2 = 1)

Figure 2 . 3 :

 23 Figure 2.3: MLS estimation for AR model (200 observations, a = 2, b = 0.2, s 2 = 1); in greenobserved data, blue -expected value of the true model, red -estimated response.

Figure 2 . 4 : 4 Figure 2 . 5 :

 24425 Figure 2.4: Cycles for LR model (100 observations, a = 0.8, b = 0.01, σ = 0.1); blue dotsobserved data, brown line -BB-cycles, k = 4

Figure 2 . 6 :

 26 Figure 2.6: Relationship between ρ and p

Figure 2 . 7 :

 27 Figure 2.7: Variance of the estimates for p = 0.5 as function of w 1

Figure 2 . 8 :

 28 Figure 2.8: Variance of the estimate for p = 0.7 as a function of w 1

1 . 5 .

 15 Optimal trading (OT) objective function; 2. Two-point full evaluation (FE2) optimisation method (to use in the steps 1 and 5); 3. Total variation (TV) less total change (TC) selection method (to use in step 3); 4. Total variation (TV) over total change (TC) selection method (possible option to use in step 3); Profit less charges (PLC) stopping criteria.

  t=1,...N (e 1 (t) -e 1 (0)) -min t=1,...N (e 1 (t) -e 1 (0)) = max t=1,...N (e 1 (t)) -min t=1,...N (e 1 (t)) = e

Figure 4 . 1 :

 41 Figure 4.1: Fitting LR model (Normal errors) to power curve

Figure 4 . 2 :

 42 Figure 4.2: Fitting LR model (Laplace errors) to power curve

Figure 4 . 3 :

 43 Figure 4.3: Relationship between H and E( l)

Figure 4 . 4 :

 44 Figure 4.4: Fit residuals for model (4.76).

Figure 4 . 5 :

 45 Figure 4.5: Relationship between l and E( Ĥ) (dotted image)

Figure 5 . 1 :

 51 Figure 5.1: Simple three-layer feed-forward ANN model

Figure 5 . 2 :

 52 Figure 5.2: Profile of the likelihood function

Figure 5 . 4 :

 54 Figure 5.4: Simulated Series

Figure 5 . 5 :Figure 5 . 7 :

 5557 Figure 5.5: Simulated series and y = ȳ (in red)

Figure 5 . 8 :

 58 Figure 5.8: Sin(x) and the fitted curve after the first step of hierarchical estimation

Figure 5 . 9 :

 59 Figure 5.9: Sin(x) and the fitted curve after the second step of hierarchical estimation

Figure 5 .

 5 Figure 5.10: Sin(x) and the fitted curve after the third step of hierarchical estimation

Figure 5 .

 5 Figure 5.11: Sin(x) and the fitted curve after the additional step of hierarchical estimation

Figure 6 . 1 :

 61 Figure 6.1: Closed-loop model: Energy-Climate

  2.1Table A: Statistical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Table A: Statistical Models (continued) . . . . . . . . . . . . . . . . . . . . . . . 2.3 Table B: Behavioural Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Table C: Reference Class Models . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Table D: Combined Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Random examples of output (codes were used instead of actual share names due to ethical considerations) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1 Results of the Estimation: Final Linear Model . . . . . . . . . . . . . . . . . . . 5.2 Results of the Estimation: Nonlinear Part . . . . . . . . . . . . . . . . . . . . . 5.3 Out-of-sample Forecasts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Some evaluation statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Table 2 .

 2

1: Table A: Statistical Models methods for energy forecasting. The models synthesised from various methods are presented in this study as the Hybrid (or Combined) models, Table 2.5.

Table 2 .

 2 2: Table A: Statistical Models (continued)

	Model(s)/Method(s)	Purpose H	Variables	Country	Year Ref
	Hybrid models
	Integration of Neural Network,	EnMF	s	Electricity consumption series data	Iran	2008. [16]
	time series and ANOVA
	ANN technique, linear and	ElCF	l	National income, GDP, CPI, population	Taiwan	2006. [130]
	non-linear statistical models
	PSO and GA optimal Energy Demand Estimating		l	GDP, population, economic structure, urbanization rate, & EnC structure
		EnDF			China	2012. [173]
	SEGARCH-ANN, WARCH-ANN	EnCF	l		Taiwan	2009. [132]
	Hybrid nonlinear models Fuzzy inference system-stochastic EnCF	l	GDP, population & gas consumption	Bahrain, Syria 2011. [16]
	frontier analysis, FIS
	ARIMA-GARCH methods	ElPF	s		China	2010. [161]
	Prospective outlook on long-term energy systems, POLES EnMA	l	Time series, socio-economic data, technological data s		2010. [26]
				& survey/estimates for various parameter		2006. [167]
	World Energy Modelling, WEM	EnMA m,l		IEA	2011. [80]

Table 2 .

 2 5: Table D: Combined Models • Very long-term: up to 50 years.

Table 3 .

 3 Some examples of the output are given in the table below.

	Share # series length average price(cents) number of cycles OF (cents) estimated return
	100	2500	7907.68	172	61846	782%
	118	2500	2264.02	226	25088	1108%
	136	2500	236.88	412	5043	2129%

1: Random examples of output (codes were used instead of actual share names due to ethical considerations)

Table 5 . 2 :

 52 Results of the Estimation: Nonlinear Part5.4.4 ForecastsThe out-of-sample forecasts are presented in the following table 5.3 for a horizon of six months.

	Horizon Actual values Linear model ANN-model
	2010-01	20124.4	18900.0	18748.4
	2010-02	18861.7	18124.1	17972.6
	2010-03	20914.6	17818.5	18590.3
	2010-04	19844.7	17782.6	18187.1
	2010-05	21149.2	17741.2	19675.9
	2010-06	21352.6	18284.8	19660.4

Table 5 . 3

 53

: Out-of-sample Forecasts

  Theils U takes a value of one with the naïve model. Values below 1 indicate an improvement comparatively to this naïve benchmark and values higher than the unity reflect a deterioration

		Linear model ANN-model
	RMSE	2480.2	1605.9
	MAPE	10.945	7.526
	Theils U	1.97	1.24
	Table 5.4: Some evaluation statistics
	of the forecasts, always with respect to this benchmark.

Mots-clès: Cycle économique, Cycle économique, Cycle financier, Persistance, Cycle-datation, Hurst Exponent, Énergie, Climat, Méthodes hiérarchiques, Réseaux de neurones artificiels.

Acknowledgements

Table 2.4: Table C: Reference Class Models is made. While lengths of a horizon and grain are positively correlated, they have the opposite effect on the forecast accuracy. At the same time concept "horizon" used in the study is interpreted here according to the official classification for energy demand (that is, excluding load forecasting):

• Short-term: up to 18 months (from hour ahead to 18 month ahead);

• Medium-term: up to 7 years;

• Long-term: up to 12 years;

Estimation of the Linear Model

The search for an adequate model applies the general-to-specific (Gets) methodology vindicated by David Hendry (see for a recent and path-breaking reference, [START_REF] Hendry | Empirical Model Discovery and Theory Evaluation, Automatic Selection Methods in Econometrics[END_REF], especially chapter 1). Following this approach, we start the process of discovery with a general unconstrained model and after some reduction operations, we have a dynamic, parsimonious and well-specified model. Some comments are due at this point. Firstly, the variable CPI is not included among the regressors as it has no statistical effect on the electricity production. Secondly, the variable "Rec" is peculiarly significant. Logically, the entry in recession immediately lowers the electricity output, but after one month, and most notably six months, the relationship becomes positive. Thirdly, the manufactured production contributes positively, but with one lag, to the electricity production.

The Enlarged ANN-model

Given equation 5.6, it remains to estimate the part

where β = 1, G is the logistic function and z t is the vector compound by Elec(-2), Elec(-4), Rec, Rec(-1), Rec(-6) and Prod(-1).

The estimation process is conducted along the lines defined in subsection 5.3.1. However, the likelihood function is peculiarly difficult to maximise. The following graph illustrates likelihoods profile in the last step of optimisation algorithm. Table 5.2 shows the ML estimators of the gamma parameters.

As a general rule, the estimated coefficients must not be interpreted in the same way as we do for a linear model. They are only used to get a better prediction of the electricity consumption. and we would like to keep this property in the coming residuals. The only linear function which will satisfy our intentions is: a 1 (x -0.5). So, at this second step of the hierarchical method we shall fit the following model:

We may suggest a variety of simple methods to fit this simplistic model (remember we promised not to use the OLS!). Of all these methods we shall choose one which is the most similar to the method we used in the first step. We shall divide ǫ 1 (x) by (x -0.5) and find the average! There is only one minor modification: we exclude from the calculations the middle part of the series (if x = 0.5 we cannot divide by zero, or if x ≈ 0.5 we prefer not to divide by values close to zero, as this will result in very high variation in the estimate). In our example, we included in the calculations first 250 observations and the last 250 observations. And our estimate for a 1 is: 0.956879. Combining this result with the previous step we have the following approximation (see figure 5 Now, we shall model the new residuals ǫ 2 (x) = y(x) -(-0.47844 + 0.956879x) with a second order polynomial. However, we shall use the type of the polynomial which will preserve the two good properties of the residuals ǫ 2 (x) -their sum is zero and there is no trend. It may be quite easily shown that the only second order polynomial that satisfies our requirements is:

Again, dividing ǫ 2 (x) by 1 6 a 2 (1-6x + 6x 2 ) and calculating the average we get a 3 = -2.10581.

Combining all three steps together we get:

And the final fit may be seen in figure 5.7. By the way, the OLS answer to the same data is: 0.996769 + 3.0449x -2.08802x 2 . We also show it in figure 5.7 (green line), but it is almost totally covered by the red line.

Chapter 6

General Conclusions and Future Research

Introduction

In this section, we present general conclusions and also suggest future research in the area of energy and climate interrelations.

General Conclusions

In this thesis, we presented our research on cycles in the South African economy. A national economy is often compared to a living organism. In this comparison, business represents the muscles; the financial markets are compared to the nervous system and the energy represents the blood. All the organs work together and cannot exist individually.

The wellbeing of these three essential components of the economy is critically important for growth and development. However, constant growth is impossible, there are periods of decline and shrinking. Do these periods of recession represent any danger or are they as natural and unavoidable as other critical changes and processes? This is the main question which we tried to answer in our research.

Our research shows beyond any doubt that cycles are naturally present in any real process as any real process is subjected to random disturbances. Even processes that may be described by linear models have cycles (see section 2.12). Therefore, any practical or theoretical research should account for the presence of cycles or such research will be incomplete.

What tools mya be used to study cycles? Standard models that are normally used for cyclical series are normally aiming at cycles with fixed periods (e.g. Seasonal ARIMA, Fourier series, etc). However, in economic applications, one often deals with irregular cycles; that is cycles