Nicolas Privault 
Understanding Markov Chains [PDF ebook] 
Examples and Applications

This book provides an undergraduate introduction to discrete and continuous-time Markov chains and their applications. A large focus is placed on the first step analysis technique and its applications to average hitting times and ruin probabilities. Classical topics such as recurrence and transience, stationary and limiting distributions, as well as branching processes, are also covered. Two major examples (gambling processes and random walks) are treated in detail from the beginning, before the general theory itself is presented in the subsequent chapters. An introduction to discrete-time martingales and their relation to ruin probabilities and mean exit times is also provided, and the book includes a chapter on spatial Poisson processes with some recent results on moment identities and deviation inequalities for Poisson stochastic integrals. The concepts presented are illustrated by examples and by 72 exercises and their complete solutions.

Cuprins

Introduction1 Probability Background1.1 Probability Spaces and Events1.2 Probability Measures1.3 Conditional Probabilities and Independence1.4 Random Variables1.5 Probability Distributions1.6 Expectation of a Random Variable1.7 Conditional Expectation1.8 Moment and Probability Generating Functions Exercises2 Gambling Problems2.1 Constrained Random Walk2.2 Ruin Probabilities2.3 Mean Game Duration Exercises3 Random Walk3.1 Unrestricted Random Walk3.2 Mean and Variance3.3 Distribution3.4 First Return to Zero Exercises4 Discrete-Time Markov Chains4.1 Markov Property4.2 Transition matrix4.3 Examples of Markov Chains4.4 Higher Order Transition Probabilities4.5 The Two-State Discrete-Time Markov Chain Exercises5 First Step Analysis5.1 Hitting Probabilities5.2 Mean Hitting and Absorption Times5.3 First Return Times5.4 Number of Returns Exercises6 Classication of States6.1 Communicating States6.2 Recurrent States6.3 Transient States6.4 Positive and Null Recurrence6.5 Periodicity and Aperiodicity Exercises7 Long-Run Behavior of Markov Chains7.1 Limiting Distributions7.2 Stationary Distributions7.3 Markov Chain Monte Carlo Exercises8 Branching Processes8.1 Defnition and Examples8.2 Probability Generating Functions8.3 Extinction Probabilities Exercises9 Continuous-Time Markov Chains9.1 The Poisson Process9.2 Continuous-Time Chains9.3 Transition Semigroup9.4 Infinitesimal Generator9.5 The Two-State Continuous-Time Markov Chain9.6 Limiting and Stationary Distributions9.7 The Discrete-Time Embedded Chain9.8 Mean Absorption Time and Probabilities Exercises10 Discrete-Time Martingales10.1 Filtrations and Conditional Expectations10.2 Martingales – Definition and Properties10.3 Ruin Probabilities10.4 Mean Game Duration Exercises11 Spatial Poisson Processes11.1 Spatial Poisson (1781-1840) Processes11.2 Poisson Stochastic Integrals11.3 Transformations of Poisson Measures11.4 Moments of Poisson Stochastic Integrals11.5 Deviation Inequalities Exercises12 Reliability Theory12.1 Survival Probabilities12.2 Poisson Process with Time-Dependent Intensity12.3 Mean Time to Failure Exercises Some Useful Identities Solutions to the Exercises References Index

Despre autor

The author is an associate professor from the Nanyang Technological University (NTU) and is well-established in the field of stochastic processes and a highly respected probabilist. He has authored the book, Stochastic Analysis in Discrete and Continuous Settings: With Normal Martingales, Lecture Notes in Mathematics, Springer, 2009 and was a co-editor for the book, Stochastic Analysis with Financial Applications, Progress in Probability, Vol. 65, Springer Basel, 2011. Aside from these two Springer titles, he has authored several others. He is currently teaching the course M27004-Probability Theory and Stochastic Processes at NTU. The manuscript has been developed over the years from his courses on Stochastic Processes.