# Stochastic processes with expected stopping time

Chatterjee K, Doyen L. 2021. Stochastic processes with expected stopping time. Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science. LICS: Symposium on Logic in Computer Science, 1–13.

Download (ext.)

*Conference Paper*|

*Published*|

*English*

**Scopus indexed**

Author

Chatterjee, Krishnendu

^{IST Austria}^{}; Doyen, LaurentDepartment

Abstract

Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical optimization criterion is the maximal expected total reward where the MDP stops after T steps, which can be computed by a simple dynamic programming algorithm. We consider a natural generalization of the problem where the stopping times can be chosen according to a probability distribution, such that the expected stopping time is T, to optimize the expected total reward. Quite surprisingly we establish inter-reducibility of the expected stopping-time problem for Markov chains with the Positivity problem (which is related to the well-known Skolem problem), for which establishing either decidability or undecidability would be a major breakthrough. Given the hardness of the exact problem, we consider the approximate version of the problem: we show that it can be solved in exponential time for Markov chains and in exponential space for MDPs.

Keywords

Publishing Year

Date Published

2021-07-07

Proceedings Title

Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science

Acknowledgement

We are grateful to the anonymous reviewers of LICS 2021 and of a previous version of this paper for insightful comments that helped improving the presentation. This research was partially supported by the grant ERC CoG 863818 (ForM-SMArt).

Page

1-13

Conference

LICS: Symposium on Logic in Computer Science

Conference Location

Rome, Italy

Conference Date

2021-06-29 – 2021-07-02

ISBN

ISSN

IST-REx-ID

### Cite this

Chatterjee K, Doyen L. Stochastic processes with expected stopping time. In:

*Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science*. Institute of Electrical and Electronics Engineers; 2021:1-13. doi:10.1109/LICS52264.2021.9470595Chatterjee, K., & Doyen, L. (2021). Stochastic processes with expected stopping time. In

*Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science*(pp. 1–13). Rome, Italy: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/LICS52264.2021.9470595Chatterjee, Krishnendu, and Laurent Doyen. “Stochastic Processes with Expected Stopping Time.” In

*Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science*, 1–13. Institute of Electrical and Electronics Engineers, 2021. https://doi.org/10.1109/LICS52264.2021.9470595.K. Chatterjee and L. Doyen, “Stochastic processes with expected stopping time,” in

*Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science*, Rome, Italy, 2021, pp. 1–13.Chatterjee, Krishnendu, and Laurent Doyen. “Stochastic Processes with Expected Stopping Time.”

*Proceedings of the 36th Annual ACM/IEEE Symposium on Logic in Computer Science*, Institute of Electrical and Electronics Engineers, 2021, pp. 1–13, doi:10.1109/LICS52264.2021.9470595.**All files available under the following license(s):**

**Copyright Statement:**

**This Item is protected by copyright and/or related rights.**[...]

**Link(s) to Main File(s)**

Access Level

Open Access

### Export

Marked PublicationsOpen Data IST Research Explorer

### Sources

arXiv 2104.07278