Closed-loop control with delayed information
Résumé
The theory of Markov control model with perfect state information (MCM-PSI) requires that the current state of the system is known to the decision maker at decision instants. Otherwise one speaks of Markov control model with imperfect state information (MCM-ISI). In this article we introduce a new class of MCM-ISI, where the information on the state of the system is delayed. Such an information structure is encountered for instance in high-speed data networks. In the first part of this article, we show that by enlarging the state space so as to include the lask known state as well as all the decisions made during the travel time of the information, we may reduce a MCM-ISI to a MCM-PSI. In the second part of this paper, this result is applied to a flow control problem. Considered is a discrete time queueing model with Bernoulli arrivals and geometric services, where the intensity of the arrival stream is controlled. At the beginning of slot t + 1, t = 0, 1, 2, ..., the decision marker has to select the propability of having one arrival in the current time slot from the set {p1, p2}, 0 p2 < p1 1, only on the basis of the queue-length and action histories in [0,t]. The aim is to optimize a discounted throughput/delay criterion. We show that there exists an optimal policy of a threshold type, where the threshold is seen to depend on the last action.