Abstract : To approach the potential multiple-input multiple-output (MIMO) capacity while optimizing the system bit-error rate (BER) performance, the joint transmit and receive minimum mean squared error (joint Tx/Rx MMSE) design has been proposed. It is the optimal linear scheme for spatial multiplexing MIMO systems, assuming a fixed number of spatial streams p as well as fixed modulation and coding across these spatial streams. However, the number of spatial streams has been arbitrarily chosen and fixed, which may lead to an inefficient power allocation strategy and a poor BER performance. In this paper, we relax the constraint of fixed number of streams p and optimize this value for the current channel realization, under the constraints of fixed average total transmit power P/sub T/ and fixed rate R, what we refer to as mode selection . Based on the observation of the existence of a dominant optimal number of streams value for the considered Rayleigh flat-fading MIMO channel model, we further propose an "average" mode selection that avoids the per-channel adaptation through using the latter dominant value for all channel realizations. Finally, we exhibit the significant BER improvement provided by our mode selection over the conventional joint Tx/Rx MMSE design. Such significant improvement is due to the better exploitation of the MIMO spatial diversity and the more efficient power allocation enabled by our mode selection.