This paper considers linear transceiver design for amplify-and-forward multiple-input multiple-output (MIMO) multiple-relay systems

aiming to minimize the mean squared error (MSE) under the total relay transmit power constraint.We first introduce the singular value decomposition (SVD) of matrix product and simplify the original optimization problem to one which takes a singular value vector and a unitary matrix as optimization variables.By employing variable substitution and penalty term

the simplified problem only has one unitary matrix constraint.Then we develop a new Riemann Euclidean steepest descent algorithm to deal with the unitary matrix constraint.Finally

simulation results demonstrate that the performance of the proposed scheme is better than traditional schemes and closes to the lower bound of MSE.