A teaching-learning-based optimization algorithm with rectangle neighborhood structure (RNTLBO) is proposed to overcome the shortcomings of low global search precision and premature convergence of the original teaching-learning-based optimization algorithm (TLBO) while handling complex multimodal functions. In the algorithm

the population space is designed as a rectangular structure

and the individual rectangular neighborhood is determined by the rectangle thickness and the individual rectangular region surrounding it. In both teaching and learning stages

the optimal individual in the neighborhood is used to guide the search

which strengthens the ability of the algorithm to explore new solutions and exploit local optimal solutions.In order to prevent the algorithm from falling into the local optimum prematurely

the individual perturbation stage guided by search boundary information is added

so that the population can maintain good diversity even in the later evolution stage. The simulation results of complex functions with shift and rotation show that the new algorithm is superior to the original TLBO and some other recently improved variants in terms of accuracy and stability in most cases.