A new method is presented for calculating the cutoff wavelegth of a waveguide with arbitrary shaped cross-section. The potential functions of a regular waveguide such as rectangular waveguide act as the associate potential functions

thus a integral equation（f. E.） with respect to the potential of the original waveguide is deduced in terms of Green’s 2nd identity. A system of linear algebric equations can be obtained by the discretion of the I. E. and the cutoff wavelengths are then calculated. Numerical results for a rectangular-to-circular waveguide are in good agreement with those by the method of lines.