Abstract –In this paper, different amplitude and phase modulation formats are investigated and compared to find the optimum one among them for high spectral efficiency (SE) Wavelength Division Multiplexing (WDM) systems with an acceptable value of the Bit Error Rate (BER). Also, the effectiveness of using the polarization interleaving scheme in minimizing the nonlinear effects in the optical fiber channel for the various modulation formats is investigated. (Optisystem v 7.0) A simulation package is used to simulate the system. The simulation results show the effectiveness of using the polarization interleaving in decreasing the nonlinear effects. RZ modulation formats are found to give a better BER value than the corresponding NRZ formats but with a smaller value of the SE. Among the compared modulation formats, RZ-AMI and the RZ-DQPSK are the only formats that are found to give the optimal performance for high capacity WDM networks.