The Torah in Parashat Shemini (Vayikra 10:6) commands the Kohanim, “Bigdechem Lo Tifromu” – not to tear their garments before entering the Bet Ha’mikdash to perform the Aboda (service). This command forbids the Kohanim from performing the service while wearing torn garments. The definition of a “torn garment” for the purpose of this Halacha is the same as the definition of the requirement for a mourner to tear his garment – namely, a rip of at least a Tefah (handbreadth).
This prohibition applies differently to regular Kohanim and to the Kohen Gadol. The Kohen Gadol may not ever tear his clothing – even when he is not performing the Aboda, and even to mourn the passing of an immediate family member. For other Kohanim, this command forbids entering the Bet Ha’mikdash with a torn garment. If a Kohen performed the Aboda in the Bet Ha’mikdash with a torn garment, he is liable to Mita Bi’ydeh Shamayim (death at the hands of G-d, as opposed to execution by the court). If he did not perform the Aboda with torn clothing, but he entered the area in between the outdoor altar and the Ulam (entrance to the Bet Ha’mikdash) with a torn garment, then he has still transgressed a Biblical prohibition, but is liable to Malkut, and not death.
This is the opinion of the Rambam and Sefer Ha’hinuch. The Ramban disagrees, and maintains that simply entering the area of the Bet Ha’mikdash wearing torn garments, without performing the Aboda, is forbidden only Mi’de’rabbanan – by force of Rabbinic enactment. In his view, the Torah prohibition forbids only performing the Aboda with torn garments, and not entering the area of the Bet Ha’mikdash with such clothing.
If a Kohen performed the Aboda while wearing torn clothing, then even though he has violated the Torah command, the service he performed is not invalidated.
The reasoning behind this Misva, of course, is to maintain an aura of majesty and grandeur in the Bet Ha’mikdash. The Bet Ha’mikdash is to be a place which inspires and uplifts people, motivating them to submit to G-d’s authority and obey His will. To that end, the Kohanim are required to be dressed in a dignified manner, as appearing in torn clothing would undermine the aura of majesty, and is thus forbidden. Furthermore, wearing torn clothing is a sign of mourning, and the Bet Ha’mikdash is to instill joy and enthusiasm for the service of G-d. Hence, it would be inappropriate to wear torn clothing – an expression of grief – in the Bet Ha’mikdash.