Uktzin, Chapter 2, Mishnah 4. This chapter is occupied with the laws of yados and shomrim: parts of a food that are not eaten themselves, yet are drawn after the food because they serve it. A yad is a handle by which the food is held, and a shomer is a part that guards and protects the food. Since such parts serve the food, they are treated as part of it: they become tamei together with it, they transmit tumah, and they are counted in when we measure whether there is a k'beitzah of food, which is the shiur required for food to be metamei something else. Our Mishnah states the rule for peels.
"Kol haklipin mitam'os u'metam'os u'mitztarfos" - all peels become tamei and convey tumah, and they combine with the food. Each of the three consequences follows from the same reason. They serve as a yad and as a shomer for the produce they enclose: they are held when the fruit is taken, and above all they shield the fruit and keep it from spoiling. A part that protects the food is not viewed as something separate lying alongside the food; it takes on the very din of the food. Therefore it is susceptible to tumah as the fruit is, it passes tumah on as the fruit does, and when we come to measure a k'beitzah in order to know whether this food can render something else tamei, the peel is counted together with the fruit.
This is the broad principle: a peel is not waste to be disregarded, but part of the food's own reckoning, precisely because it stands guard over what is inside.