Chapter 8 of Maseches Berachos, Mishnah 6. The Mishnah deals with the candle and the spices of Havdalah, lists three cases in which we do not recite a berachah over them, and finally establishes the measure required for the berachah over the candle.
A candle and spices "shel goyim" - belonging to non-Jews:
The Mishnah rules that we do not recite a berachah over the candle or the spices of non-Jews. The Gemara explains two separate reasons:
The candle - because this candle "did not rest," that is, it was lit on Shabbos. For the berachah of Havdalah one must use a candle with which no melachah was performed on Shabbos. True, a non-Jew is not commanded regarding the prohibition of melachah, but nonetheless, since work was done with this candle, it may not be used for the berachah.
The spices - the concern is that the festive meal of the non-Jews was a meal of avodah zarah, and these spices may be forbidden to derive benefit from. Therefore we do not recite a berachah over them.
It is worth noting that the Rambam explains the disqualification of the candle of non-Jews with this very same reason, even though his wording differs somewhat from that of the Gemara: with the candle as well there is a concern that it was used for avodah zarah. This very matter is what the Mishnah goes on to spell out in its continuation.
A candle and spices "shel meisim" - of the dead:
The Mishnah continues and rules that we do not recite a berachah over a candle or spices connected with a deceased person. It was the custom to light a candle in honor of the deceased, and this is still done today. Such a candle does not stand there to provide light, as the wording of the berachah "borei me'orei ha'eish" indicates, but is placed there for honor alone, and therefore it is not fitting to use it for the berachah. So too the spices of the deceased are not intended for smelling, but to drive away the foul odor rising from the body.
A candle and spices "shelifnei avodah zarah" - that are before an idol:
Here the matter is even clearer: this is definitely avodah zarah, and it is forbidden to derive benefit from avodah zarah. Since one has no permission to derive any benefit from it at all, it certainly may not be used for a berachah.
The measure for the berachah over the candle:
The Mishnah concludes by ruling that we do not recite a berachah over the candle "ad sheye'otu le'oro" - until a person can make use of its light, meaning that he must be close enough to it. The Gemara gives a measure for this: the ability to distinguish between the coin of one country and the coin of another. Although the measure that emerges from the Mishnah is mere proximity, our custom is to actually make use of the light.
The custom is to look at one's fingernails by the light of the candle, and several reasons have been given for this:
Rabbeinu Yonah: this is a convenient use, since a person's fingernails are right there on his hands.
And some say: fingernails are a sign of berachah, because they are always growing, and growth is considered a sign of blessing.
The Rosh: distinguishing between the nail and the flesh is the same measure of discernment that the Gemara established regarding distinguishing between the two coins.
In summary: in this Mishnah we learned three cases in which we do not recite a berachah over a candle or spices - those of non-Jews (because the candle did not rest, and out of concern that the spices belonged to avodah zarah), those of the dead (which do not stand for light and fragrance but for honor and for driving away a foul odor), and those before an idol (from which benefit is forbidden). We also examined the measure for the berachah over the candle - distinguishing between one coin and another - and our custom to actually make use of the light by looking at our fingernails, along with the reasons for it.