Bechorot, chapter 1, Mishnah 3. In this Mishnah, we begin discussing situations of doubt - cases where there is no certainty whether the animal before us is obligated in the laws of peter chamor.
Two separate matters stand to be clarified:
The prohibition of benefit: If the animal is indeed a peter chamor - a firstborn male donkey - it may not be used until it is redeemed. This is a prohibition of benefit, a matter between the owner and the Creator, in the realm of prohibition and permission.
The financial obligation to the kohen: There is a separate obligation to give the kohen the animal used to redeem that peter chamor. Usually, this is a sheep, goat, or lamb; the Mishnah uses the term "taleh", meaning an offspring, a young lamb, which is the cheaper version of a sheep, and it is likely that this would be the practical choice.
From this, in a situation of doubt where the kohen cannot prove that he is owed anything, we apply the rule of "hamotzi mechavero alav haraayah" - the burden of proof rests on the claimant. Therefore, if the kohen cannot prove that this donkey is obligated in the redemption of a peter chamor, he is not entitled to claim the redemption sheep. Nevertheless, the owner is obligated to redeem the male donkey that is a doubtful firstborn, because until he does so, he is not allowed to derive benefit from it and use it.
One donkey that had not previously given birth:
Two males: "Chamor shelo bichrah veyaldah shenei zecharim" - a donkey for which this is her first pregnancy and she has never given birth. The owner left her at night when she was pregnant, and in the morning he found two male foals in the enclosure. He does not know which of them is the firstborn, but it is certain that one of them is a firstborn and requires redemption, and the lamb upon which it is redeemed belongs to the kohen. Therefore, the Mishnah says: "Noten taleh echad lakohen" - the kohen receives only one lamb. The owner says: "I do not know which of the two donkeys is the firstborn, but whichever one it is - it is redeemed upon this lamb", gives the lamb to the kohen, and from now on is allowed to derive benefit from the two males.
A male and a female: In the morning, two offspring were found in the pen, a male foal and a female foal. It is possible that the male was born first, and then it requires redemption and the sheep upon which it is redeemed belongs to the kohen; and it is possible that the female was born first, and then there is no obligation of redemption at all. From the owner's perspective, there is a possibility that the male was born first, and therefore he must redeem it out of doubt, since he is not allowed to use the donkey until he redeems it. On the other hand, the kohen cannot prove that the male was born first, and therefore he is unable to extract anything. The Mishnah says: "Mafrish taleh echad leatzmo" - he designates one sheep for the redemption of the male donkey if it is necessary, and keeps the sheep for himself without any restriction: he is allowed to eat it, shear it, and do whatever he desires. Both the donkey and the sheep are completely permitted to him.
Two donkeys that had not previously given birth:
Two males: At night there were two donkeys pregnant with their first pregnancy, and in the morning two newborn male foals were found in the enclosure. It is therefore clear that both are firstborns, and thus the owners are obligated in two lambs to the kohen. Although it is not known which of the donkeys belongs to which of the mothers, there is no practical difference. The owner takes two lambs, says: "The two firstborn donkeys are redeemed with these lambs", and gives both of them to the kohen.
A male and a female: In the morning two offspring were found, a male and a female. The male is certainly a firstborn, except that it is not known from which mother it was born, and therefore the owners are obligated in only one redemption and one lamb to the kohen.
Two males and a female: Even if two males and a female offspring were found, it is certain that one of the donkeys gave birth to a male, for if one gave birth to a female - the other gave birth to a male. It turns out that one of the two males is certainly a firstborn, and it is possible that both are firstborns. How should he act? He should say: "If only one of them is a firstborn, the certain one is redeemed upon this sheep, and I am giving the sheep to the kohen; and if both are firstborns, I will redeem the first upon this sheep and the second upon an additional sheep". The first sheep, which certainly belongs to the kohen, is given to the kohen; while the second, about which there is no certainty, remains in the hands of the owner. Therefore, the Mishnah says: "Noten taleh echad lakohen" - for one certainly is owed to him; while the second, since the kohen cannot prove that it is his, remains with the owner by the law of "hamotzi mechavero alav haraayah". And from now on, both the males and the female are permitted without any restriction.
Two females and a male, or two females and two males: At night there were two donkeys pregnant with their first pregnancy, and in the morning two females and one male foal were found, or two females and two male foals. In this case, it is possible that both first-time donkeys gave birth to a female, and if so, there is no obligation whatsoever toward the kohen, and by the law of "hamotzi mechavero alav haraayah" we do not give him anything. On the other hand, it is possible that the males were firstborns, and therefore - whatever the number of males may be - they must be redeemed upon lambs, and the lambs remain in the hands of the owner. And this is what is stated in the Mishnah: "Ein kan lakohen klum" - since the kohen cannot prove that he is owed anything, for it is possible that the two females are the firstborns, he receives nothing.
In summary: In this Mishnah we learned the method of deciding in cases of doubt regarding a peter chamor, by distinguishing between the two matters: the obligation of redemption imposed on the owner due to the prohibition of benefit, which exists even in doubt, and the financial right of the kohen, which only exists in certainty, by the law of "hamotzi mechavero alav haraayah". Based on this foundation, it is determined in each and every case how many lambs are given to the kohen and how many remain in the hands of the owner.