Shevuos, Chapter 4, Mishnah 6. In order to understand the content of this Mishnah, we first need to introduce the principle of "modeh bekenas patur" - one who admits to a fine is exempt.
The rule of "modeh bekenas patur":
When the Torah establishes a penalty for a particular act, and the litigant comes to court and admits that he is guilty of that act, then since this payment is only a fine, that is, purely a penalty, we learn from various places in the Torah that the moment he admits it he is exempt from the fine. This exemption remains in force even if witnesses later come and testify about the act, since his admission preceded their arrival.
Because of this rule, if a person told witnesses to come and testify for him about something that is a fine, they do not become liable for the oath of testimony. For the defendant is in a situation where, had he admitted, he would have been exempt, so the very fact that he does not admit is a contributing factor in his liability. The witnesses alone, then, are not what obligates the money and brings about payment of the fine; rather, the defendant's failure to admit the transgression is one of the causes of the penalty. For this reason, in a case of a pure fine, the witnesses are not liable.
Mixed claims - fine and money:
Our Mishnah deals with cases that are a mixture, where the claim being made includes both a fine and money, that is, actual monetary damage that is collected and not merely a penalty. The language of the Mishnah: "mashbia ani aleichem" - a person tells potential witnesses to swear to him that if they do not come and testify for him, they will be liable.
"she'yesh li beyad peloni nezek" - a claim for damages, that so-and-so damaged him and owes him money. This is not a fine and not a penalty, but a monetary debt for a loss that he caused him.
"vachatzi nezek" - half-damages is generally considered a fine, and accordingly the witnesses would be exempt. However, the Gemara explains that the case is one of half-damages for tzeroros: an animal was walking, and pebbles flew out from under its feet and broke vessels. This half-damage payment is considered money, and we know as a halachah transmitted to Moshe at Sinai that it is a monetary obligation and not a fine. Therefore the witnesses are liable, since the absence of their testimony is what causes the claimant's loss.
"o tashlumei kefel" - one who claims that his fellow owes him double payment because he stole an object from him. True, the second half of the payment is a fine and a penalty, but the first half is the principal, the money that he owes him. And since a principal is being claimed here, the witnesses are liable.
"o tashlumei arba'ah vachamishah" - one who steals a sheep or an ox and slaughters or sells it, who is liable for fourfold and fivefold payments. Even though there is a fine here, there is also a principal, and because of the principal the witnesses are liable.
"veshe'anas ish peloni es biti" and "ufitah es biti" - these carry an obligation of a fine, but they also carry an obligation for humiliation and blemish, that is, a monetary claim for the reduction of personal worth and for shame.
"veshehikani beni" - a son or daughter who struck their parent and actually made a wound, drawing blood, is liable to death, and wherever there is liability to death there is no monetary liability. Here, however, the case is a scenario in which no blood was spilled, and therefore there can be damages - injury, pain and the like - which are monetary damages for which the son is liable, since no death penalty accompanies the act. For this reason witnesses who refuse to testify are liable for the oath of testimony.
"veshechaval bi chaveri" and "veshehidlik es gedishi beyom hakippurim" - that his fellow wounded him and drew blood from him, and likewise that he set fire to his pile of grain, and both acts were done on Yom Kippur. One who wounds on Yom Kippur or who kindles a fire on it is liable to kares, and even so the obligation to pay stands in place: a person pays for having struck someone or for having set fire to his fellow's grain. As long as there is no liability to death, the obligation for the monetary damage remains.
In summary: in all the cases listed in the Mishnah the witnesses are liable, since they involve a monetary claim, even if a fine or a penalty accompanies it. In a case of a pure fine, however, the witnesses are exempt, because the defendant's failure to admit is a contributing factor in his liability, and the witnesses alone are not what obligates the money.