Mishnah 3 in Chapter 11 of Tractate Sanhedrin continues to expand on the laws of the rebellious elder. The point of the Mishnah is that in order for a person to be held liable as a rebellious elder, he must issue some ruling, instruction, or interpretation regarding the words of the Torah, which constitutes a "matter of judgment" as the verse states - a matter of interpretation and halachic ruling, as opposed to something explicitly written in the Torah in black and white.
"Chomer bedivrei sofrim midivrei Torah" - There is greater stringency in the words of the Scribes than in the words of the Torah:
There is greater stringency regarding the Sages' interpretation of the Torah's words than regarding the Torah's words themselves. The Mishnah demonstrates this with two cases:
"Ha'omer ein tefillin, kedei la'avor al divrei Torah - patur" - One who says there is no tefillin, in order to transgress the words of the Torah, is exempt. The elder rules that there is no such thing as tefillin, and says that the time for tefillin has passed, and even though the Torah instructs us to don them - he instructs not to. This ruling is not considered an instruction and is not considered a halachic decision, since the Torah states this explicitly - "totafot between your eyes" and the like - and every schoolchild knows that the Torah requires donning tefillin. The very denial of their existence in the verses demonstrates that he is not reading the verses at all, and therefore this does not constitute a ruling.
"Chamesh totafot, lehosif al divrei sofrim - chayav" - One who says there are five totafot, adding to the words of the Scribes, is liable. This refers to one who says that there are five compartments in the head tefillin, as opposed to the four that are supposed to be there. The number of passages is not explicitly stated in the Torah, but is derived through exegesis: We have a tradition that these four passages go into the tefillin and that they must be four separate passages, and we learn this from the fact that the word "totafot" is mentioned three times - twice missing a letter vav and once full - yielding four parts. It turns out that he is seemingly coming to add to the tradition of the Sages, but anyone who adds actually detracts: Tefillin with five compartments are invalid. Since he gave an instruction and an interpretive ruling on the verses, which contradicts the tradition of Israel and the ruling of the Sanhedrin - he is liable as a rebellious elder.
The Halachah - For which rulings is a rebellious elder liable:
The matter is somewhat complex, since in the Mishnah and the Gemara itself several opinions are cited regarding which types of instructions and rulings constitute grounds for the law of a rebellious elder. The Rambam ruled on the matter, and according to his ruling:
A matter for which intentional transgression carries the penalty of karet and unintentional transgression requires a sin-offering - If he intentionally committed that transgression, he would be liable to karet or worse, such as death, and if he acted unintentionally, he is obligated to bring a sin-offering. Only severe transgressions of this type, those negative commandments, are the matters through which a person can become a rebellious elder.
Tefillin - Even though they are a positive commandment, their uniqueness lies in the fact that anyone who adds to them detracts, whether by adding to them or subtracting from them, and therefore the law of a rebellious elder potentially applies to tefillin as well.
A ruling with potential implications for karet - Such as a rebellious elder who rules on establishing the calendar, adding a month or a day to the month, in a way that alters the day on which Pesach will fall. Since eating chametz on Pesach is prohibited under penalty of karet, it turns out that his ruling against the opinion of the Sanhedrin creates an implication that could lead to a liability of karet, and therefore this case is also included in the law of a rebellious elder.
In contrast, other types of rulings are not included in this: If he ruled that for the Four Species one must take an apple tree branch instead of a lulav, or that one should not blow the shofar, or that one must blow a cow's horn, and the like. None of these carry the punishment of karet, and therefore even if he ruled so, he is not liable to the death penalty as a rebellious elder.
In summary: In this Mishnah we learned that the liability of a rebellious elder depends on him issuing an interpretive ruling - "a matter of judgment" - and not on denying something explicitly stated in the Torah, and hence "greater stringency applies to the words of the Scribes than to the words of the Torah." In practice, liability only applies to a ruling concerning a matter for which intentional transgression carries the penalty of karet and unintentional transgression requires a sin-offering, to matters of tefillin where anyone who adds detracts, and to a ruling with an implication that could lead to karet, such as establishing the calendar; whereas for rulings that do not involve karet - he is exempt.