Yevamos, Chapter 4, Mishnah 5. The Mishnah establishes: "Mitzvah b'gadol l'yabem" - the mitzvah of yibum rests upon the oldest of the surviving brothers. The source for this law is derived from the verse "And it shall be that the firstborn whom she bears": the firstborn is the one who is meant to fulfill the mitzvah of yibum. This is not the plain reading of the verse, but rather a derashah of Chazal, from which we learn that the obligation falls upon the oldest of the brothers.
The order in which the brothers are approached:
"Lo ratzah - mechazrin al kol ha'achin" - if the oldest does not wish to perform yibum, we turn to the remaining brothers, one after another.
"Lo ratzu - chozrin etzel hagadol" - if they all refuse, we return to the firstborn and say to him: "Alecha mitzvah, o chalotz o yabem" - the mitzvah rests upon you, and you must fulfill one of the two: chalitzah or yibum.
The reason we return to him: the mitzvah rests upon him from the outset. Initially, once he refused, it was possible to turn to the other brothers; but once they too pushed the matter aside, the obligation returns to its original place - to the oldest.
In summary: in this Mishnah we learned that by strict law the mitzvah of yibum rests upon the oldest brother, as derived from the verse "And it shall be that the firstborn whom she bears"; if he refuses, we turn to the remaining brothers, and if they refuse, the obligation returns to the oldest, who must perform either chalitzah or yibum.