Informal:
* Mon frère est chiant. (My brother is annoying.) - This is the most common and informal way to say it.
* Mon frère est insupportable. (My brother is unbearable.) - This is a slightly stronger version of the previous phrase.
* Mon frère me gonfle. (My brother gets on my nerves.) - This implies that he's constantly bothering you.
* Mon frère me fatigue. (My brother tires me out.) - This suggests that you're finding his behavior exhausting.
Formal:
* Mon frère est agaçant. (My brother is annoying.) - This is a more polite way of saying it.
* Mon frère est exaspérant. (My brother is exasperating.) - This is a more formal way of saying he's very annoying.
You can also use more specific phrases depending on the type of annoying behavior:
* Mon frère me fait toujours des blagues nulles. (My brother always tells bad jokes.)
* Mon frère est toujours en train de me chatouiller. (My brother is always trying to tickle me.)
* Mon frère est toujours en train de me prendre mes affaires. (My brother is always taking my things.)
The best way to say it will depend on your relationship with your brother and the context of the situation.