As nouns,
mathematical notation is a hypernym of
system of numeration; that is, mathematical notation is a
word with a broader meaning than system of numeration:
system of numeration: any notation for the representation of numbers
mathematical notation: a notation used by mathematicians