As nouns,
mathematical notation is a hypernym of
numeration system; that is, mathematical notation is a
word with a broader meaning than numeration system:
numeration system: any notation for the representation of numbers
mathematical notation: a notation used by mathematicians