As nouns,
mathematical notation is a hypernym of
sexadecimal notation; that is, mathematical notation is a
word with a broader meaning than sexadecimal notation:
sexadecimal notation: any notation that uses 16 different characters
mathematical notation: a notation used by mathematicians