As nouns,
mathematical notation is a hypernym of
hexadecimal notation; that is, mathematical notation is a
word with a broader meaning than hexadecimal notation:
hexadecimal notation: any notation that uses 16 different characters
mathematical notation: a notation used by mathematicians