As nouns,
hexadecimal notation is a hyponym of
mathematical notation; that is, hexadecimal notation is a
word with a more specific, narrower meaning than
mathematical notation:
mathematical notation: a notation used by mathematicians
hexadecimal notation: any notation that uses 16 different characters