subgroup vs mathematical group

subgroup and mathematical group both are nouns.

Word	Noun	Adjective	Verb	Adverb
subgroup	Yes	No	No	No
mathematical group	Yes	No	No	No

As nouns, mathematical group is a hypernym of subgroup; that is, mathematical group is a word with a broader meaning than subgroup:

subgroup: (mathematics) a subset (that is not empty) of a mathematical group
mathematical group: a set that is closed, associative, has an identity element and every element has an inverse

Other hypernyms of subgroup include group.

subgroup (noun)	mathematical group (noun)
(mathematics) a subset (that is not empty) of a mathematical group	a set that is closed, associative, has an identity element and every element has an inverse
a distinct and often subordinate group within a group