sex vs maleness

sex and maleness both are nouns.

Word	Noun	Adjective	Verb	Adverb
sex	Yes	No	Yes	No
maleness	Yes	No	No	No

As nouns, maleness is a hyponym of sex; that is, maleness is a word with a more specific, narrower meaning than sex:

sex: the properties that distinguish organisms on the basis of their reproductive roles
maleness: the properties characteristic of the male sex

Other hyponyms of sex include masculinity, androgyny, bisexuality, hermaphroditism, femaleness, feminineness.

sex (noun)	maleness (noun)
activities associated with sexual intercourse	the properties characteristic of the male sex
the properties that distinguish organisms on the basis of their reproductive roles
all of the feelings resulting from the urge to gratify sexual impulses
either of the two categories (male or female) into which most organisms are divided

sex (verb)	maleness (verb)
tell the sex (of young chickens)
to evoke sexual feelings