Throughout history, nutmeg has been prized for its medicinal properties. Ancient Arabs used it to treat nausea, shortness of breath, even skin disorders. In ancient China, women prized it as an aphrodisiac. In India, nutmeg has a long history as an aphrodisiac and thought to increase sexual stamina. Research has found it to increase mating behaviors in mice, however, there is no evidence to prove the same happens in humans. Studies show nutmeg used in high doses (not the small amounts used in cooking) to be a narcotic and hallucinogen and in even larger doses (like eating 2 whole ones), fatal.
The fact that nutmeg does produce an hallucinogenic effect may be the reason it is considered an aphrodisiac. People just think they want sex..