The growing problem of male infertility should be a cause for concern. When a man has had frequent unprotected sexual contact with a female but is not pregnant, it's called male infertility.