Some Email servers simply don't respond to invalid credentials and let the caller timeout eventually. They do this to prevent spammers from trying to hack onto your email server by constantly trying invalid credentials.
For example if you take our email server, if you fail the connection too many times it blocks your IP for about 10 minutes, and if this occurs more more than once or twice it will block your IP for a few days.
When the IP is blocked the email program will get no response for a connection and thus hang for a while until the connect times out.