Just to chime in that I've had the same problems.
1) I built an ldap query using the handy little win2k3 admin pack, which lets you create a query and then have the text of what you did. When I run it in the "Users & Computers" thingy, my request takes a fraction of a second.
2) I take the same code, and drop it into a DirectorySearcher(), doing a for/each to build a collection, and it takes about two minutes and beats the tar out of my web server's processor.
Haven't tried going against an IP instead of the name yet, although I don't understand why that would matter a lot since (I would hope) that when I send my request over that's the end of the conversation with the domain controller; maybe the reference variable is holding a reference all the way back to the LDAP source? :shudder:
So like everyone, I'm not sure why this is causing so much of a headache, and if I come up with anything, I'll of course post it.