In developing a solution in Azure to [REDACTED], I discovered a “bug” in HttpClient that seems to be somewhat common knowledge but I figured that I would share it with you, in case you run into the same problems.
This bug surfaced – moreso – because we’re using Azure, than anything else. You see, anything in Azure should be considered multitenancy; meaning that your app is running parallel to – potentially – hundreds of other apps within the infrastructure’s backplane.
So, I was using Parallel.ForEach and .ctor’ing a new HttpClient per thread and making a call to obtain data from a rest endpoint from a unique sub-url; which was entirely dependent on data that I had previously obtained via another rest call. Obscurity is strong with this one, I’m aware.
Every once in a while, I would get the exceptions (by unboxing them): “Only one usage of each socket address (protocol/network/port) is normally permitted: <ipAddressGoesHere>.”
The socket would still be in use, even if it were. This is because the socket that the HttpClient uses is put into TIME_WAIT. So, you have an open socket in use to that host and because the host hasn’t closed the socket, if you instantiated all new HttpClients (which used new ephemeral ports), you could potentially run out of ports to consume.
…but, wait, there’s more!™
The HttpClient is considered reentrant (and this is where our true problem comes in). This means that some (if not all) of your non-disposed of HttpClients could be re-used to try to go what the HttpClient considers a currently in-use object (because the port is still considered open when it’s in TIME_WAIT).
In fact, if we chase down the SocketException –> Win32Exception –> hResult, we can see that this comes from the system as 0x00002740, which is WSAEADDRINUSE.
The solution? Since public static (I think just static, really) instances of HttpClient are considered thread-safe, the singleton model is what we want to go with.
So, instead of instantiating an HttpClient per call, you would instantiate a singleton instance that would be used per-host in your class instance. This allows the HttpClient and it’s port to be re-used (thus, reducing potential ephemeral port exhaustion as a byproduct). And since it appears that Azure re-instantiates your class per run (if you’re using the TimerTrigger, for example), then you create a scenario where the object’s lifetime is bound to your class. (Assuming you call HttpClient.Dispose() before the run completes and the object moves out of scope.)
“…but MSDN says to use ‘using’ for IDisposable objects!”
Yes, this is true but, again, we have to consider that even though .Dispose() might be called when we leave scope, we have no control over when GC actually comes through and disposes of the object from the Gen1/Gen2 heaps. We also cannot control when the TCP port is actually closed because that’s dependent on the host. So, even if HttpClient.Dispose() is called, you’re still at the whims of the keep-alive configured on the host for the actual port to be closed.
Diagram from the IETF? Diagram from the IETF.
So, even though it’s been practically beaten into you throughout your CS career to use ‘using’, there are times when the singleton model (and not invoking using) are more favourable to your software design needs, expectations, and requirements than what you’ve been taught is the best practice.
Happy coding! 🙂