You need to capture the performance counter parameters of server to exactly get idea what is slowing down your system and its not really easy, some times we need to capture the snapshot of memory and analyze them later for analysis as well. Actually it has several factors may be you allocate the limited resources to your  application pool and that may slow down your server, may be there is memory leakage or may be you block threads when doing some long running operation so it depends what is slowing down your system and it only give you information when you capture performance counters.

Another option is using Azure Application Insight , integrate it in your application and it will give you detail information of Server and Client parameters.