The problem with web.config is that it is a big, messy gloop of everything. You have whole sections such as <appSettings> and <connectionStrings> which will vary between your development, staging and production servers, and other sections which should be the same on all machines, such as HTTP handler and HTTP module definitions, remoting, and so on. In effect, you're mixing code and configuration in one file.
Now I like to organise my projects so that configuration settings are stored outside of the web application root. This way it's easy to version your deployments: I can put the web files at, say, c:\inetpub\wwwroot\304, then when I upgrade to build 305 I just change the IIS settings to point to c:\inetpub\wwwroot\305. Rolling back if necessary then becomes similarly trivial. I would then put the configuration files in, say, c:\inetpub\wwwroot\config, and wouldn't need to touch them from one deployment to the next.
There's no problem here with appSettings: you can do <appSettings file="..\config\appSettings.config" /> to include an external file outside your application root. However, there are several problems elsewhere:
First, while the configSource attribute can point to other files for sections such as <connectionStrings>, unlike appSettings these are restricted to being within the web application's directory tree. This means that I can't simply upload a new build and change the IIS settings to point to the new one, I also have to copy extra files into the new upload directory, adding an extra step to the deployment process.
Second, some settings that need to change between servers are buried in sections that should be the same across all installations but may need to vary from one build to the next. The <compilation> section is an example here: the debug attribute should be set to true on your development box but false on your production server, and it contains a whole lot of stuff that should be the same across all servers, e.g. referenced assemblies, whether you are using C# 2.0 or 3.0, and so on. This is complicated by the fact that when you use configSource it's all or nothing -- you can't have one part of the section in web.config and the rest in the included file.
I'd like to see three changes here:
1. The configSource attribute should be allowed to point to files outside of your web's root directory. If there are security issues here (as is quite possible in a shared hosting environment) this could be restricted to code running with appropriate HostingPermissions, but it should certainly be allowed for applications running under full trust.
2. I would also like to see some way of settings specified in configSource files to be merged with the settings in the corresponding sections in web.config.
3. I'd also like to see some way of specifying which setting takes precedence in the case of a conflict. With appSettings, if I recall correctly, the settings in web.config take precedence over the included file, whereas it would be more useful to ask for the settings in the included file to take precedence, so you could use web.config for defaults and the included files for server-specific settings.