Hi,
Web Setup project installs the website into default virtual directory such as "C:\Inetpub\wwwroot".
If we want to install the website to another location, we can create Custom Action to create virtual directory that is specified by "TARGETDIR" of Web Setup project.
For more information about using Custom Action to create Sites and Virtual Directories in IIS with System.DirectoryServices, see http://msdn.microsoft.com/en-us/library/ms524896.aspx
In order to provide user with dialog where user can input target directory, we can use Orca.exe.
For more information, see http://www.codeproject.com/KB/install/ChangeVDirWebSetupProject.aspx
For more information about Orca, see http://support.microsoft.com/default.aspx/kb/255905
In your case, you can try to debug your Custom Action and step through the source code to see whether there is any issue while installing (Add System.Diagnostics.Debugger.Break() in Custom action and use Visual Studio debugger to attach to its process while Custom action breaks).
I look forward to hearing from you.