what kind of error you are getting on modalpopup. Is there any javascript error on left corner of IE or some other error?

Firstly, you'd better download a sample here. It should work on .NET Framework 2.0. If ASP.NET Ajax Extension v1.0 is not installed, please install it first. You can get the install file here. (ASP.NET AJAX Extension for .NET Framework 3.5 is integrated into .NET Framework 3.5).
If it works, please compare it with your sample. Especially, the web.config part.
If all these cannot figure out your problem, please download the version of .NET Framework 2.0 here (the version 1.0.20229 with source code). Before use it, please recompile it first. The installation steps, please follow the first two video tutorials here.
If you are using .NET Framework 2.0, please compare your sample's web.config with the downloaded sample. Otherwise, if you are using .NET Framework 3.5, please compare it with the downloaded sample either.
Note:
Version 3.0.20229
2008-02-29 release of the AJAX Control Toolkit targets the official release of .NET Framework 3.5 and Visual Studio 2008.
Version 1.0.20229
You can also download the Toolkit for .NET Framework 2.0, ASP.NET AJAX 1.0 and Visual Studio 2005.
Version 3.0.30512
2009-05-13 release of the AJAX Control Toolkit targets the official release of .NET Framework 3.5 SP1 and Visual Studio 2008 SP1.

Some tutorials:
http://www.asp.net/ajax/ajaxcontroltoolkit/samples/
http://www.asp.net/ajax/downloads/
http://www.codeplex.com/Wiki/View.aspx?ProjectName=AtlasControlToolkit
http://www.asp.net/learn/Ajax-Control-Toolkit/tutorial-47-cs.aspx
http://www.asp.net/learn/ajax-videos/video-81.aspx

Best regards,