I am having the same problem displaying a modla dialog. It used to work, but when the panel that contains the button that ran the code to display the modal dialog is wrapped in an update panel, it quit working.
Private Sub btnShowSpargerParms_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnShowSpargerParms.Click
MakeModelInputTable()
Response.Write("<script language='javascript'>showModalDialog('SpargerChooser.aspx', null,'status:no;dialogWidth:400px;dialogHeight:650px;dialogHide:false;help:no;scroll:no');</script>")
End Sub
MakeModelInputTable gets some parameters from the current page, puts them into a session variable, then I display the modal dialog which does some calculations in the pageload using the parameters in the session variable, then displays the results on the modal dialog.