Hi,
Firstly, you can define a button in the IFrame page to trigger closing popup panel. And define OnClientClick event, which can execute hidepopup() javascript event in parent page before executing server click event.
<asp:Button ID="Button1" runat="server" Text="Button" OnClientClick="javascript:window.parent.hidepopup()" OnClick="Button1_Click"/>
In the parent page, you need to declare hidepopup() event.
function hidepopup()
{
var modalPopupBehavior = $find('ModalPopupBehavior'); // check the behavior you defined
modalPopupBehavior.hide();
}
Hope it helps.