Hi,
Based on my test, I can't reproduce your issue. So I suggest to try to run the following code to open model popup which is inside another one.
<asp:ScriptManager ID="ScriptManager1" runat="server">
</asp:ScriptManager>
<asp:Button ID="Button1" runat="server" Text="open first modalpop" />
<asp:Panel ID="Panel1" runat="server" BackColor="#CCFFFF" Height="323px" Width="743px">
<asp:Button ID="Button2" runat="server" Text="open other modal" />
<asp:Button ID="Button3" runat="server" Text="cancel" />
<asp:Panel ID="Panel2" runat="server" BackColor="#FF3300" Height="194px" Width="430px">
<asp:Button ID="Button4" runat="server" Text="cancel" />
</asp:Panel>
<cc1:ModalPopupExtender ID="ModalPopupExtender2" TargetControlID="Button2" CancelControlID="Button4"
PopupControlID="Panel2" runat="server">
</cc1:ModalPopupExtender>
</asp:Panel>
<cc1:ModalPopupExtender ID="ModalPopupExtender1" TargetControlID="Button1" CancelControlID="Button3"
PopupControlID="Panel1" runat="server">
</cc1:ModalPopupExtender>
If it still not work, please post the code here.