The DOCTYPE has no effect on the CSS itself. Rather, the DOCTYPE (or lack of one) affects how the browser renders the page.
Generally, browsers have two modes - "Standards" mode and "Quirks" mode. There is tons of information on the web about Standard vs. Quirks mode, and how each is triggered by each major browser depending on the included doctype. Each browser is a little bit different.
In nearly ALL browsers, if the page has no doctype, the page will be rendered in Quirks mode. Basically, this means that it will use the IE5 Box Model to render output. IE5 used a very unorthodox (non-standard) method of interpreting the rendering of margins, borders, and padding for boxes.
If you never had a doctype before, that means your site has been rendered using Quirks mode all along. And since you developed it that way, you made it look "right" for Quirks mode. The second you add a doctype in there, browsers will render it in Standards mode, which will totally break any "Quirks-mode" site.
This is always a problem when you attempt to use modern browser technology (like AJAX) in a site which was never properly updated to comply with modern standards.
Using a different doctype is very unlikely to work, because most likely ANY doctype will force your site into Standards mode, and that is what is causing the trouble.
I know you won't like hearing this, but the best way to solve this problem is to include doctypes in all your pages and rework the design to render properly in Standards mode.