http://docs.info.apple.com/article.html?artnum=25710
This shows the fonts necessary for 10.3 there might be an update for 10.4 somewhere on the site.
Tim

Thanks Tim!
Here’s the link to 10.4 in case anyone needs it.

It seems like I can remove all fonts except Geneva.dfont, Keyboard.dfont, LastResort.dfont, Monaco.dfont and Lucida Grande.dfont.

Here’s some additional information that I found about this:

Additionally, five fonts are installed in System/Library/Fonts that you may want to replace with your PostScript version of the fonts. Service bureaus might want to replace fonts with a customer’s version on a per-job basis using a font manager. These fonts are Courier.dfont, Times.dfont, Symbol.dfont, Zapf Dingbats.dfont and Helvetica.dfont.

While Helvetica is not an essential system font, some applications still require an active version of Helvetica in order to operate correctly. If you regularly use any applications that require Helvetica: As soon as you remove Helvetica from your system font folders, be sure to activate a version of Helvetica using your font manager (Suitcase or Font Reserve). It doesn’t matter which font format you use (PostScript, OpenType, TrueType, etc), but for professional creative, print and publishing environments, we recommend using a PostScript version of Helvetica.

Oystein