Font Book may not be perfect, but I've regretted every third-party utility I've installed, especially when they turn out to have their own little quirks that make hours of work counterproductive (such as destroying previous organization). Linotype FontExplorer™ X was the least irritating, and I've never fully recovered from installing it.
There's no reason they should slow down your computer after they "load" (when the software launches), which takes a few extra moments, but if you have to stop and find and then activate fonts when you need them, it may be more distracting and time consuming.
Another possibility: less may be more. There may be such a thing as too many fonts (just to kick off the discussion).

Yeah i guess less is more, but with my job i need access to as many fonts as possible really, which is why other people at work use suitcase, for all the different styles. I did try installing the demo of suitcase the other day, but decided to get rid of it pretty much straight away which then made me need to re-install illustrator because of errors which was frustrating.

I guess my best option is to get an SSD so it will all load up faster, just a bit pricey though.

Use a font manager! I quite like NexusFont.