Depending on a series of factors, not least age of intended readers.

Start with something much smaller than a book!

hhp

I recommend you start by checking out these books. Though both written decades ago, they remain the standard references for book typography:

The Design of Books by Adrian Wilson

Bookmaking by Marshall Lee

There's no formula, but some factors that are particularly important to consider are the size of the page (or width of the text column) and the x-height and width of the font. Also budget constraints, printing method and paper choice, the kind of book you want it to look like and, as Riccardo mentioned, the target readership.