Before you create a experiment I suggest you talk a bit more with the folks already doing this kind of research so that you get an idea of what research has already shown so you can better understand what it would take to do a meaningful piece of research. Peter Enneson & Kevin Larson might be willing to talk to you. I applaud your enthusiasm; But as they say, "Festina Lente,” the Latin phrase meaning “hasten slowly”.

I’ll set the same word in different fonts, have subjects read through them in random order, record how long each took. I still have to decide what “reading” means here; I’ll figure out the details over the week.

Those test are done for decades. You should do a little research before you start your own survey.
Also, I guess you need to set yourself a particular goal. "Testing legibility" or »Testing readability« is way too broad. You will not get any useful results when you don't focus on single parameters.

When I'm developing news text faces, I pick an existing face as a benchmark and try to match the character count, with identical leading, column width, and h&j settings.

I then try to match the apparent size of my new face to the benchmark.

This approach is necessary because my clients always compare my new face to their existing face, and want it to look "as big" -- which is code for readability -- if not bigger, and they don't want to lose words.

It's sometimes difficult to get an exact match, and the comparative settings may involve fractional point sizes and leading sizes. However, the main thing is character count.

Having done this, it's still not unusual for a client to nuance the setting once they have the finished font. For instance, I had optimized one typeface for 9 on 9.5, and my client bumped it up to 9.3 on 9.5. Another client decided that their face needed negative 10 tracking. Yet another likes to horizontally scale 95%!

Minute changes to the polyvariant complex of size, leading, scaling, tracking and h&js can have a huge effect, which is why the idea that typeface readability can be reduced to a simple measurement is completely banal.

I didn't do much research before I posted this. Now that I see that this has already been done a million times, I'll just play around with it so I can see the difference myself. I won't worry too much about whether the methodology is rigorous. I think I'll print some random phrases in different faces and sizes and see how hard they are to read from different distances. Later, I may also try setting random texts in different faces and timing myself to read them, but I still have to figure out how to choose suitable texts without reading them.

I think it would be good to choose a text you can read because otherwise your are not - reading. It might be a good idea to keep reading about methodologies as well. If your goal is not to compare readability though go with what feels good & maybe read Lawson's "Letters of credit". And Bringhurst. See the books area in "resources" up above.

Eben: read Lawson’s “Letters of credit”

You're confusing two (very good) books, Eben:

Walter Tracy, Letters of credit.

Alexander Lawson (with Dwight Agner), Printing Types.

Ah yes. Thanks John! Notwithstanding my confusion re: attribution, it is wonderful how both books repay re-reading. So make that ( at least ) 3 books.

Alexander Lawson (with Dwight Agner), Printing Types.

Or, Alexander Lawson, Anatomy of a Typeface.