I'm fairly sure it started with Zapf Dingbats, as the whimsical name ITC chose for the font. The word "dingbat" was originally one of those words for arbitrary unnamed objects like "doohickey" and "thingamabob".

NOED:

dingbat |ˈdɪŋbat| informal
noun
1 a stupid or eccentric person.
2 a typographical device other than a letter or numeral (such as an asterisk), used to signal divisions in text or to replace letters in a euphemistically presented vulgar word.
ORIGIN mid 19th cent. (in early use applied to various vaguely specified objects): origin uncertain; perhaps based on obsolete ding [to beat, deal heavy blows.] Sense 1 dates from the early 20th cent.

The OED says that the term entered the English Language in the mid-1800s, source uncertain, to describe a miscellany of objects in general. The OED further states that the term was applied to a miscellany of type in the early Twentieth Century.

Well, Unicode does call them "miscellaneous symbols."