The "family name" can't be more than 27 characters, and can't include hyphens. If you're working in FontLab, the "Yin/Yang" symbol in the Name section will catch some of these issues.

Jason: ‘The "family name" can't be more than 27 characters, and can't include hyphens.’

Although the problem on the Mac could well be caused by the naming of the font, I don’t think the Font Family name has anything to do with this.

Assuming that we’re talking here about OpenType fonts, according to the specs there seems to be no limitation to the number of characters in the Font Family name, nor that the use of a hyphen is not allowed. The ‘27 characters’ seem to refer to the specs of the PostScript Type1 format. As one can read in Adobe’s Font Naming Issues, Technical Note #5088, 12 April 1993: ‘The FontName generally consists of a family name (specifically, the one used for FamilyName), followed by a hyphen and style attributes in the same order as in the FullName. For compatibility with the earliest versions of PostScript interpreters and with the file systems in some operating systems, Adobe limits the number of characters in the FontName to 29 characters.’

Macintosh Font Menu Names were also restricted: ‘The standard names used in Macintosh font menus come from the name of the FOND resource associated with a Type 1 outline font. FOND resource names are limited to 31 characters, and may contain spaces. Because of a bug in a major software application which limits font menu names to 30 characters, Adobe recommends staying within this lower limit. In addition, System 7 for the Macintosh requires names to be unique for the first 28 characters.’ I have to check MS Office 2011 for Mac on one of our test systems, but AFAIK MS Word for Mac always required a Full font name of maximum 30 characters.

So, probably you can easily solve this problem by using the Compatible Full Name (Name ID 18) for the Mac: ‘Compatible Full (Macintosh only); On the Macintosh, the menu name is constructed using the FOND resource. This usually matches the Full Name. If you want the name of the font to appear differently than the Full Name, you can insert the Compatible Full Name in ID 18.’ With 27 characters one is sure to stay on the safe side.

I think what makes matters a bit more complex here is the Topic: Font Family Naming in FontLab Studio 5 article, which mentions limitations which are obviously related to the way FontLab Studio handles font naming internally. Jason is referring to this, I reckon.

FEB

Two things come to mind:
1) Is maybe a name table record with NID2 missing?
2) Does the font not appear in the menu at all or is it merely sorted into the CE or any of the other non-Latin sections further down the menu?

Adam Twardoch has covered Word 2011 family/menu name issues pretty thoroughly.

http://typophile.com/node/95677
http://forum.fontlab.com/index.php?topic=313.0

Last year there was a brief discussion on the OpenType list regarding the buggy way MS Word 2011 for Mac handles font names. After Adam stated ‘The way Word retrieves naming info is a mystery to me […]’, Read Roberts replied on Monday 16 July:

‘This is a known bug on Mac MS Word 2011. It is not in previous versions. We saw the problems in Adobe fonts, and looked into it in Dec 2011. A font family will not appear In the hierarchical font menu for a given font file if there is no font file in the family for which the Cocoa API returns a base font.  In order to see the problem behavior, I needed to install several fonts from the same family: the API seems to have trouble in the fairly specific case of  multiple style groups in a set of fonts linked by having the same Mac name ID 16 (preferred name), and where some of the style groups do not have a Regular face.
        We saw this when we rebuilt the Adobe Type Library to build Mac menu names name ID1,2,16 and 17) the same way as Windows name ID is built. (Incidentally, we made this change partly to be in accordance with the OT spec, and partly because the way we originally built the Mac menu names was due to a compromise to make OTF work in Mac OSX in 2000. Discussion with Apple engineers last year informed us that a) this is no longer necessary and b) parts of the Apple font handling logic no longer expect more than four faces with the same name ID 1).)
        We are not re-building our font library as we do expect that this bug will be fixed in some future release of MS Word for the Mac.

Especially style-linking can cause problems and requires the usage of Name ID 16 and 17. Read gave a nice example on the OpenType list on 1 December 2010:

‘An additional note about building name ID 4: when Adobe builds name ID 4 for new fonts, we use the name id 16 and 17 when available, rather than only name ID 1 and 2.  This avoids name ID 4 conflicts within a large family, where name ID 1 + name ID 2 for one face might be the same as name ID 1 for
another. Example:

AGarmondPro-SemiBold: is the bold face of AGarmondPro-Regular
   name ID 1 = "Adobe Garamond"
   name ID 2 = "Bold"
   name ID 4 by OT spec v1.5 and earlier  = "Adobe Garamond Bold"

AGarmondPro-Bold: is the regular style face within the style linked group  (
AGarmondPro-Bold and " AGarmondPro-BoldItalic").
   name ID 1 = "Adobe Garamond Bold"
   name ID 2 = "Regular"
   name ID 4 by OT spec v1.5 and earlier  = "Adobe Garamond Bold"’

At DTL we have never used Name ID 16 and 17 for the Mac so far (despite Read’s remark that ‘[…] parts of the Apple font handling logic no longer expect more than four faces with the same name ID 1 […]’). We use any style name for the Font Subfamily name and we use Name ID 16 and 17 for Windows to duplicate this. By keeping the length of Name ID 18 at a maximum of 31 characters, our fonts seem to function well under MS Word 2011 for Mac (I just gave this an extra check):

TTName 0 1 0 0 "Copyright (c) Dutch Type Library, 1994-2012. All rights reserved."; #Macintosh
TTName 0 3 1 0x409 "Copyright (c) Dutch Type Library, 1994-2012. All rights reserved."; #Windows
TTName 1 1 0 0 "DTL Fleischmann DOT"; #Macintosh
TTName 1 3 1 0x409 "DTL Fleischmann DOT Medium"; #Windows
TTName 2 1 0 0 "Medium Italic"; #Macintosh
TTName 2 3 1 0x409 "Italic"; #Windows
TTName 4 1 0 0 "DTL Fleischmann DOT Medium Italic"; #Macintosh
TTName 4 3 1 0x409 "DTLFleischmannDOT-MediumItalic"; #Windows
TTName 6 1 0 0 "DTLFleischmannDOT-MediumItalic"; #Macintosh
TTName 6 3 1 0x409 "DTLFleischmannDOT-MediumItalic"; #Windows
TTName 7 1 0 0 "DTL Fleischmann is a trademark of the Dutch Type Library."; #Macintosh
TTName 7 3 1 0x409 "DTL Fleischmann is a trademark of the Dutch Type Library."; #Windows
TTName 8 1 0 0 "Dutch Type Library"; #Macintosh
TTName 8 3 1 0x409 "Dutch Type Library"; #Windows
TTName 9 1 0 0 "Johann Michael Fleischmann, Erhard Kaiser"; #Macintosh
TTName 9 3 1 0x409 "Johann Michael Fleischmann, Erhard Kaiser"; #Windows
TTName 10 1 0 0 "retail edition"; #Macintosh
TTName 10 3 1 0x409 "retail edition"; #Windows
TTName 11 1 0 0 "http://www.dutchtypelibrary.com"; #Macintosh
TTName 11 3 1 0x409 "http://www.dutchtypelibrary.com"; #Windows
TTName 13 1 0 0 "By downloading, unpacking […]"; #Macintosh
TTName 13 3 1 0x409 "By downloading, unpacking […]"; #Windows
TTName 14 1 0 0 "http://www.dutchtypelibrary.nl/PDF/Licenses/DTL_FS_License.pdf"; #Macintosh
TTName 14 3 1 0x409 "http://www.dutchtypelibrary.nl/PDF/Licenses/DTL_FS_License.pdf"; #Windows
TTName 16 3 1 0x409 "DTL Fleischmann DOT"; #Windows
TTName 17 3 1 0x409 "Medium Italic"; #Windows
TTName 18 1 0 0 "DTL Fleischmann DOT Medium Ita"; #Macintosh
TTName 19 1 0 0 "The quick brown fox jumps over the lazy dog."; #Macintosh
TTName 19 3 1 0x409 "The quick brown fox jumps over the lazy dog."; #Windows

In the Font Family Naming in FontLab Studio 5 article some rules and restrictions are given for the lengths of names in FLS. As I mentioned earlier, some of these rules and restrictions seem to come forth from the way FLS handles naming internally. For instance, the less-than-30-characters restriction for the PostScript name differs from the ‘no longer than 63 characters’ in the OT spec.
        A way to circumvent these restrictions is using for instance OTM for importing (and exporting, of course) the naming table (CFF-naming entries require manual editing though). This can also be useful when for instance fonts have to be renamed for use on the web. Small changes can be made in a text editor (in batch) then. 

FEB

The problem is most likely in the Full name. It must not be longer than 31 characters -- if it's longer, Word 2011 for Mac won't display the font (although the spec actually permits 63 characters there).

We have incorporated this limitation into TransType 4 and will incorporate it into our new applications.

Thanks to all for your input. Shortening the name and tweaking the Mac name outputs appears to have solved the problem.