I don't think there are any rules, per se, for ornaments & borders. There are a lot of discussions, though.
There used to be an Opentype feature for Ornaments, but I think there is so little support for it, it quietly got ditched. See Open Type Ornaments in Indesign? (How to) for some sample OTF code for this (and additional remarks on using them in InDesign), and Opentype Access to unencoded ornaments & dingbats for comments on a designer with the same question.
As for encoding, you can (a) use 'alternates' for various characters -- a..z, for example -- to 'switch on' your borders, (b) use a single alternate to access all of them -- the bullet seems to be Adobe's preferred catch-all --, although this has various related drawbacks (esp. when using InDesign -- see Adobe Forum: Opentype ornaments for a complaint related with this method), or (c) assign unique Unicodes to your ornaments (it's up to you whether or not to assign unique Unicode codes, as this has no effect on any Opentype features you might want to add to support your ornaments).
If you do assign Unicodes, then use the PUA range for this. If an application does not support the OTF feature you use, and you did not assign Unicodes, there is no way to get your borders into a document!
The PUA range is the "Private Use Area" (or sth. similar); you can assign whatever character you like to the code range U+E000..U+F8FF. Adobe's Thomas Phinney's (2006) thoughts on using the PUA for ornaments (Eliminate Private Use Encoding in Revised Fonts?):
About the only thing we’d use PUA for in new fonts would be ornaments or dingbats that really don’t have their own codepoints.
.. followed by a list of comments six times longer than his thoughts ;-)