I wouldn't make changes to the frac feature. Rather, I would order the frac feature lookups before the numeral style selection features (which is normally the case anyway), and then simply add mapping from the default numerator and denominator glyphs to the variant forms in the numeral style selection feature lookups. So, for example, you would add mappings of /one.numr/ to /one.numr.osf/ in the onum feature.

PS. This sort of question is better asked in the Build forum.

Ok. I did it. John, it came to my mind thanks to your tips :)

It turned out to be very simple. I just added old style glyphs (numerators, denominators, fraction and slash) and made some changes to the frac feature:

lookup FRAC {
sub [slash slash.osf] by [fraction fraction.osf] ;
} FRAC;
lookup HOCH {
sub [zero one two three four five six seven eight nine] by [zero.numr one.numr two.numr three.numr four.numr five.numr six.numr seven.numr eight.numr nine.numr];
} HOCH;
lookup TIEF {
sub [fraction fraction.osf] [zero.numr one.numr two.numr three.numr four.numr five.numr six.numr seven.numr eight.numr nine.numr]' by [zero.dnom one.dnom two.dnom three.dnom four.dnom five.dnom six.dnom seven.dnom eight.dnom nine.dnom];
sub [zero.dnom one.dnom two.dnom three.dnom four.dnom five.dnom six.dnom seven.dnom eight.dnom nine.dnom] [zero.numr one.numr two.numr three.numr four.numr five.numr six.numr seven.numr eight.numr nine.numr]' by [zero.dnom one.dnom two.dnom three.dnom four.dnom five.dnom six.dnom seven.dnom eight.dnom nine.dnom];
} TIEF;

Here’s what I did in the Modern Suite.

First, I made a set of quite small superiors and inferiors (because I have never liked these—especially superiors for use in references—to be too obtrusive).
Then I made a set of numerators and denominators that were larger, for use as the default fraction style, with the default figures, which are lining.

To change to the smaller size of fraction style, I included a change from “numerator/denominator” style to “superior/inferior” style in the Oldstyle figures feature—shown below in bold. (Making sure the Fraction feature came before the Oldstyle features in sequence.)

There is no need to include this change in the Fraction feature, as John has already mentioned.

feature onum {
# Oldstyle Numerals
sub @figs by @figs_onum_tnum;
sub @figs_lnum_pnum by @figs_onum_pnum;
sub @figs_numr by @figs_sups;
sub @figs_dnom by @figs_sinf;
} onum;


Nick,
You are right, there is no need to change frac feature. I've removed changes and it works well.
In fact, my scenario is similar to yours except that I've created new glyphs for osf fractions instead of using inferiors and superiors.
I work in Glyphs app so all glyphs with .osf suffix are automatically included in onum feature, so that approach was more elegant for me.

Thank You.