The Routledge informs us that we don't Duchen on Yom Tov which falls on Shabbat (Yom Kippur apart.) The Hertz and the new-ish Sacks/Koren Siddur contain the same instruction. This is from the Sacks/Koren Siddur
Interestingly the newer Koren Machzor (at least the volume for RH) uses different language which fudges the issue.
'If Kohanim say' suggests that in some circumstances Kohanim don't say (presumably Shabbat, for a Cohen who is in the room should bless), but those circumstances are not spelled out either on the page (for either day of RH) or in the Halachic Guide at the back of the Machzor. I suspect, and I'm asking, but haven't yet had a confirmation, the intent is to say subtlely, that the tradition not to Duchen on Yom Tov/Shabbat is not to be taken too seriously - without any fear of embarrassing communities or Rabbis who don't.
Some of the sources on the subject
SA OH 128:44 Rem'a
Our custom in all lands is that we don’t raise our palms unless it is Yom Tov since at that time we are in the Happiness of Yom Tov and “The glad of heart shall bless [Tov lev hu yivarech]” and this is unlike other days, even unlike the Shabbatot of the year, when we are troubled thinking about earning a living and avoiding Malachah.
Mishna Brurah ad loc.
We don’t raise our palms unless it is Yom Tov - whether the festival falls on weekday or Shabbat but there are places where it is customary not to raise palms when Yom Tov falls on Shabbat but this custom is not at all essential. (Ein minhag zeh ikkar clal) as many commentators have written.
Mem Ayin [not sure who this is] has written that there is a terrible [garua] custom that a few places do not raise palms when Shabbat falls on Yom Tov. It's not right and I don't know the reason for the thing. It's possible that the rabbis of the world [I'm not really familiar with this phrase, I think it is meant pejoratively, i.e. they might be ordained, but they don't know what they are talking about] said that there is a problem of making petitions on Shabbat, but this doesn't apply on Yom Tov …
Moshe Feinstein IM 3:18
Behold, this person has a Synagogue where the custom is that Cohanim don't lift up their palms even on Yom Kippur which falls on Shabbat. This is an error and we should reproach them. For even in those places where the practice is not to lift up palms on Yom Tov which falls on Shabbat [we still do this on Yom Kippur] … and moreover even the custom which exists in some places not to lift palms on Yom Tov which falls on Shabbat, that's not a powerful custom, even in those places. In recent times there is virtually no place where there is this terrible [garua] custom, as is explained in the Arukh HaShulchan 128:64 [which explains that] the custom that used to apply in ancient times, of not lifting hands when Yom Tov fell on Shabbat, has been annulled in our times for the custom has no reason or sense… And specifically in these years of pain [tzoros] we need blessing, and the mercy of the Heavens is good. We should command the Cohanim to lift their palms and those rabbis of the world, it's better that they should not [stop the lifting of palms] especially on Yom Kippur which falls on Shabbat.
And One Last Thing
A friend, with an inside scoop into mid-to-late C20th Anglo Jewry shared that the 'real' reason for non-Shabbat Duchening in Minhag Anglia is 'lest the Cohanim should carry their slippers to Shul.'
Ahhh, Minhag Anglia :-)
Actually, it makes a whole of things fall into place, including why we do Duchen on Shabbat Yom Kippur (think of the Shofar).