The description of Spells (page 10) says that the dice "may" remain, which is conditional. It may remain, and it may not.
The description of the red dice (page 9) is not conditional. The die "is" removed from the dice pool at the end of the active players turn.
So, during your turn you can play a card that lets you roll the red die, and play a spell. After rolling, you can then put the red die on the spell to use in a later test in the same turn, but at the end of the turn the red die is removed and the spell goes away.
That would be my way of interpreting the rules. It makes the red and yellow die useable for only the active player, which seems to be the intention.
However, if you don't like that, how about this house rule. You can store the red or yellow dice on a spell, but in order to use their result on a later players turn, the active player has to use the appropriate card to add the die to their dice pool.
So player A would play a Unique Item, and a Spell Card, then rolls, and puts the Red Die on the Spell. Completes or fails the tasks and play passes to player B. In order for player B to use the Red Die result in their turn, they also have to play a Unique Item card, putting the Red Die into their Die Pool.
But I would say that the rule book isn't clear on the interaction between red and yellow dice and Spells.