So you can Stoneskin the target of the original attack, but not the other defenders affected by the attack (not attacked directly). I wouldn't play it that way, but a very literal reading of the rules seems to support it.
My reasons are based more on how it plays than RAW. Since the Spiritspeaker has to be in range, pay a fatigue, & exhaust the card, it's not like Stoneskin is going to be madly abused on every attack. I've seen the opinion that the Spiritspeaker is the weakest class, so limiting it at all seems like a bad idea.