I originally felt the same way as the OP does about this. The affiliation card should bear the emblem of the deck. Handwaving it away as a game mechanic that is immaterial from a flavor perspective doesn't work for me. If I see the Sith emblem on my opponent's side of the table, I should expect to get my ass kicked by dark side powers, see Vader's cruelty destroy my card advantage, etc. I should not expect to see a flood of TIE fighters and Star Destroyers, with a token appearance by Darth Vader rounding it all up as a "Sith" deck.
BUT. I think there is definitely a cost to not using your main affiliation's card. With it, you are guaranteed to always have the resource match necessary to play the majority of your cards. When you use an off-color affiliation card, you help ensure that you'll have the resource match, but by doing so you heavily skew the deck toward favoring a specific draw, at least when using the token 8x/2x affiliation makeup. As an example, I tried building a Vehicle-heavy Imperial Navy deck. I splashed in 2x Black Squadron Assault (the Sith objective with Vader's TIE Advanced), but found very quickly that the Sith affiliation card wasn't pulling its weight enough for the 4 cards out of 50 it helps bring into play. So instead I switched my affiliation to Imperial Navy, and if I draw one of the offending cards while I don't have a Black Squadron Assault in play to pay for it, I stomach it, and either discard it or use it for the edge battle.
TLDR; your affiliation choice should be dictated by the cards you want to play the most, not by how many resource matches you can provide for each affiliation.