It's a real problem, for sure. I think a program could help a lot with keeping track of what pilots get experience and what pilots are available for missions, especially when list-building. For the tabletop, during battles I think an analog tracker like a pilot/ship card works best, since it is always visible.
For representation during play, I'm thinking just pieces of sticky-note over the PS of the pilot's card.
In general, the main issues I am worried about are:
Problem: One player wins the first few battles and becomes unstoppable as the game goes on due to pilot and ship progression mechanics.
Pilot Death and Ship destruction keep progression manageable.
Pilots cannot fly consecutive missions (should slow down progression)
Increase cost for ships and pilots as they improve.
Slow down rates of progression
Yes. As mentioned before, I'm limiting it to 1 PS per mission survived. The game as normally is a fairly deadly, and PS is not the greatest advantage in the game. But, there are other types of progression that should be kept in check.
At least in the outset, I'm going to have the strategic commanders of the players (they represent wing commanders, not admirals), as well as the circumstances coming out of previous missions, direct what missions get played, and try to make sure that those missions have at least some balance. So, in that respect, the effects of the progression will be kept in check by the fact that the underdog strategic commander (ie. the program) will 'know' that he's the underdog and refuse to fight in a battle in which that underdog status will be exploited (too much).
To pick up on your prohibition of fighting consecutive battles, I'm modeling that by way of having ships that were damaged in a previous battle spend some 'time out' in the repair bay. However, in a precarious position, the player can choose to put them into the battle, knowing that they go into it with some damage cards already on the table.
Problem: Too many limitations, caveats, and things to keep track of make the game not fun for some players (see warhammer 40k).
I thought that WH40K was a fairly popular game. I enjoyed it for a bit, though I grew bored with it because it didn't have a campaign aspect. When I tried to introduce it, my friends were simply not that interested. I'm hoping that a wider circle of friends/players will stave off that same problem this time.
Also, I plan to have much of the 'keeping track' problem done by the computer.
3: Balancing player choice with plot progression:
Problem: Forcing players to play specific missions can be problematic, especially if the missions are imperfectly balanced (almost definitely the case). But a campaign is not worth much if it doesn't build towards something.
"Free-form" campaign mechanic allows players to advance the campaign at their pace.
The way I want to do this, after the first stages of the campaign, is to make it possible for the players to make some operational-level decisions. However, they're going to first have to earn the trust of their (automated) strategic commanders. Also, I want to have a more free-form opportunity available. Let's say that they feel they consider their strategic commanders mistaken. They might appeal to their commander's commander (ie. me) with an idea of their own, and I'll see if that's implementable. Of course, their direct commander might not be pleased with them for going over their head. So, essentially, that's a modulated version of the "free-form" that you're talking about.
4: Basic Progression balance:
Problem: Do increases in PS and ship upgrades affect imperials and rebels equally? Are the many ship upgrades available in the ship progression tables fair? Answer to both: probably not.
Possible fix: Change rate of requisition and experience gain for rebels and imperials.
In my mind, it's clear that the rebels face vastly different constraints than the imperials do. I plan on modeling that by having the rebels face scarce resources, and the need to create the logistical supply train to meet those resource needs. They'll have to get YT-1300s and GR-75s to ferry materiel from home base or to purchase stuff on the black market.
For the Imperials, their constraint is that they have to deal with a rigid order of battle. Initially, all they get is TIE/ln fighters. They have to earn the right - and have the need - to use different starfighters and bombers.