While a colony can easily start with a 4 PF, remember that an Industrial Colony requires 6-10 PF just to found (The average startup for even a research colony is 5 PF). While the PF scales as a colony grows, you also have to perform at least one endeavor per size growth or your colony stats suffer and can eventually cause your colony to fall apart. By the time you have performed 9 lesser endeavors, and the greater endeavor (founding the colony) without any PF gain you have lost out on 12+ PF (in addition to the startup costs). This makes a colony a break even deal at best.
As for harvesting resources for PF, remember that you have to convince the colony each time to give you the PF instead of applying the resource towards its growth. Failure to do so causes a loss of 1d5 Complacency. That can kill your colony quite quickly. You would have to convince the colony to strip mine its own world for your personal proffit. While possible from time to time, I doubt your GM would allow you to do so continually. I would probably allow it once a year with good roleplaying. The Rogue Trader could probably couch it in terms of his Tithe (provided he supports the colony well) and good Imperial citizens would accept it. Also note that many of the added structures that raise your colony stats, and therefore its PF and utility, also require lesser endeavors. You could litteraly spend an small campaign just performing the 20-30 endeavors to build up a colony to a major city or hive level. While that will interest some players (like my own), some of your players are likely to find it more boring than they would like. You can perform these as background endeavors, but that risks a higher degree of failure and perhaps losing more PF as well.
I am not saying that colonies are not worth it. Canny players will center much of their endeavors around the colony and its system resources to reap the benifits of their colony bonuses. This will allow them to gain greater PF from the bonus AP often available through colony structures. Also note that you can settle multiple colonies in a system, or even on a world. The largest colony size is a Hive, and many worlds in the Imperium have several Hives. My players settled an Agricultural Colony on the world with fertile plains and a verdant ecosystem to ensure food supplies for their expansion in the system. They plan on settling mining and industrial colonies on a nearby moon, and eventually the Explorator plans on developing a small forge world there (free of the stifiling dogma of the more repressive Calaxian forges). The group intends for the system to be self sufficient, as least as far as is possible. It may also be desired to keep your colonies small, especially on less hospitable worlds used primarilly for mining or research. How you do that is between your GM and you.