I haven't seen anybody doing something like that, but always heard rumors and speculations about it being possible or not.
I would like to brainstorm about it, though at the start I only have indications for difficulties to offer.
So first a problem that may arise making something a 'generation through points' tool in general. It can be exploited.
Whenever you offer a system that let's you spend points to choose and improve abilites, there will always be the PG-fraction that instantly has its calculators readied to find the optimal solution for their character. There is this urge to make the things efficient and well thought out- -I know it all to well.
So for a concrete example, if you want to make the jump from 3 to 2 DP for the block ability cost 2 of the class-generation points, everyone would feel themselves pushed to just choose that for either dodge or block - whatever they want to go with later. But looking into the books tells you that most combat classes have both at cost 2 DP, so that you choose after you elected your class. This is even more of a problem for the cheapening of secondary skills.
You can generalise this problem to the point where class generating players would do the hell choosing just fluff relevant skill or whatever boni, because that would always draw on their capability as fighters/mages dotdot. E.g.: Why would I have +10 style per level as an acrobatic warrior, if I could just get another +10 acrobatics, which leads to incredible uncounterable combat boni later with the right build? Same with leadership and persuation, or any cheapening of skill branches.
So the original classes kind of serve as tool to force some more 'inefficient' but stylish and realistic-real-human fluff onto the players and the GM.
Making it a paying system would draw that away or in return punish those who actually invest points into secondary abilities (/or any other kind of 'unfocused' build), because they will always stand back a bit against the full center hardliners.
This is allready a huge problem to consider.
Another one, in combination with the first, is the missconception that a open class generating system will inherently lead to new and interesting, but through the equality of the generations points, well balanced classes. As I said first, people tend to optimise their character build if there are no forces restraining or punishing them for doing so. So most classes will come out as (overstated examples) "Evokation-magic Ninja" with all points in summoning, zeon, hide/stealth and rest in battle, but no general magic, no banish, bind, or control. Another one: "KI Weaponsmaster" 10 MK per level too low? Class change to technician for 60 DP later? Meh! How about a combo class that allows class change for 40DP without cost, where you also get rid of the "Feats of Strength" per level bonus and the cheap lifepoints multiple (that nobody uses anyway) and get increased MK, maybe KI for 1 DP allready (as preperation for later) in return?
Optimisation like that is breaking for sure. You will see players justifiying their classes with very strange artifical logic ("he has only trained how to make poisons his whole life, so he gets an +40 poisons per level bonus"). Convenient logic, but still not real at all.
But this is honestly a problem of any point paying system that exists so far, so you can expect that there have been different approaches to counter this (very natural) behaviour.
One for example is to make every further investment into an allready improved group of skills/or whatever even more expensive than the last time, so that the costs grow exponentially. You will never get the same amount of ware for the same price, which leads to some pondering if the focus of +10 here might be worth it, if you could get an overall spread +10 in five different skills for the same cost, and wouldn't that be more handy and efficient now?
I think that is kind of intelligent, because it doesn't actually restrain the player/GM in anything. Strategy is stays open, but you have to sacrifice something if you still go powerstriving to the limit. It is more a kind of politics instead of a solid wall. Encouragement instead of prohibition.
This only appears suitable for more longterm investment systems however, so it would be more a thing for the DP investment. Mini payment systems, where every option does only have 2 or 3 steps don't really work with it, but the principle is a good thing.
Another example are the conditions of the world. If your campaign consists of die hard fights out of every tree shadow, of course players will optimise for fight. If you are only playing detective stories, players may even focus on their non-combat abilites only and crazily overhype them.
There is a game named Shadowrun, where in the 4th edition the generating and skilling system is totally free, but still doesn't come out unbalanced. I think one of the main reasons for that is that their world really requires any kind of skill, and on the same time low skill levels aren't so heavily punished (while higher scores don't give you that much of an advantage on most/standard occasions). This setup lead to an overall very realistic character building for any player, where every skill was used and even tertiary knowledge skills would be sometimes purchased for fluff … aand because it may actually be useful.
Totally class and limitless, yet balanced - I was impressed. So, if you can manage to construct the right mixture of pressure in your rpg world and have a system that let's everybody achieve at least something in everything they try, you can actually have that naturally without any kind of politics and barrier. Of course, for Anima, this would be not yours to decide.
The last example is what Anima actually does with its clear spending limits in the DP growth. I don't think this is a particularly clever system, because I hate hard restrains for their absolute decision barrier and slightly predetermined development, but it is quite a valid solution too. Since you don't have anything to do with the 40 extra DP per level, you could as well invest them into something personality building aside (though Anima kind of messed up that there is no individual limit on every skill, so it may be more efficient to focus and have one 'superskill' instead of being an all around Mr. Useless). So at least some character building gets forced on you (a good thing here).
I would recommend this solution for a small payment system like this, though I am not sure. You could determine that a prowler class could spend around 18 points (60%) for secondarys while a fighter class only gets 9 or 12 or whatever.
There could be type related limits on single skills, MK or Magic, or everything could just get increasingly more expensive (like with the attribut point 10 which costs double with the payment method).
What I am sure of is just, that there is some management and detouring needed if you want to prevent strangeness overload. Creators will still seek optimal solutions, but you could establish some valid balance between a 'concentrated' and a diversed build with the right politics, so ideally nobody is getting punished.
Taking the pressure of am I?