Jump to content
Sign in to follow this  
keyboardr

TLT by the numbers

Recommended Posts

 

How are you dealing with dice modification here?

 

His numbers are for no dice modification at all. Here's the graphical version.

 

 

TLT_with_0_attack_focus_vs_0_defense_foc

 

Here's a few more with action economies. This is just a teaser until I get around to publishing a more comprehensive TLT analysis after Worlds.  :)

 

 

 

 

 

 

......

 

 

 

 

 

 

 

 

Very informative graphs!  

 

Still boggling my mind that FFG hasn't hired you or someone else as a statistical consultant, or at least done something to get a solid statistical grasp on their game mechanics.  This is pure gold in terms of maintaining game balance, and a balanced game is a healthy game.  They're flushing money down the toilet every day they don't take advantage of tools like yours, if you ask me.

Share this post


Link to post
Share on other sites

How are you dealing with dice modification here?

 

His numbers are for no dice modification at all. Here's the graphical version.

 

 

TLT_with_0_attack_focus_vs_0_defense_foc

 

Here's a few more with action economies. This is just a teaser until I get around to publishing a more comprehensive TLT analysis after Worlds.  :)

 

 

 

 

 

 

......

 

 

Very informative graphs!  

 

Still boggling my mind that FFG hasn't hired you or someone else as a statistical consultant, or at least done something to get a solid statistical grasp on their game mechanics.  This is pure gold in terms of maintaining game balance, and a balanced game is a healthy game.  They're flushing money down the toilet every day they don't take advantage of tools like yours, if you ask me.

Who says they don't have math nerds working for them doing the exact same thing? Board games live and die by stats, math majors would be naturally drawn to game companies, I assume.

Share this post


Link to post
Share on other sites

 

 

 

How are you dealing with dice modification here?

 

His numbers are for no dice modification at all. Here's the graphical version.

 

 

 

Here's a few more with action economies. This is just a teaser until I get around to publishing a more comprehensive TLT analysis after Worlds.  :)

 

 

 

 

 

 

......

 

 

Very informative graphs!  

 

Still boggling my mind that FFG hasn't hired you or someone else as a statistical consultant, or at least done something to get a solid statistical grasp on their game mechanics.  This is pure gold in terms of maintaining game balance, and a balanced game is a healthy game.  They're flushing money down the toilet every day they don't take advantage of tools like yours, if you ask me.

Who says they don't have math nerds working for them doing the exact same thing? Board games live and die by stats, math majors would be naturally drawn to game companies, I assume.

 

 

I think it's not only fairly clear from a lot of the early releases that they didn't employ 'math nerds' to look over the game statistically, I think they've said they don't use any intensive statistical analysis to inform their design during interviews they've given.  

Share this post


Link to post
Share on other sites

Still boggling my mind that FFG hasn't hired you or someone else as a statistical consultant, or at least done something to get a solid statistical grasp on their game mechanics.  This is pure gold in terms of maintaining game balance, and a balanced game is a healthy game.  They're flushing money down the toilet every day they don't take advantage of tools like yours, if you ask me.

 

At the risk of going off-topic on the thread...

 

I turned down joining playtesting because it would require me to sign away all IP (MathWing included). If they want to hire me as a consultant in the future then the ball is in their court. Slightly longer explanation here: 

https://community.fantasyflightgames.com/topic/128417-mathwing-comprehensive-ship-jousting-values-and-more/page-10#entry1845818

 

 

Who says they don't have math nerds working for them doing the exact same thing? Board games live and die by stats, math majors would be naturally drawn to game companies, I assume.

 

That depends if said math nerds would be willing to take a significant pay cut. FFG is fairly well known for, as a general rule, having low salaries in the industry. Also, I'm sure FFG has some playtesters who do math at some level. I get the impression that the general business philosophy in this regard is along the lines of "why pay for it if we can get it for free?", and "it's good enough". Ultimately it's a business decision. We're still buying plastic toy spaceships.

 

An important distinction: there are different levels of competency and skillsets. Not just any "math guy" can walk in and solve their problems, otherwise it would have been solved already within their playtesting group. Game design and balance could have been significantly better out of the gate at wave 1 through the current wave now if they had the appropriate personnel and tools. Therefore I do not see evidence that they have math nerds with the necessary qualifications working for them.

Share this post


Link to post
Share on other sites

Therefore I do not see evidence that they have math nerds with the necessary qualifications working for them.

Not only do you need math nerds with a particular set of skills, but they have to be in a decision-making role.

ETA: That is, it's hard to say based on publicly available information exactly what kinds of evidence are being brought to bear at what points in the design process.

Edited by Vorpal Sword

Share this post


Link to post
Share on other sites

 

Therefore I do not see evidence that they have math nerds with the necessary qualifications working for them.

Not only do you need math nerds with a particular set of skills, but they have to be in a decision-making role.

 

 

Which still goes back to being a business decision! The nuclear option is to hire a Technical Balance Director that derives the mathematical models describing the games that the designers can invent, and gets veto power over design elements and costing. But that would command a large salary -- almost certainly well in excess of what Frank and Alex get paid combined. As long as we are still buying plastic spaceships, I doubt they will ever see a reason to spend more money on design and development.

Share this post


Link to post
Share on other sites

I was doing some statistics on TLT vs just a normal 3 dice or 4 dice attack and came back with some interesting numbers. It's long been thought that TLTs were disproportionately effective against low agility ships (e.g. Decimator). From what I'm seeing, it looks like the reverse is true and only just slightly. Essentially, the number of evade dice being thrown doesn't really change its damage output compared to a normal 3 dice attack.

 

Defense Dice | Mean Damage | Equivalent Attack Dice

      0      |     1.75    |     3.50

      1      |     1.47    |     3.62

      2      |     1.19    |     3.67

      3      |     0.93    |     3.67

      4      |     0.72    |     3.66

      5      |     0.54    |     3.64

      6      |     0.40    |     3.62

 

What does change, however, is the Hit % (i.e. the chance you'll do any damage at all).

 

Defense Dice |     Hit %   | Equivalent Attack Dice

      0      |    98.44%   |     4.75

      1      |    92.95%   |     4.78

      2      |    83.50%   |     4.74

      3      |    71.52%   |     4.65

      4      |    58.76%   |     4.55

      5      |    46.62%   |     4.44

      6      |    35.95%   |     4.33

 

TL;DR: unless your opponent only has one hull left or has more focus tokens than you (or once per turn effects), you're better off with a 4 dice attack than TLT. Also, your opponent's agility value doesn't change your TLT's advantage over a 3 dice attack.

 

Full data set available at https://docs.google.com/spreadsheets/d/1PgePFqFcd7puTcN9RRecyyCvJWWAhvIb1Z4Npwucf30/edit?usp=sharing

 

Do you have any std-dev on these numbers?  It'd be interesting to see how 3 dice compares to TLT with stddev.  The bigger the variance the more advantage to the underdog.

Share this post


Link to post
Share on other sites

 

 

Therefore I do not see evidence that they have math nerds with the necessary qualifications working for them.

Not only do you need math nerds with a particular set of skills, but they have to be in a decision-making role.

 

 

Which still goes back to being a business decision! The nuclear option is to hire a Technical Balance Director that derives the mathematical models describing the games that the designers can invent, and gets veto power over design elements and costing. But that would command a large salary -- almost certainly well in excess of what Frank and Alex get paid combined. As long as we are still buying plastic spaceships, I doubt they will ever see a reason to spend more money on design and development.

 

 

No offense, but that sounds like a terrible way to design a game. 

Share this post


Link to post
Share on other sites

 

 

 

Therefore I do not see evidence that they have math nerds with the necessary qualifications working for them.

Not only do you need math nerds with a particular set of skills, but they have to be in a decision-making role.

 

 

Which still goes back to being a business decision! The nuclear option is to hire a Technical Balance Director that derives the mathematical models describing the games that the designers can invent, and gets veto power over design elements and costing. But that would command a large salary -- almost certainly well in excess of what Frank and Alex get paid combined. As long as we are still buying plastic spaceships, I doubt they will ever see a reason to spend more money on design and development.

 

 

No offense, but that sounds like a terrible way to design a game. 

 

 

Uhhh, sounds like a great way to make a well-balanced game to me.  Mathematical balance is just a starting point, you know.  The point behind a lot of the Mathwing analysis would be to better inform game design decisions, not to design a game full of ships with exactly the same jousting efficiency. 

 

Good mathematical analysis could for example have averted many of the outright pricing errors we've seen over the years and led to a much better balanced game, with much better ship variety.  

Edited by Babaganoosh

Share this post


Link to post
Share on other sites

 

 

The nuclear option is to hire a Technical Balance Director that derives the mathematical models describing the games that the designers can invent, and gets veto power over design elements and costing. But that would command a large salary -- almost certainly well in excess of what Frank and Alex get paid combined. As long as we are still buying plastic spaceships, I doubt they will ever see a reason to spend more money on design and development.

 

No offense, but that sounds like a terrible way to design a game. 

 

 

Uhhh, sounds like a great way to make a well-balanced game to me.  Mathematical balance is just a starting point, you know.  The point behind a lot of the Mathwing analysis would be to better inform game design decisions, not to design a game full of ships with exactly the same jousting efficiency. 

 

Good mathematical analysis could for example have averted many of the outright pricing errors we've seen over the years and led to a much better balanced game, with much better ship variety.  

 

 

1) With this approach mathematical analysis isn't used to design the game, just balance it. This is an important distinction. This frees up the developers to explore the mechanical and creative aspects of the game, which is generally what makes a game "fun". The math is just there to make it balanced. Both are important.

 

2) Sithborg's comments in this context usually have an anti-MathWing perspective, so this at least is consistent with his previous posts.  :)

Share this post


Link to post
Share on other sites

 

2) Sithborg's comments in this context usually have an anti-MathWing perspective, so this at least is consistent with his previous posts.  :)

 

Yeah, but what's the confidence interval on that assessment? :D

 

I generally agree with Juggler.  Sure, we don't have a lot of strict knowledge for how things run internally, but we CAN look at the results as evidence, and doing so makes it VERY hard to think they had a competent mathematical model.  Heck, we know for a fact their rules didn't get that much design scrutiny, it's hard to believe there's a lot of deep math going on there.

 

I'm not sure I agree that the math guy on the team should have veto power - there are a lot of design and business reasons why you might want to create mathematically imbalanced units, from making big names better to just selling more of the expensive stuff.  You might also apply balance via other game mechanisms, such as limiting how many "better" units you can select.  But that said, gut feeling is wrong often enough that the design team NEEDS deeper math.  Gut feeling gave us the TIE Advanced, after all.

Share this post


Link to post
Share on other sites

I'm not sure I agree that the math guy on the team should have veto power - there are a lot of design and business reasons why you might want to create mathematically imbalanced units, from making big names better to just selling more of the expensive stuff. 

 

Well, certainly the CEO or someone else intermediary could obviously over-ride if they wanted a certain ship to intentionally be overpowered. Alex has already said that he doesn't decide which ships get made for each wave. Veto might be too strong a word or approach; in reality any good game designer like Alex would love to have those tools available and I don't see many situations where you would get the game designer and math guy on completely different pages. Ideally the design flow is:

 

  1. Designer: "I came up with X and want it to accomplish Y"
  2. Technical director: "X is a little bit off the mark from Y, it is slightly underpowered/overpowered."
  3. Designer: "Revision 2, how's this?"
  4. Technical director: "Looks good!"
  5. Playtesters get a hold of it and test. Results fed back to stage 3. Iterate as needed.
  6. Designer and technical director sign off on it.
  7. CEO: "Power up the factory!"
  8. ...
  9. Profit!
Edited by MajorJuggler

Share this post


Link to post
Share on other sites

Veto might be too strong a word or approach; in reality any good game designer like Alex would love to have those tools available and I don't see many situations where you would get the game designer and math guy on completely different pages.

I just think the math guy should be more in the business of providing information than necessarily making the decisions.  He should be able to provide extensive knowledge of how different decisions will affect balance, but the decisions concerning balance include a great many considerations of design outside the balance itself.

 

It's also very difficult, if not impossible, to completely contain every aspect of game design in the math.  What's the exact value of the Moldy Crow title, or a green two bank vs. a white?  Quantifying the impact of an extra attack die is straightforward, but quantifying an ability like Dash is much harder.

Share this post


Link to post
Share on other sites

Given some of the issues the game has had, I am unconvinced that having a playtester with such power would be a bad thing.

 

Exactly. They really seem to love their turrets. Low risk, high reward. 

Share this post


Link to post
Share on other sites

Here is the problem with MajorJuggler's idea. You essentially just gave a playtester more power than a designer. That is essentially what checking stuff against the math is, a different form of playtesting. 

This is very, very wrong.  There is no separation between testing - of ANY kind - and designing.  Just like with software, testing is an absolutely integral part of the development process.

 

Even then, you never separate the math from the design process.  Designers now will use math - they create a unit or upgrade and decide how much it's going to cost.  They do that via math, except they do it in their heads or by gut feel.  The difference is that they do it via BAD math, effectively saying "Well, 3 + 5 feels like 6, so we'll go with that."  Developer head-math is what gives us things like the TIE Advanced and Major Rhymer.

 

I am (as stated above) sympathetic to the idea that you shouldn't give the math guy all the power, but your aversion to even considering it is pretty far off base.

Edited by Buhallin

Share this post


Link to post
Share on other sites

Of course there should be a dialogue between design and playtesting. And just because I don't hold the math up as the holy grail of this game, doesn't mean I don't think there isn't a place for it. It just comes down to where the final call comes down to. Perhaps MajorJuggler just described it bad, but what he described sounded terrible to me. 

 

But I guess I'm just more Ameritrash than Euro. 

Share this post


Link to post
Share on other sites

I fear that if a game had to stick within the boundaries of a previously designed mathematical model, it would be closer to chess or go than to X-Wing, in the sense that the design space would be girdled by what the mathematical model can accurately put a metric on.

One good thing that the devs have kept on doing is that they have been thinking out of the box in respect to the design space, adding extra "dimensions" all the time to keep the game fresh. That is inherently something hostile to a mathematical model, because if you keep changing the game (and I am not talking about just churning out more ships that differ only on statline and action bar), the model stops faithfully representing the game.

If the math guy had a veto power, that would either make the game much less variated, or he would need to invalidate his model and recreate it for the new design space, every time; all this depending on the particular personality of this math guy (if he is more or less lazy to accept making constant changes to his model or ban any deep changes to keep his model pristine).

 

If MathWing were the driving model that the designers had to stick to, in this game we would have only jousters, no abilitiy synergies, no recovering of shields, no asteroids, etc. Anyway, MajorJugger cannot calculate the efficiencies of those unforseen combos, the different interactions between asteroid placement, blocking, stress and dial quality. Only gather statistics from tournaments and "gut feel". And one undesirable effects of the model is that its predictions affect the kind of lists people take to tournaments, that is precisely the source of statistical samples that feed the model. There is a feedback loop there that makes the model predictions affect themselves, although I must admit that it's not clear in which magnitude this is true.

 

How can a math model accurately determine the right point cost of, for example, Twin Laser Turret? Is 6 points unbalanced? If so, is it both unbalanced in an HWK? In a Y-Wing? And a K-Wing?

What determines the cost of an upgrade? The upgrade own merits, or also the ships it can be attached to? And vice versa, how do you assign a point cost to a ship? By looking at the ship's direct attributes (statline, action bar, pilot skill, upgrade bar variety, dial), or also considering the cost of the upgrades and synergy combos it indirectly has access to?

And will that assesment stay true for future additions to the game, or will it be either rendered useless or become a burden to all later development?

 

Could a math expert overcome these challenges in order to create a perfect math representation of all the possible interactions of the game, during every single iteration of the game (and iteration here doesn't mean Wave, but all the single development steps) so that it allows us to predict the perfect and effective costing of every pilot and upgrade. It sounds extremely hard, but perhaps it is possible.

Now the question is, would this actually be the most efficient way to balance the game, really? 

I sincerely doubt it.

Share this post


Link to post
Share on other sites

Of course there should be a dialogue between design and playtesting. And just because I don't hold the math up as the holy grail of this game, doesn't mean I don't think there isn't a place for it. It just comes down to where the final call comes down to. Perhaps MajorJuggler just described it bad, but what he described sounded terrible to me. 

 

This all comes down to organizational decision making.  First, how do you incorporate your math analyst in your structure?  You have grouped them with playtest, which I will take to mean QC.  This can absolutely be the case, where the analyst can verify that costs are in balance.  On the other hand, the analyst could straight up assign costs, putting them into a design role with playtest performing the QC functions.  In either case you need to decide if QC is 'advisory' or 'approval'.  Are they merely feedback to design or do they have veto power?  Furthermore, It is conceivable that design/creative and QC report to a single person ("product head" or whatever) with final say, or just have QC report to design.  The xwing lead designers seem very reasonable and self aware, so this last option seems like what they have and it seem to mostly work pretty well.

 

 

But I guess I'm just more Ameritrash than Euro. 

 

I'm curious what you are trying to say here.

Share this post


Link to post
Share on other sites

 

But I guess I'm just more Ameritrash than Euro. 

 

I'm curious what you are trying to say here.

 

 

Ameritrash and Eurogame are two terms to describe the nature of a board game.

Ameritrash games put higher importance on theme over mechanics. If the game is about StarWars starfighters, then it is mandatory that it feels right like in the movies, with proper art and game pieces that reinforce that feeling. Balance and mechanical perfection are second tier. And there is usually frequent use of chance in the shape of dice or card decks.

Eurogames put higher importance on the game mechanics over the theme. The gameplay elegance is paramount, and it is quite abstract, balanced and perhaps, deterministic. The theme doesn't matter or matters very little. In a StarWars starfighter eurogame, the gameplay could be something like optimizing the way the player gathers several triangle shaped pieces while balancing the expenditure of square shaped pieces that move in particular patterns over a grid. There are no dice rolls or chance of any kind, and a good player can't simply be beaten by a player of lower skill. The same game could have its theme exchanged with whatever else (instead of StarWars starfighters, now it is farmers in feudal Japan harvesting rice) and the game would be 100% unaffected.

Edited by Azrapse

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...