Pages

Friday, September 29, 2023

On The Reliabilty of Data II

The 2023 edition of the Wargames, Soldiers, and Strategy Magazines' Great Wargaming Survey is a wrap and the results are in.  Total responses this year were 9,282 completed surveys.  That count is down from 2022 but 9,282 respondents still represent a solid sample size.  Several new questions were added into the 2023 survey.  There are still questions that have yet to see any analysis, some new twists to earlier analysis, and revisiting some old questions with a look using fresh data.  Many topics to explore in the coming twelve months.

First, I start off the 2023 analysis cycle with a look at a topic first examined in May of 2023.  That topic is the On the Reliability of Data.  Why examine this topic again so soon after the last investigation?  For one, the survey counts are down from 2022.  Second, the age question was asked differently.  Third, I wondered if discussing the conundrum of first-time respondents would move the needle at all.  Finally, we can assess the stability of the data with one more year added into the mix.

Prior Survey Response by Year
As identified in last year's analysis, the percent of respondents having taken a prior survey remained in the low 50% range.  After mentioning this result, I wondered if surfacing this tendency would affect results.  Well, impossible to say whether mentioning this attribute had any impact upon the results but those having taken the survey before jumped to 66%.       
Years Spent Wargaming (Duration)
What about the number of years spent in the wargaming hobby?  Did that remain consistent to prior years?  No!  The group claiming to have 31 or more years in the hobby spiked to nearly 45%.  Perhaps the hobby is graying?   
Respondent Location by Year
Year after year, the location of respondents seems stable.  We see roughly the same mix of locations each year. 
Age Group by Year
Like last year's analysis, the tendency of age cohorts to increase over time continues.  In tandem with the Duration chart above, 2023 saw a jump in the number of respondents in these older age cohorts.  Is this phenomenon due solely to an aging population of could something else be driving these results?
One frustration from recent years is that compliance with answering the Age question has been slipping.  In 2019, Age was changed from an age range to a discrete age to accommodate the psychometric study (one day, I will dig into those data!).  Age was a required field.  In 2020, Age became an optional field but remained as discrete.  2021 and 2022 kept that same format but non-compliance increased to the point that in 2022 nearly 1 in 5 respondents failed to answer the age question.  Wanting to reverse this trend, I suggested returning to age cohorts rather than asking for discrete age.  The result?  Non-compliance almost entirely vanished.  Could these non-compliant respondents have resided primarily in the older age cohorts?
Primary Interest by Year
Finally, we revisit primary interest in wargaming.  The trend seen earlier of a shift from Fantasy/Sci-Fi to both Mixed and Historicals continues.  If responses are drawn from a representative sample then the notion that non-historical wargamers shift away from preferring purely non-historical wargaming as they age holds.  Is the survey catering to and drawing from fewer non-historical wargamers over time or is the trend illustrated here accurate?  Other analyses may be able to confirm.  
Even with some shifts in the data, I still maintain that the hypothesis that these data are drawn from similar populations, across time, cannot be rejected outright.

Many questions.  Some answers.  Your comments encouraged.

36 comments:

  1. If the shift towards on older response holds true then it would make sense that you also see a move towards historical wargaming which has pretty much always been the case I think.

    Christopher

    ReplyDelete
    Replies
    1. Do fantasy/sci-fi gamers generally gravitate toward historicals as they age? That is a good question. Data suggests that.

      Delete
  2. Yup a first time respondent here who started out as a historical gamer drifted into sci fi and is very slowly drifting back into historical. I think the results that you’ve shown support my own perception of the hobby overall. I wonder how or if the interest in historical / sci fi / fantasy differ geographically. Are Europeans more into historical gaming for instance than their American cousins?

    ReplyDelete
    Replies
    1. Thanks JBM! Doubly thanks for completing the survey for the first time.

      Past analyses show that UK wargamers tend to be older and more historical than their European continental counterparts. UK gamers hold a slight edge over Americans with respect to historical gaming. Both topics for another time.

      Delete
  3. That 45% spike is a headline figure!

    Would combining the past 3 years info and taking an average give a better / more stable representation of wargaming behaviours? And then set that against the average of the previous 3 years.

    I know the survey itself is not fully consistent, but in some key areas it will be.

    ReplyDelete
    Replies
    1. As I hinted at in the post, I wonder if that Duration spike at the upper end was from the Old Cohort deciding to answer the Age question?

      As for stability, the numbers seem stable enough to me that averaging across years for these questions seems unnecessary. You are correct that the questions do change over time. Do you believe that there has been a regime shift in attitudes from 2018-2020 to 2021-2023?

      Delete
    2. Well two things, 3 year slots seem like a fair chunk of time to make comparisons by smoothing out but not expecting things to have moved radically and secondly, I also wondered whether we have a pre and post Covid effect?

      Delete
    3. Pre-COVID and Post-COVID effects may be something to look into as well as the COVID Effect. A quick scan of the data suggests that remote gaming is decreasing.

      Delete
  4. The data shows we are an aging community. I'm one of our younger club members and I've gamed for 30+ years

    ReplyDelete
    Replies
    1. I guess we are not bringing up new recruits at a fast enough pace.

      Delete
  5. I expect that consistent questioning leads to better data reliability? A topic for future surveys.

    Many of us, who were in their 50's in 2016, have crossed into the next category. I'm not quite there yet but consistent survey-takers will lend a hand into that trend also. But, does that indicate a lack of participation by younger age categories or simply the aging process at play?

    In my mind, part of the shift towards mixed and historical only collections is economic. Prices for the leading sci-fi/fantasy figures and games are shocking at times. Historical products are going up as well of course, but not nearly as fast. Plus, there is no 'planned obsolescence' in historicals, unless one switches sizes!

    Thanks again for continuing your series of analyses of these data!

    ReplyDelete
    Replies
    1. Greg, good to see that you continue to enjoy and appreciate the analysis of these survey results.

      One would think repeatedly asking the same question year after year might provide better data. The curiosity, here, is that (before 2023) roughly 50% of respondents are first time participants in every survey year.

      You make an interesting point and observation about genre and figure cost. Having never bought non-historicals, besides a handful of Star Wars Legion plastics, I have no idea the cost of non-historicals. The survey definitely points to a shift toward 'Mixed' primary interests from both ends of the spectrum.

      Delete
  6. I had to laugh Jon - all these people, happy to admit they play with toy soldiers, but seemingly reticent to reveal their age! I wonder what would happen if you added a gender question to the mix?! It would actually be quite interesting, but I would suspect there would only be a handful of ladies completing the survey - personally, I have only ever come across one female wargamer.
    It still seems odd to me that, over time, the number of participants has not gradually increased, and the percentage of first-time respondents dropped significantly. I first did the survey about three years ago, probably thanks to a hint/link on your site but, having done it, why would I not do it again every subsequent year? Perhaps if you do not follow this blog, you would be asking yourself "why bother" - this is the only place I ever see the data analysed or put to any use that is of interest to gamers, as opposed to sellers of gaming figures.

    ReplyDelete
    Replies
    1. Some people are very reluctant to give out PII. I am happy to see that reintroducing age cohorts resolved the issue of non-compliance; for now anyway. Gender is asked on the survey but I have not reported on that since it is a male-dominated hobby.

      "Why bother" is a good question to ask. Perhaps the uptick in repeat respondents and Old Gamers is due to readying this blog? One can wish...

      I do republish these posts on WSS' website so maybe a few read my posts there? There are rarely any comments on WSS website so I never really know.

      Thank you for continuing to complete the survey and read about the results here. Much appreciated.

      Delete
  7. You mentioned the total number of respondents had dropped from last year Jonathan. Was it a significant drop, or have the numbers always been around the 9,000 mark?

    ReplyDelete
    Replies
    1. Total counts have hovered in the 9,000 to 11,000 range. 2022 saw 10,892 respondents to 2023's 9,282. That is down about 15% from 2022.

      Delete
  8. I can see the "graying" of the hobby even when I look in the mirror. I'm a relative newcomer to the hobby - and am getting close to two decades. The majority of the folks I game with are well into 30-40 years, or more. Funny how time flies when passionate with a hobby.

    ReplyDelete
  9. Thanks for going through the data Jonathan.
    I had a thought about the aging of the hobby. Could it be because many of those in their 30-40s do not have the time to game due to family commitments. This certainly applied to my situation.

    ReplyDelete
    Replies
    1. You are welcome!
      I agree that gaming time is likely at a premium for those in the prime of raising families and climbing corporate ladders. The survey results illustrate consistently that the 31-40, 41-50, and 51-60 groups maintain a roughly equal percentage of the respondents at about 25%. I recall that spending is lower in the 30-40 age group too so that supports your experience.

      Delete
  10. Yep, old guys rule. :-) I think the advantage of the hobby, the steadfastness of the tin (or plastic) soldier comes into play as you age. I have figures that are quite old and still quite useful .

    ReplyDelete
    Replies
    1. “I have figures that are quite old and still quite useful.”

      Just like old wargamers!

      Delete
  11. Maybe only 50% remembered that they’d answered the survey before. It is an aging hobby. 😉
    I assumed that historicals were more popular with the older cohorts because they had always been into historicals and that, not because they switched later in life. Just that when they started out sci-fi and fantasy were not that big.
    Chr/Nundanket

    ReplyDelete
    Replies
    1. Good points! Aging hobby leads to forgetfulness and there was no fantasy/sci-fi 50 years ago. There is some evidence that some wargamers do make a switch to historicals later in life.

      Delete
  12. The age of players and length of gaming time is always interesting JF. My son and his friends ( all mid 20s) only game fantasy/sci-fi and the games are what I would consider large scale skirmish. My friends and I who have been gaming for 35+ years ( 50s-60s-70 year olds) have very large armies with big units. The young'uns actually look at all our figures and think it's not worth trying to get into something that requires 1000s of figures- although I'd be happy to give him a couple of armies. But the cost of a single figure for a single giant demon or carnivorous leviathan is usual the cost of a couple of unpainted cuirassier regiments!

    ReplyDelete
    Replies
    1. Thanks for your feedback, John! Interesting to see that younger gamers think amassing large armies is not worth the effort. Do they say why?

      Delete
    2. Funnily enough JF, their initial response was 'too expensive' . But when I pointed out that my unpainted Viking army cost actually less than the full force of my son's unpainted Plague Marines...there was a lot of umming and ahhhring...and after so more discussion it was about the perception of requirement. I painted nearly 250 Vikings- my sons force was less than 70 figures - although several were very big. It's a perception that to game like my friends and I do ( the old guys) - you need so many figures. We have introduced them to The Men WHo Would be Kings and some of the other skirmish games and this has grabbed their attention as not needing 'big armies'.

      Delete
    3. You are doing good work in converting the next generation.

      Delete
  13. Interesting stuff there Jon. The numbers completing the survey for the first time each year is interesting, which makes me wonder how they heard about the survey and why previous gamers do not return to complete it each year. As others have mentioned there is the issue of 'why bother?' when you've done it before. Over the years I've been doing it my responses have barely changed and generally only in relation to how the questions are put.

    The 45% jump is a surprise and in a way links in to the above, as the jump could be affected by first time responders or maybe people have just jumped into a new age category/time spent gaming due to being a year older etc?

    As for the greying of the hobby, at the recent Colours show, it was certainly full of middle aged and older 'white' men, with a smattering of younger men, a few women (gamers or suffering wives?) and some children. So if this continues then naturally the charts will move and we will see a decrease in historicals, a rise in mixed and SciFi cum Fantasy?

    ReplyDelete
    Replies
    1. "Why bother" is a good question is ask and address. Perhaps the incentives offered are enough to cross that hurdle? Maybe not?

      On the big jump in gaming duration, I find it hard to believe that that many respondents passed over to the "31+ years wargaming group" naturally in one year. My bet is on a generally older population than in prior years. While the hobby may be graying, this is too large of a jump to support that hypothesis.

      Thanks for your insights, Steve!

      Delete
    2. The potential prizes are certainly an incentive and the free voucher helps too. Personally I would still do it even without the aforementioned, if only to show that there are other options than 28mmm etc!

      Delete
    3. If only we could vote more than once...

      Delete
  14. I find the march of mixed gaming to domination interesting. There was a time, at least in the UK, when there was a lot of snobbery about sci-fi/fantasy gaming from historical gamers and inverse snobbery back. Perhaps we now see a more relaxed attitude to swapping codes?
    Anthony Clipsom

    ReplyDelete
    Replies
    1. Thanks for your comments, Anthony. I think that snobbery still exists only it is found at the extremes of this spectrum. I know deniers in both camps but those camps appear to be shrinking.

      Delete