The relaxation of random sampling methods may introduce bias. Having a sample not drawn randomly, a sample may not fully represent the population as a whole. Inferences may be misleading. The possibility of sampling bias has been a regular concern raised by more than one reader over the years of publishing aggregated survey results.
What if the same target population is sampled at regular intervals? Does repeated sampling improve confidence in the underlying method? At each interval (annually for the GWS), the survey collects a different sample of the target population. This is a repeated cross-sectional study. Repeated cross-sectional studies can be used for analyzing population changes over time even though the samples, themselves, are not identical.
Is the concern for sampling bias justified? Does a sample size hovering around 10,000 annually mitigate bias? Can survey analysis provide reasonable and useful results in the face of potential sampling bias?
What if the same target population is sampled at regular intervals? Does repeated sampling improve confidence in the underlying method? At each interval (annually for the GWS), the survey collects a different sample of the target population. This is a repeated cross-sectional study. Repeated cross-sectional studies can be used for analyzing population changes over time even though the samples, themselves, are not identical.
To gain confidence in the methods used and results presented, each survey ought to be drawing similar samples from the target population.
To assess this hypothesis, a few of the demographic attributes are selected for the years 2016-2022.
Prior Survey Response by Year
In a repeated survey, the earliest years ought to see fewer repeat respondents than in later years as the probability of having completed the survey increases. What is interesting is that the percentage of repeat respondents stabilizes at about 50% since 2020. About half of each sample is generated by respondents not participating in the survey before. Surprising result?
Years Spent Wargaming by Year
When capturing the number of years spent wargaming, besides 2016 and 2017, the percentage of respondents in each duration bin are relatively stable. Wargamers gaming 31 years or more remains locked into the upper 30% range.
Respondent Location by Year
When examining respondent location over time, again, the time series remains stable.
Age Group by Year
Examining the time series by age shows an interesting but not unexpected result. Given that prior analyses suggest that the population of wargamers is getting older in aggregate, samples drawn from a similar population ought to produce this tendency. The chart below exhibits the tendency of older cohorts increasing over time.
Primary Interest By Year
Finally, consider a respondent's primary interest in wargaming.
Over time, we see a shift from Fantasy/Sci-Fi to both Mixed, and Historicals. Given the tendency for non-historical wargamers to gravitate away from preferring purely non-historical wargaming as they age, the underlying samples seem drawn from similar populations.
While GWS samples may not be drawn randomly, the hypothesis that these data are drawn from similar populations across time should not be rejected outright.
Can results from several years of the GWS be relied upon for useful inferences on the population of wargamers? The answer is up to the individual. I know my position.
I am glad to see that my gaming, mixed, is a thing. But mixed up or down? Is imaginations the early nineteenth century mixed? Keep up the good work, Jonathan.
ReplyDeleteThank you for the vote of confidence in this work, Joe! ImagiNations certainly seems a "Mixed" genre to me.
DeleteWe're a self selecting group I guess as we have chosen to do it via a publication but I wouldn't have thought the information was less valid but I do drawings for a living and not data , so what do I know!
ReplyDeleteBest Iain
Good to see you have no reservations about using these survey results to gain insight into the hobby!
DeleteI should think that the GWS is fairly accurate. Enough new people are coming in and those that have previously participated previously may not have done so every year.
ReplyDeleteIs there a way to separate out the new participants from previous? The ones who've done 2 surveys from those who've done 7? Would it matter?
This may have been answered before but has the number of participants gone up each year?
Thanks for the effort you put into this analysis always interesting to read.
Glad you have confidence in the survey results, Dan, and always pleased to see the results interesting to readers! I agree that with a large sample size and a large percentage of turn-over each year, the survey captures a reasonable cross-section of wargamers.
DeleteSince the responses are all anonymized before I receive them, no way to tie respondents to past results.
In reality for good or bad or the bit in the middle, GWS is the most thorough and in fact the only ‘mainstream’ source of data capture that we have, so it only has itself to benchmark against.
ReplyDeleteFor us gamers there is some entertainment in it, with the trade being the only people who might find the data useful and even then, the established traders know their audience fairly well and the cash tills do all the number crunching they probably need (good year / bad year).
So after me working through all that, I have now spoilt the survey for myself by wondering what is the point! :-)
I agree, Norm, that there is no better cross-genre look into the hobby. While gamers may find the results offer some entertainment, for me, the survey puts anecdotal observations into context with actual data to either support or refute these perceptions.
DeleteIf The Trade only looks at past sales figures for guidance, they may be missing out on important developments and trends not captured in their own sales demand.
Sorry to see that you have ruined the survey for yourself. Hopefully, you continue to offer your thoughts on the survey for better or worse.
Of course, that was just a bit of tongue-in-cheek :-)
DeleteWhew!
DeleteAlways interesting Jonathan;thanks for posting. what do you think has grabbed you the most, in your years of doing this?
ReplyDeleteYou are welcome! Glad you continue to follow these survey results.
DeleteWhat has grabbed me the most? Well, this is a tough question to answer! I am routinely surprised and gain a deeper understanding of wargamer preferences with each analysis. One of the most surprising topics focused on a cluster analysis over a selection of wargamer responses wherein the attributes "clustered" in a most interesting manner. I may repeat this analysis to see if tendencies have changed in the last five years.
I don't really mind if the data is biased or not. It's not really being used for anything serious. The whole survey to me is just some fun social comparison. 😀
ReplyDeleteThat is the spirit, Stew!
DeleteAs you point out the bias disappears the more the sample size approaches the size of the population. So even if the sample size was only, say, five percent of the population then any potential bias through self-selection would be greatly minimised. Of course the trick is knowing how big the population is, and how to actually define the population in the first place. I like reading the results of these as it reminds me that the hobby is still going strong beyond my gaming group and regular blog acquaintances.
ReplyDeleteAs you state, Lawrence, increasing the sample size masks some of the bias and anomalies. Estimating the size of the wargaming population is a question i get asked regularly.
DeleteGlad you continue to enjoy seeing these post-survey analyses.
Thank you!
10'000 -ish participants is pretty good, but I feel that this survey doesn't reach and therefore isn't filled out by a lot of younger gamers, primarily the "Warhammer/40K" camp of which there are a large number of gamers who subscribe solely to GW games systems and who don't even consider that there are other gaming options out there, or readily sneer at even the mention of such.
ReplyDeleteStill, this survey is always interesting to both fill out and then to see your analysis thereafter Jon. The subtle changes to our hobby per the answers you illustrate with your keen mind and well written reports each year are often eye opening and force one to then compare to how one's own hobby has changed, or not, as we age.
Dai, you would be surprised at the number of Fantasy/Sci-Fi gamers completing the survey each year. Every year, three of the top four periods are WH40k, Science Fiction, and Fantasy. The survey reaches even the fantasy/sci-fi groups.
DeleteI appreciate your support and encouragement for my work on the survey!
Always something new to think about in those surveys! I'd have thought (without too much thought!) that having roughly 50% churn in the respondents each year that over time the surveys are capturing a reasonably representative sample of the hobby wargaming population. That's adding 5k of new respondents a year. 20,000 new responders in 4 years. How many miniature wargamers in the world do we think there are?
ReplyDeleteA circle representing the subset of wargamers that do respond, is moving around inside the circle representing the total wargaming population. Imagine the small circle starts at 12 o'clock inside the big circle, it's moving clockwise at roughly 1/2 its diameter each year, covering more and more ground. With reasonably consistent responses across a number of questions, over years, it would be a remarkable coincidence if the genuine responses of a changing subset of wargamers changed to align with previous years responses if the sample were not representative. Not sure I've expressed that clearly. :-)
Chris, I can visualize what you describe and you express yourself perfectly! Seems a logical extension that if half the sample is changing yet the results remain consistent that any sampling bias may be at a minimum.
DeleteI agree that a 50% turn off brings in a substantial chunk of the total population each year. I am always surprised that the churn is this high. I often wonder if respondents forget that they completed the survey before or if this is in fact actual churn of new respondents.
Very well described and perfectly clear! Thank you!
Maybe given the age of many of us, it’s to be expected that some of us forget we’ve done it!
DeleteChris.
Forgetfulnesses is one concern. Another concern is the tendency to answer any question incorrectly no matter the question.
DeleteIn the Human Factors world, data is often extrapolated from just say 7 respondents, for a product or service to be developed. We used to say that surely this cannot cover the population as a whole, but apparently some Norwegian chaps proved that it could. We would then point out election pollsters and their forecasts based upon say 2K of those that answered a call etc, that got the Brexit outcome completely wrong!
ReplyDeleteSo I think given the above, some 10K of responses per annum is as good a coverage of our hobby as you are likely to be able to get. As others have said, this size means that things should even out to give a pretty good overview without any bias unduly affecting the results.
One thing that has been touched upon many times is having the same questions each year, so the data can be reasonably compared to see if there are 'changes' showing in the results. It would be good to know what the 'industry' makes of this survey and if they use it at all in their planning for new products etc.
Steve, 10,000 respondents seems a valid sample to me too and, perhaps, the best we can expect. Having the same question asked year after year helps lends stability and benchmark to the results so that any changes maybe identified for what they are.
DeleteI know some in the industry at least glance at these surveys results because I occasionally get follow-up questions and requests for data queries.
Interesting post Jonathan. I enjoy the insights that arise from the survey, but accept that ultimately it is not "scientific". However, the number of respondants lend it credence... certainly until we have a sense of what the wargaming population actually is. Maybe the UN can conduct a world census for us!!!???
ReplyDeleteKeep up the great analytical work on this. It's enjoyed and informative.
Much appreciated, Richard! Very encouraging to see that you continue to enjoy the summaries and inferences falling out from these efforts.
DeleteMathematics is not my strong suit, but I do enjoy reading analyses of data - thank you for all you are doing for this.
ReplyDeleteIs there any group out there that is using this information for any purpose other than entertainment?
Glad you enjoy these continuing dives into the survey data, Greg!
DeleteAs for your question regarding actual use of these results, I can offer only anecdotal and secondhand confirmations. Nothing verified.
10,000 is a large number for a survey, so it should give us a reasonably accurate set of figures, but hey, what the hell do I know???
ReplyDeleteYou know plenty, Ray!
DeleteVery interesting survey and analysis Jonathan.
ReplyDeleteI wonder if there is a way to estimate the number of wargamers. Games Workshop last year's revenue was (according to google) $464 million.
So, assuming an average gamer spend of $500 around a million customers? Does the survey ask how much people spend so that an average can be worked out?
I must admit this is more than I thought there would be. But with 40K popping up more in media I guess there are more of us than I thought.
Thank you, Ben!
DeleteHobby spend is a question asked in this survey. From that, we could compute the average spend per respondent. Could that figure be used to back into the number of wargamers based upon GW sales? Good question! My hunch (based upon my own spending) is that $500/year is too low.
I agree, $500 is probably to low for the average wargamer but I am not sure if the average would exceed $1,000.
DeleteI had a look at the GW annual report and they said they sold to independent retailers " In 2021/22 we had 6,200 independent retailers (2021: 5,400) in 72 countries" There are a lot of wargame shops out there!
I will take a look at spend distribution from the survey. Personally, I am over the $1,000 mark easily.
DeleteThat is a lot of retail outlets! I wonder how much of those sales figures are locked away as store inventory on shelves and yet to transfer into gamers’ hands?
The strangest stat in all this to me is the fact that about 50% of respondents claim they have not done the survey previously. Of course, I don't think you have mentioned actual numbers, so it could be that all the original respondents 5 year ago have done the survey every year since, but an additional 50% have done the survey for the first time each subsequent year - so you start with 2000 respondents - next year there are 4000 (50% are new) next year there are 8000 (4000 are new) and so on ?? Otherwise, it really does not make sense to me - why would significant numbers do the survey only once then drop out the following year, to be replaced by a new batch of respondents?? Any insights or thoughts on that, Jon?
ReplyDeleteKeith, this is one of the stats that has been discussed for several years. The last three years an average of about 10,000 wargamers has completed the survey each year. The result does seem odd that nearly half of each year's respondents have not taken the survey in the past.
DeleteI can come up with a number of reasonable explanations why this trend continues. None can be verified, though. Perhaps, like Chris states above, the wargaming population is large and we are only capturing a small subset each year? Perhaps, respondents forget they complete the survey? Perhaps, the survey is completed only once and the respondent is done. Perhaps a new batch of wargamers discover the survey each year? I could go on and on.
Still, it is a puzzle to me to see this magnitude of new entries year after year.
I'm sure that the annual survey has some use/interest to a certain sort of chap but for me it's wholly pointless as I care not a jot what others do and would fully expect them to reciprocate. I'd rather enjoy my hobby and my chums, both physical and ethereal and let other do their thing. Perhaps I'm just an antisocial mard?😉
ReplyDeleteDavid, glad you weighed in on this topic! You are one of the fellas I had in mind when I wrote this piece.
DeleteWhether or not you pay attention to what others are doing within the hobby, I think these broad examinations are useful. Well, at least to some.
Isn't one facet of attending shows to see what others are doing? Do you but hobby magazines for a similar reason?
The need for some reasonably verifiable data about the hobby and its participants has been talked about for as long as I've been in the hobby; this is probably the closest we are going to get, with self selection and other biases acknowledged. $ 500 spend per year seems way low to me, but I suppose if you're a p[layer rather than a game host, that could be true especially for RPG's. Just dropped almost $100 on new brushes!
ReplyDeleteThanks for your comment, Peter! You and I know with certainty that our expenditures are much more than $500 per annum. New brushes? Excellent! I may have found brushes that hold up to W&N Series 7 for less cost. I am in still in the experimentation phase but these are tough brushes.
DeleteI got a couple of Raphael synthetic fiber ones to try as well as the usual 3-0 W&N Series 7. Awaiting delivery..
Delete