Addressing Data Reliability
Questions about data collection methods and reliability often arise, so I frequently provide select meta analyses to address these concerns. The GWS relies on voluntary responses from anyone who completes the survey. Without random sampling, responses are based upon convenience sampling. In convenience sampling, respondents must know about the survey, have access to it, and be interested in wargaming, which naturally limits the dataset to this group. Each annual survey includes all valid submissions in a cross-sectional study, capturing data from many participants at a single point in time.
Non-random sampling can introduce bias, meaning the sample might not fully represent the broader wargaming population. Any significant bias could potentially lead to misleading inferences.
While the possibility of sampling bias has been a recurring concern from readers over the years, is this concern justified? Does collecting a large sample size per year help mitigate this risk of bias introduction? How large of a sample is required? Can reasonable insights still be drawn despite potential issues? What about sampling the same target population repeatedly but drawing different samples? Annual survey analysis like the GWS rely on repeated cross-sectional studies, allowing analysis of changes over time even with varying samples. To build confidence, each survey should draw similar samples from the population.
A quick initial review reveals one immediate surprise: participation dropped significantly over the last three years. After 9,282 responses in 2023, the 2024 survey garnered only 5,995 responses. In 2025, total responses dropped yet again to 5,005 responses. Reasons for the continued fall off in responses are unclear but reports suggest that only 78% of respondents beginning the survey completed it. Many of these dropouts were due to failure to click "Submit" once finished. Could this participation reduction skew upcoming analyses for the 2025 reporting cycle? Speculation aside, data might offer clues on bias. As the first in the 2025 reporting series, this post examines data reliability and compares select demographics from 2023, 2024, and 2025.
Prior Survey Indicator
With a decrease in responses in both 2024 and 2025, the percent of those repeat survey respondents falls from about two-thirds in 2023 to roughly 60% in 2024 and 2025. Still, overall percentages are similar across all three years.
Location
The mix of respondents' home countries remains relatively stable from 2023 to 2025. Fewer respondents originated in USA/Canada and Australia/New Zealand in 2025 to the benefit of UK/Ireland respondents but no significant change in the overall distribution.
Age Group
When examining age demographics, a shift in age cohort emerges. In 2025, the 61+ age groups saw a marked uptick in responses. Outside of a heavier weighting to the older cohorts, 2023 and 2025 distributions look similar. For age distributions, 2024 looks like the anomaly. As past surveys demonstrate, older cohorts tend more toward historical wargaming. Will the survey confirm this inference?
Education
Highest education level attained remains consistent across all three years. No significant difference with respect to education.
Primary Interest
With an age group shift to the 61+ age cohorts as noted above in Age Groups, we might expect a corresponding shift toward historical wargaming as primary interest. These data suggest that may be the case. This is exactly the result that the breakdown of primary interest shows. Fantasy/Sci-Fi primary interest grouping lost ground in 2025 to both Historical and Mixed classifications. Historicals gained about four percentage points from 2024 to 2025. Fantasy/Sci-Fi lost five-and-a half percentage points from 2024 to 2025. Why this shift and what effect (if any) will this hold for following analyses?
Duration
Finally, when examining responses to how long a respondent has been wargaming, the results are not surprising given the age and primary interest shifts seen above. The percentage of gamers having been in the hobby for 31 years or more increased by 7.5 percentage points from 2024 to 2025. Notice that 2025 results follow the distribution pattern seen in 2023 more so than the pattern seen in 2024. Again, was 2024 an anomaly?
Insights on Reliability
Despite a non-scientific approach to survey sampling, the GWS demonstrates again that sampling tends to remain consistent year after year. I return to the topic of data reliability frequently to help mitigate concerns about bias (and usefulness) of these surveys. Even with a move toward older gamers, cross-sectional, repeated sampling, in this case, seems to produce a stable sample from which to investigate inferences. With fewer responses again this year, will established trends hold? Some may and some may not. Watch as results and analysis of the 2025 survey unfold.
There are a few new questions this year and I look forward to exploring these results. I would enjoy seeing your thoughts on why fewer gamers completed the survey and/or the demographic attributes of the GWS Class of 2025.
I've no clue why kess gamers filled in the forms. Are people getting bored with it, haven't got the time? Or just don't care enough to fill it in?
ReplyDeleteI'd say these 3 things could be part of the cause.
Perhaps boredom. Perhaps like many gamers, attention span is short, and gamers are easily distracted by something new?
DeleteAs it was addressing issues I wasn't that interested in (and ignores things I am), I decided this year it wasn't a worthwhile way of spending my time.
DeleteWell, OK. Suggest some questions you would like to see.
DeleteIt is more the case that it's run by a magazine that is principally interested in wargaming with model soldiers. It is therefore not polling (and probably not interested in) wargamers who don't use model soldiers.
DeleteLast year featured a few questions on computer and hex-and-counter Wargames.
DeleteRay has a point. I could understand that the novelty wore off.
ReplyDeleteLooks like the respondents are aging, maybe the interest or presence is lower due to that, we will die off eventually. Not today, mind you!
Yeah, I am sure the novelty has worn off for many. Just look at the responses here! The surprise to me was the drop in the Fantasy/Sci-Fi genre response. We are not dead yet!
DeleteLike Ray, I don't really know why the numbers would drop so dramatically; as the historically inclined side of gaming gets older and older, I guess a percentage are no longer with us each succeeding year, but I am sure the grim reaper does not explain a drop of around 4000 in two years!
ReplyDeleteI still find it strange (and have commented on this before) that almost 50% of people who fill in the survey any given year, have not done so previously, The fact that despite the constant addition of new respondents, the total number continues to drop, would indicate more and more people who have completed the survey once, never bother doing it again - and I do find that rather odd. It's a niche hobby, there are not that many of us really, and the survey takes about 5 minutes to do - so why wouldn't you do it every year, once you know about it? Perhaps the answer is, a lot of people are not aware of the survey and don't remember to go looking for it again the next year - could the gradual decline in the number of blogs explain this - or do people who do their hobby "social media" via Facebook etc still promote the survey and urge readers to complete it - that's not a rhetorical question, I don't use FB and have no idea what people put on it re wargaming etc!
Thanks for your insightful feedback, Keith! To touch on your point about the prior survey question, anecdotally, some view the survey as a "one and done" proposition. That is, the survey only needs to be completed one time rather than an annual event. Not sure how you change that mindset. Survey visibility is likely a contributing issue. While I receive an email from WSS announcing the survey, many probably do not. Like you, I have no visibility to other social media means of seeing any announcements.
DeleteI didn't get an email from then this year either, only finding out about it via your Blog Jon.
DeleteThat is weird. Did you provide your email address when taking last year's survey? Are you on their mailing list? I receive updates quite frequently on the content of all of their wargaming/history publications.
DeleteYes and yes! I seem to remember when you posted on your Blog that the survey was live, that others hadn't received an email either?
DeleteI didn't get an email either, I didn't even realise the survey was live until well after it had ended and only because you mentioned it on the blog here. I don't recall it being mentioned on any of the forums I hang out on either. So unsurprisingly I didn't fill it in.
DeleteMartin, not surprising that you did not fill it out! Perhaps next year?
DeleteI wouldn't have known when the survey was on each year if it wasn't for you flagging it up Jonathan and I would guess that there are quite a few respondents who are in the same boat. So perhaps you are directly responsible for a few dozen of us remembering to fill out the survey, and if you didn't remind us then there would be a drop off. I wonder if that could be the case with others who get their cue elsewhere.
ReplyDeleteThanks, Lawrence. My reach is quite small in a very small niche! Survey promotion is likely a major contributing factor.
DeleteInteresting stuff Jon as always. I guess for me the reason for a drop and potentially my own involvement would be no real sense of the purpose of the survey.the first few are interesting as it perhaps answers some questions about the hobby, but year on year the change seems fairly limited as the hobby is stable. Despite your analysis and best efforts the variations don’t for the average wargamer mean a great deal, hence I would suggest people will lose interest.
ReplyDeleteAppreciate for insights, Matt! Loss of interest and lack of purpose certainly seem reasonable explanations. I really doubt that more than a few actually find and read any post-survey analyses that I publish. I still find the results interesting and new questions help keep the survey "fresh" for me. Besides, I get a chance to continue exercise my data analysis skills.
DeleteI wonder whether in 2023 we were still in the last throws of Covid, so gamers were getting their wargaming fixes online rather than FtF, so maybe the survey was one of those? As others have said, after a few years and essentially answering the same questions, has the interest and gloss worn off? I also wonder whether having prizes for taking part in the survey affect uptake or not? IIRC this year there are prizes, but last year I think not...?
ReplyDeleteOn a slightly related issue, I was watching an interview with John Stallard on Youtube recently, and he essentially said that he doesn't bother with market research etc, taking the view that if "you build it, they will come". I can't remember if you've touched on this before or not Jon, but what is the data used for at the end of the day and by whom?
The transition from the confines and isolation brought about by COVID to a return to normalcy certainly could play a role in response. Today's potential audience is not as captive as in past years. In my recollections, prizes seemed more numerous than in today's offerings.
DeleteStallard's comments are a luxury for many manufacturers. Perhaps that is the case with this hobby? We know that the "Oh, Shiny" factor distracts many. Just looking at the number of unfinished projects and changes in direction suggest that Stallard may be correct. Are these data used? My experience is that some of it is. Some questions are a direct response from some manufacturers. Past questions on BREXIT and foreign purchases tendencies stand out to me.
I'm pretty sure that Pendraken's top ranges are WWII (by a country mile), then Napoleonics and finally Ancients, the latter admittedly covers quite a broad spectrum for sure. I'm sure the same is true for most manufacturers. Other ranges are either loss leaders or just about viable to produce given the costs of masters etc. I do remember Pendraken giving a broad overview of how many sales it would take to break even on a new range, and it was into the 1,000's IIRC!
DeletePendraken's top ranges mirror the results we see year after year on the historical side. That is (1) WWII (2) Napoleonics (3) Ancients.
DeleteThanks Jon, the drop in respondents is essentially a drop in participation and perhaps that is just reflective of society post Covid.
ReplyDeleteI sat behind a young woman on the train a couple of days ago and she was using the journey to deal with her daily social media obligations! I was transfixed, I have never seen anyone scroll through screens and apps so quickly, literally a second or two at each page. If something caught her eye, she might hover for 3 - 4 seconds and was off again. I was struck by how pedestrian our blogs are and it begs the question whether most peoples transaction with their screen is hugely more superficial than I could have imagined and where does creating work sit amongst that!
The migration from blog to FaceBook might be an echo of that same issue.
Every time I do something, the doctor, dentist, hospital, upgrade my phone package, visit financial services, buy coffee from a big chain, the phone pings and there is a survey …. How Did We Do!
An automatic reflex is to just delete the message and perhaps the WSS survey has just caught that cold …. Delete!
Screens are increasing grabbing (stealing!) our time and against such volume, perhaps we are coping by being selective with just integrating with stuff that matters uniquely to each of us personally.
Personally and honestly, I have no idea what the WSS data is used for and how important it is ….. whats it for? Having regard for that, it does take time to fill in and I can never remember what I wrote anyway. I only complete it because I know you like it :-)
Our kids are constantly skimming through content when on their phones, so I'm not sure how much they actually take in from what they see. It all seems to be clickbait to me:(. So for them the WS&S survey would seem like a 3 hour exam;)!
DeleteI'm glad I'm not alone in forgetting some of my responses to the questions Norm!
Thank you, Norm! I witness much the same with respect to social media when I am out. People ought to look up when walking in public! I see so many near misses and last-minute dodges out of the way. Technology can steal the small enjoyments of life.
DeleteMy recollection is that the survey began as a way to keep wargamers engaged during the gaming slowdown during Dog Days of August. I am happy to see that you continue to fill out the survey, if only for me! Studying the survey results has given me a better grasp of the hobby on a number of issues. I still find them interesting, but I may be alone...
If our blogs are pedestrian and read only superficially, where does that leave us, the still dedicated few? I guess we each must answer the question if the effort if worth it. With comments turned off on both your blog and website, reader interaction is not sought. With no feedback or interactions, perhaps we ought to consider taking our blogs private?
Thanks, Steve! I can (usually) still remember my answers to the survey.
DeleteI filled it in this year after missing the last couple, I found it rather strange about the drop off of contributors but think Ray may have something in his theory. Anyway a good start for this years survey be interested in what follows as you dissect it further.
ReplyDeleteThanks, Donnie! Stay tuned for (hopefully) some interesting analyses.
DeleteInteresting comments from yourself and all the others Jon.
ReplyDeleteInterested that many respondents answer the questions but fail to hit submit. But I wonder why 2024 is so different to 2023 in this respect. Did the ‘look and feel’ of the survey change so that many people failed to see/realise they had to click one more time to submit? If the survey was essentially the same you’d expect the same ‘failure’ rate.
Chris/Nundanket
Chris, it is an odd situation to see the non-completion rate go up. Even more interesting is the 2025 demographics look more like 2023 than 2024. The survey tool does change from year to year, but I do not recall having any issue with finding and hitting "Submit." The look and feel may have been different. While I did not see any reference to this situation in the 2024 survey, I wonder if 2024 saw a similar non-submission?
DeleteI filled the survey in, but only because I got the link from you post, Jonathan. I don't subscribe to any magazines anymore. So, would never get the prompt for the survey from WSS directly. I'd be interested to see what the trend is for WSS subscriptions and if there is any correlation with fewer respondents to their survey. Probably not, but who knows?
ReplyDeleteI wonder if a little of the fall off is because some struggle to see the relevance of the survey. If the perceived purpose is to find out about the state of the hobby, but then see no development or change based on that knowledge. Why repeat the process year on year? I'm just stirring the pot with this!
Richard, you may stir the pot all you wish! Subscription info would be interesting to see. Few want to divulge those numbers, though.
DeleteInteresting posts and comments Jonathan.
ReplyDeleteLike Norm I struggle to remember what I have written and only complete it because I enjoy your analysis. Otherwise I would not see any value in it for myself.
Maybe people complete the survey to the end and don't submit just to see what the questions are?
Thanks, Ben! Going through all of the questions just to see the questions without submitting is an interesting take.
DeleteYIKES! I'm going to hit the last age column next year!
ReplyDeleteCoincidentally (or rather, because of the long years) there were quite a few questions where I struggled to find any valid answer for myself. In other words, for the first time I felt more out of touch. I wonder if others in a similar situation decided to drop out?
Which questions gave you pause? Maybe rewording would help? Suggestions?
DeleteThe drop in response rate is perplexing. With an aging group of respondents it is also odd that a reasonable churn in who is responding is also occurring which suggests an issue with gaining engagement from the available pool of potential respondents. I do wonder if the GW crowd are responding in the same proportions as they represent in the hobby population. Maybe promoting the survey via clubs, shows/conventions and hobby retailers (assuming that doesn’t already occur) might change the response rate and responder profiles.
ReplyDeleteThe drop in response counts from 2023 to 2024 is quite puzzling. Much of the drop off from 2024 to 2025 could be non-completion. The GW group is likely responding at a much smaller rate than they represent of the total miniature Wargaming market. Seems many did not see the survey promotion. We need to figure out how to broaden that reach. Thanks for your feedback!
DeleteAn interesting post, and the resulting discussion has touched on most points I might have considered commenting upon.
ReplyDeleteOn the subject of participation levels, one consideration might be that the ‘GW crowd’, who were mentioned, are shielded to a degree by the exclusive policies of their rules and models provider…’GW or nothing’ seems to be their motto.
On fantasy-playing contributors in general, WSS mag and other Karwansaray output being historical in nature, the readership and player pool being communicated to will, to a degree, be ‘of that ilk’ , perhaps?
Thanks, Martin! As I often say, the commentary is often the best part.
DeleteI am sure that the massive GW crowd is under-represented in this survey but GW-type fantasy/sci-fi periods always bubble to the top periods lists. Since the “Mixed” grouping is always the largest percentage by far, perhaps many of these gamers are buried within the group and they are more diverse than imagined?
oh my, me and my doctorate are in the top 6%. tut tut tut and other highbrow expressions. 🤣
ReplyDeleteat last, I am the elite in something! You plebians!
I find it interesting that people forget to hit the submit button. It's rather obvious.
Yes, you are at the head of THIS class!
Deletehmmm.. I confess that a big part of my motivation for completing the survey was to get the discount voucher and put it towards a subscription to WSS magazine!
ReplyDeleteI do wonder if people simply think 'I did that before, no need to repeat'. I don't really see much evidence of WSS using the results ( though perhaps they might be influencing choices of subject matter etc behind the scenes) should they think of publishing a summary of survey findings in the magazine?
I suppose the 'non-completion' rate might simply be due to people getting bored part-way thorough, or interrupted and never getting back to it..
Getting the voucher is reason enough to take the survey. I have yet to use my voucher. Probably buy a back issue or two.
DeleteI know, anecdotally, that some think the survey is one and done so your reasoning makes sense. Much of the non-completion in this year's survey was specifically attributed to failure to hit "Submit." after having completed the survey. Weird, I know.
Survey results have been used to answer hobby questions. You may not see much evidence of it, but results do gain some attention.