Addressing Data Reliability
Questions about data collection methods and reliability often arise, so I frequently provide select meta analyses to address these concerns. The GWS relies on voluntary responses from anyone who completes the survey. Without random sampling, responses are based upon convenience sampling. In convenience sampling, respondents must know about the survey, have access to it, and be interested in wargaming, which naturally limits the dataset to this group. Each annual survey includes all valid submissions in a cross-sectional study, capturing data from many participants at a single point in time.
Non-random sampling can introduce bias, meaning the sample might not fully represent the broader wargaming population. Any significant bias could potentially lead to misleading inferences.
While the possibility of sampling bias has been a recurring concern from readers over the years, is this concern justified? Does collecting a large sample size per year help mitigate this risk of bias introduction? How large of a sample is required? Can reasonable insights still be drawn despite potential issues? What about sampling the same target population repeatedly but drawing different samples? Annual survey analysis like the GWS rely on repeated cross-sectional studies, allowing analysis of changes over time even with varying samples. To build confidence, each survey should draw similar samples from the population.
A quick initial review reveals one immediate surprise: participation dropped significantly over the last three years. After 9,282 responses in 2023, the 2024 survey garnered only 5,995 responses. In 2025, total responses dropped yet again to 5,005 responses. Reasons for the continued fall off in responses are unclear but reports suggest that only 78% of respondents beginning the survey completed it. Many of these dropouts were due to failure to click "Submit" once finished. Could this participation reduction skew upcoming analyses for the 2025 reporting cycle? Speculation aside, data might offer clues on bias. As the first in the 2025 reporting series, this post examines data reliability and compares select demographics from 2023, 2024, and 2025.
Prior Survey Indicator
With a decrease in responses in both 2024 and 2025, the percent of those repeat survey respondents falls from about two-thirds in 2023 to roughly 60% in 2024 and 2025. Still, overall percentages are similar across all three years.
Location
The mix of respondents' home countries remains relatively stable from 2023 to 2025. Fewer respondents originated in USA/Canada and Australia/New Zealand in 2025 to the benefit of UK/Ireland respondents but no significant change in the overall distribution.
Age Group
When examining age demographics, a shift in age cohort emerges. In 2025, the 61+ age groups saw a marked uptick in responses. Outside of a heavier weighting to the older cohorts, 2023 and 2025 distributions look similar. For age distributions, 2024 looks like the anomaly. As past surveys demonstrate, older cohorts tend more toward historical wargaming. Will the survey confirm this inference?
Education
Highest education level attained remains consistent across all three years. No significant difference with respect to education.
Primary Interest
With an age group shift to the 61+ age cohorts as noted above in Age Groups, we might expect a corresponding shift toward historical wargaming as primary interest. These data suggest that may be the case. This is exactly the result that the breakdown of primary interest shows. Fantasy/Sci-Fi primary interest grouping lost ground in 2025 to both Historical and Mixed classifications. Historicals gained about four percentage points from 2024 to 2025. Fantasy/Sci-Fi lost five-and-a half percentage points from 2024 to 2025. Why this shift and what effect (if any) will this hold for following analyses?
Duration
Finally, when examining responses to how long a respondent has been wargaming, the results are not surprising given the age and primary interest shifts seen above. The percentage of gamers having been in the hobby for 31 years or more increased by 7.5 percentage points from 2024 to 2025. Notice that 2025 results follow the distribution pattern seen in 2023 more so than the pattern seen in 2024. Again, was 2024 an anomaly?
Insights on Reliability
Despite a non-scientific approach to survey sampling, the GWS demonstrates again that sampling tends to remain consistent year after year. I return to the topic of data reliability frequently to help mitigate concerns about bias (and usefulness) of these surveys. Even with a move toward older gamers, cross-sectional, repeated sampling, in this case, seems to produce a stable sample from which to investigate inferences. With fewer responses again this year, will established trends hold? Some may and some may not. Watch as results and analysis of the 2025 survey unfold.
There are a few new questions this year and I look forward to exploring these results. I would enjoy seeing your thoughts on why fewer gamers completed the survey and/or the demographic attributes of the GWS Class of 2025.
I've no clue why kess gamers filled in the forms. Are people getting bored with it, haven't got the time? Or just don't care enough to fill it in?
ReplyDeleteI'd say these 3 things could be part of the cause.
Perhaps boredom. Perhaps like many gamers, attention span is short, and gamers are easily distracted by something new?
DeleteRay has a point. I could understand that the novelty wore off.
ReplyDeleteLooks like the respondents are aging, maybe the interest or presence is lower due to that, we will die off eventually. Not today, mind you!
Yeah, I am sure the novelty has worn off for many. Just look at the responses here! The surprise to me was the drop in the Fantasy/Sci-Fi genre response. We are not dead yet!
DeleteLike Ray, I don't really know why the numbers would drop so dramatically; as the historically inclined side of gaming gets older and older, I guess a percentage are no longer with us each succeeding year, but I am sure the grim reaper does not explain a drop of around 4000 in two years!
ReplyDeleteI still find it strange (and have commented on this before) that almost 50% of people who fill in the survey any given year, have not done so previously, The fact that despite the constant addition of new respondents, the total number continues to drop, would indicate more and more people who have completed the survey once, never bother doing it again - and I do find that rather odd. It's a niche hobby, there are not that many of us really, and the survey takes about 5 minutes to do - so why wouldn't you do it every year, once you know about it? Perhaps the answer is, a lot of people are not aware of the survey and don't remember to go looking for it again the next year - could the gradual decline in the number of blogs explain this - or do people who do their hobby "social media" via Facebook etc still promote the survey and urge readers to complete it - that's not a rhetorical question, I don't use FB and have no idea what people put on it re wargaming etc!
Thanks for your insightful feedback, Keith! To touch on your point about the prior survey question, anecdotally, some view the survey as a "one and done" proposition. That is, the survey only needs to be completed one time rather than an annual event. Not sure how you change that mindset. Survey visibility is likely a contributing issue. While I receive an email from WSS announcing the survey, many probably do not. Like you, I have no visibility to other social media means of seeing any announcements.
DeleteI wouldn't have known when the survey was on each year if it wasn't for you flagging it up Jonathan and I would guess that there are quite a few respondents who are in the same boat. So perhaps you are directly responsible for a few dozen of us remembering to fill out the survey, and if you didn't remind us then there would be a drop off. I wonder if that could be the case with others who get their cue elsewhere.
ReplyDeleteThanks, Lawrence. My reach is quite small in a very small niche! Survey promotion is likely a major contributing factor.
DeleteInteresting stuff Jon as always. I guess for me the reason for a drop and potentially my own involvement would be no real sense of the purpose of the survey.the first few are interesting as it perhaps answers some questions about the hobby, but year on year the change seems fairly limited as the hobby is stable. Despite your analysis and best efforts the variations don’t for the average wargamer mean a great deal, hence I would suggest people will lose interest.
ReplyDeleteI wonder whether in 2023 we were still in the last throws of Covid, so gamers were getting their wargaming fixes online rather than FtF, so maybe the survey was one of those? As others have said, after a few years and essentially answering the same questions, has the interest and gloss worn off? I also wonder whether having prizes for taking part in the survey affect uptake or not? IIRC this year there are prizes, but last year I think not...?
ReplyDeleteOn a slightly related issue, I was watching an interview with John Stallard on Youtube recently, and he essentially said that he doesn't bother with market research etc, taking the view that if "you build it, they will come". I can't remember if you've touched on this before or not Jon, but what is the data used for at the end of the day and by whom?