DYNOMIGHT ABOUT RSS SUBSTACK

358 games of puzzle storm

358 games of puzzle storm

Aug 2022

A year ago, I started playing puzzle storm. This is a short game where you try to rapidly solve as many chess puzzles as you can in a few minutes.

puzzle-storm-small

Now, I don’t play—or even really like—real chess. It feels too much like mortal combat, and I get a similar feeling of reward by doing math and (hypothetically) contributing something to the universe.

But I found puzzle storm to be an ideal distraction when I need a break at work: It’s fun, but it’s also short and sort of frustrating, so I’m rarely tempted to play more than a few games.

I got very low scores when I started, but improved rapidly. After a couple of weeks—inspired by Scott Alexander’s experiment with WordTwist—I thought there might be something to learn about skill acquisition or fluctuations in cognitive performance over time, so I started recording all the scores in a big file that looks like this:

oct 2
930 10
933 13
1521 13
1525 10
oct 5
1620 11
1624 8
1933 12
...

That was a year ago. Since then I’ve played 358 games, for a total time-wastage of around 18 hours.

Just to get this out of the way: I’m not good at puzzle storm. Even after a year, my top score ever is a somewhat-embarrassing 19. Someone who is actually good looks like this.

Hypothesis #1: I got better over time

Here are my scores on all the games, with a small jitter added. To make the trend easier to see, I added a moving average of 15 games.

scores-ordered

This hypothesis is not supported.

Hypothesis #2: I got better when I played more often

My play was extremely irregular. What if we use the date and time for each game? Because moving averages are weird when you have irregular sampling, here I used a smoothed (loess) curve.

scores-days

It looks like I did improve at first, but I regressed horribly when I stopped playing for 5 months and regressed a bit when I stopped for one month later on. So I think this hypothesis is true.

Hypothesis #3: My brain is OK

There’s something else I should maybe mention about the above chart:

scores-days

Physically, Covid wasn’t too bad for me, but I just couldn’t think—I’d have conversations and be astonished by the gibberish coming out of my mouth. (Looking back, this is when I decided that the world urgently needed to know everything about ethylene.)

So that was a lesson on the fragility of the human condition. After I recovered, I seemed to slowly regain my ability to think. But I was a little neurotic: If I wasn’t back to full capacity, would I notice? My desire to measure this is what led me to start playing puzzle storm again.

I’m pretty sure I’m fine, but this data isn’t very conclusive either way. Did I get worse during the gap because of Covid, or just because my skill atrophied? Did I plateau later on because of a skill ceiling, because I played less often, or because Covid cooked my brain? It’s hard to be sure. I really wish I had continued playing at a steady pace the whole time.

Hypothesis #4: I’m smarter in the morning

We can also look at averages at different times of day. Here’s how many games I played for each one hour period during the day: (The 8am bin shows the number of games between 8am and 9am.)

num-played-by-hour

And here’s my average performance for each time.

by-hour

The orange lines show 90% confidence intervals, just to stick it to the 95% interval mafia.

So: Zero evidence that I’m any smarter or dumber at any different time of day.

There are, however, two major caveats:

  1. I played puzzle storm whenever I felt like it and had time. This is not random. Subjectively, I don’t like to play when I’m tired, and when I’m at my best, I don’t get distracted as much. It’s possible I am smarter/dumber at different times of day, but this selection effect killed off the signal.
  2. I’m not sure how much this game measures intelligence. This is probably obvious to people who play chess, but I was surprised that almost all my improvement seemed to come from unconscious pattern recognition, not “thinking”. After playing a while, you mostly start to “feel” the promising moves. Still, there’s some conscious processing to filter the moves the unconscious mind suggests.

Hypothesis #5: I’m smarter on certain days of the week

Here’s the total number of games I played on each day:

num-played-by-day

And here’s my average performance on each:

by-day

I miiigghhtt be slightly better during the middle of the week. But there’s no conclusive evidence.

Even if this were true, it could be a “recent practice effect”: Since I played more during the middle of the week, on those days I had more recent practice.

Discussion

To summarize:

  • I’m not good at puzzle storm.
  • I improved a lot in the first couple of weeks before I started tracking, but not much in the following year.
  • Practice seems to help, but skill decays quickly with time off.
  • There’s no evidence I’m smarter at any time or on any day of the week, though it’s hard to be sure with this kind of data.

In retrospect, I’m not sure puzzle storm is ideal for this purpose. There’s a ton of luck: Some puzzles are easier than others, and this effect is compounded by the way the game gives you extra time for streaks of correct answers. I also had a lot of trouble with misclicks. All this variance makes it hard to detect the true signal.

If I were to do this again, I’d choose a less random game. There are lots of cognitive tests out there, but they aren’t fun, which is a non-starter for me.

What’s really needed is to play a game on a regular schedule. The ideal would be a game that is:

  • Short
  • Fun (this disqualifies WordTwist)
  • But not too fun (not addictive)
  • Gives a high-resolution output (not just won/lost)
  • Has a skill component
  • Has a general cognitive performance component
  • Is low variance

Does such a game exist?

Comments at reddit, substack.

new dynomight every thursday
except when not

(or try substack or rss) ×
Why doesn't advice work?

(or at least work better)

In ancient India, there was a long-running feud between the Pandavas and the Kauravas. Duryodhana, leader of the Kauravas, planned a huge war to end things forever. Krishna warned that this would lead to the total destruction of both sides...

Obvious travel advice

Not entirely about food

1. Mindset matters more than where you go. 2. Who you go with matters more than where you go. 3. After seeing each other for a few months, many new couples take a short trip, which often ends in an...

Thoughts on seed oil

Don't get distracted.

A friend has spent the last three years hounding me about seed oils. Every time I thought I was safe, he’d wait a couple months and renew his attack: “When are you going to write about seed oils?”

You’re Invited to a Colonoscopy!

thoughts on tubes

Colonoscopies are the first-line method for preventing colorectal cancer in America —and almost nowhere else. But do they work? We finally have a comprehensive trial, but it’s left gastroenterologists with more questions than answers.

The midwit home

Less automation and less agony

Reading a book one night, you decide to turn on the lights. And suddenly it’s obvious. Hauling your body across the room just to flip a switch is absurd. So you decide to get smart lights. Two hours later, your...

My heuristics for interacting with humans

On game trees and reasonableness

I don’t sense that I’m viewed as particularly skilled at human interaction. Still, some poor fools sometimes ask me for advice, and I find myself repeating the same little speech. For context, in life as a human you will often...

Taxonomy of procrastination

There's a little accountant named Jim that lives in my head

Nobody gets everything they want in life. That's OK. If everyone was a sportscaster-rockstar-scientist-model-author-influencer-billionaire, we still wouldn't be happy because everyone else would be too busy to be impressed. But still, it's a little sad when you don't at least...

Buy more copies

Sometimes weirdness does pay

I’ve always wanted to believe that you could get enormous advantage in life just by willing to be weirder than other people. And when I read about people like Amos Tversky (leave movies early! go jogging in your underwear! throw...

Advice on being managed

Some obvious-in-retrospect thoughts

When you shift from being managed to also sometimes managing others, you have a predictable shift in perspective and a lot of obvious-in-retrospect insights. In the spirit of “saying obvious things is good” here are a few. Be honest Since...

Does gratitude increase happiness?

After decades of research, we now have a huge body of studies and meta-reviews to summarize it. What do they say?

I was wrong about gratitude. I thought it was a guaranteed way to become happier and went around proclaiming we should be thankful because: "Hokey, unfashionable techniques like practicing gratitude turn out to have strong scientific evidence behind them." Sorry...

Nobody optimizes happiness

People don't seem to try very hard to make themselves happier. Why not?

Everyone I know is scheming for the future. They've got big goals and get up every day and work like mad to try to achieve them. I've always found something odd about that: Despite all this effort, people don't seem...

Thoughts on the potato diet

Nine observations about what happens when you eat only potatoes

You've probably heard about the potato diet. If not, here it is: 1. Eat potatoes. 2. As many as you want. 3. Oil and salt are OK. 4. Don't eat other stuff. I thought this sounded delightfully absurd so I...

Things you're doing but don't want to be doing

An analogy between coat racks, desire paths, arguing, vacuuming, reading, social media, drinking, vacations, and colonoscopies.

You learn a lot about people from their bedrooms. Some have TVs or books or laptops. Some have blackout curtains or stuffed animals or bottles of pills. But, vast as human experience is, one thing is consistent: Everyone has once-worn...

How many extra days of life do you get from taking statins?

How much do statins extend lifespan? An analysis based on a meta-analysis of six peer-reviewed papers.

It’s hard to say how much running increases lifespan. To test it, you should take thousands of people, tell half to run, and then follow everyone for years while making sure they follow their instructions. That’s not easy, and even...

Effective selfishness

We could be a lot better at taking care of ourselves.

Here are some things I'd like to know about how to live my life: 1. If I eat Brussels sprouts for dinner tonight instead of pizza, how much longer do I live (in expectation, in minutes)? 2. What should I...

How to run without all the pesky agonizing pain

Why you shouldn't torture yourself when training to run.

I used to think the people I saw running were insane. They were confused about life. Whatever the benefits of running, nothing could justify that much suffering. Runners were cut from a different cloth. They had a strength of will...