Skip to content

Commit 5c9611e

Browse files
feat: migrated landing about maia content to blog post
1 parent 3dc8df3 commit 5c9611e

File tree

6 files changed

+45
-1049
lines changed

6 files changed

+45
-1049
lines changed
File renamed without changes.

src/blog/posts/introducing_maia.md

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,12 @@ But despite the wide availability of super-strong chess engines, they haven't be
2626

2727
We are introducing Maia, a new human-like chess engine. It is a customized version of AlphaZero trained on human games with the goal of playing the most human-like moves, instead of being trained on self-play games with the goal of playing the optimal moves. In order to characterize human chess-playing at different skill levels, we developed a suite of 9 Maias, one for each Elo level between 1100 and 1900. As you'll see below, Maia is the most human-like chess engine ever created.
2828

29+
Maia's goal is to play the **human** move — not necessarily the best move. As a result, Maia has a more human-like style than previous engines, matching moves played by human players in online games over 50% of the time.
30+
31+
![During training, Maia is given a position that occurred in a real human game and tries to predict which move was made](/blog/maia-diagram_transparent.png)
32+
33+
During training, Maia is given a position that occurred in a real human game and tries to predict which move was made. After seeing hundreds of millions of positions, Maia accurately captures how people at different levels play chess.
34+
2935
If you're curious, you can play against a few versions of Maia on [Lichess](https://lichess.org) or download them yourself from the [GitHub repo](https://github.com/CSSLab/maia-chess).
3036

3137
# Measuring human-like play
@@ -43,12 +49,14 @@ To rigorously compare engines in how human-like they are, we need a good test se
4349
- In each bin, we put all games where both players are in the same rating range
4450
- We drew 10,000 games from each bin, ignoring Bullet and HyperBullet games
4551
- Within each game, we discarded the first 10 ply (a single move made by one player is one
46-
ply) to ignore most memorized opening moves
52+
"ply") to ignore most memorized opening moves
4753
- We also discarded any move where the player had less than 30 seconds to complete the
4854
rest of the game (to avoid situations where players are making random moves)
4955

5056
After these restrictions we had 9 test sets, one for each rating range, which contained roughly 500,000 positions each.
5157

58+
We tested each Maia on these 9 sets of positions, one for each rating level between 1100 and 1900. Every Maia made a prediction for every position, and we measured its resulting move-matching accuracy on each set.
59+
5260
# Previous attempts
5361

5462
People have been trying to create human-like chess engines for decades. For one thing, they would make great sparring partners. But getting crushed like a bug every single game isn't that fun, so the most popular attempts at human-like engines have been some kind of _attenuated_ version of a strong chess engine. For example, the "Play With The Computer" feature on Lichess is a series of [Stockfish](https://stockfishchess.org/) models that are limited in the number of moves they are allowed to look ahead. Chess.com, ICC, FICS, and other platforms all have similar engines. How human-like are they?
@@ -86,19 +94,51 @@ Importantly, every version of Maia uniquely captures a specific human skill leve
8694

8795
This means something deep about chess: there is such a thing as "1100-rated style". And furthermore, it can be captured by a machine learning model. This was surprising to us: it would have been possible that human play is a mixture of good moves and random blunders, with 1100-rated players blundering more often and 1900-rated players blundering less often. Then it would have been impossible to capture 1100-rated style, because random blunders are impossible to predict. But since we _can_ predict human play at different levels, there is a reliable, predictable, and maybe even algorithmically teachable difference between one human skill level and the next.
8896

97+
Each Maia captures human style at its targeted skill level. Lower Maias best predict moves played by lower-rated players, whereas higher Maias predict moves made by higher-rated players.
98+
8999
# Maia's predictions
90100

91101
You can find all of the juicy details in the paper, but one of the most exciting things about Maia is that it can predict _mistakes_. Even when a human makes an absolute howler -- hanging a queen, for example -- Maia predicts the exact mistake made more than 25% of the time. This could be really valuable for average players trying to improve: Maia could look at your games and tell which blunders were _predictable_, and which were random mistakes. If your mistakes are predictable, you know what to work on to hit the next level.
92102

93103
![Move matching accuracy as a function of the quality of the move played in the game](/blog/delta_human_wr.svg)
94104

105+
Maia is particularly good at predicting human mistakes. The move-matching accuracy of any model increases with the quality of the move, since good moves are easier to predict. But even when players make horrific blunders, Maia correctly predicts the exact blunder they make around 25% of the time. This ability to understand how and when people are likely to make mistakes can make Maia a very useful learning tool.
106+
107+
Because we trained 9 different versions of Maia, each at a targeted skill level, we can begin to algorithmically capture what kinds of mistakes players at specific skill levels make – and when people stop making them.
108+
109+
![In this position, Maia levels 1100–1400 correctly predict that White will play the tempting but wrong move b6](/blog/example_board.png)
110+
111+
In this position, Maia levels 1100–1400 correctly predict that White will play the tempting but wrong move b6 (the move played in the game). It threatens the Queen, but after …Qxc5 White's big advantage is mostly gone. Maia levels 1500–1900 predict that, on average, players rated 1500 and above will play the correct bxa6, forcing the Queenside open to decisive effect.
112+
113+
# Personalizing to Individual Players
114+
115+
Next, we tried pushing the modeling of human play to the next level: could we predict the moves a _particular_ human player would make? It turns out that personalizing Maia gives us our biggest performance gains. We achieve these results by fine-tuning Maia: starting with a base Maia, say Maia 1900, we update the model by continuing training on an individual player's games. This approach allows personalized Maias to achieve up to 65% accuracy at predicting particular players' moves.
116+
117+
![By targeting a specific player we can get even higher move prediction accuracy compared to Maias targeting just the player's rating](/blog/maia_transfer_val_accs.svg)
118+
119+
The paper for this work is available [here](https://www.cs.toronto.edu/~ashton/pubs/maia-individual-kdd2022.pdf).
120+
121+
# Play Maia and More
122+
123+
You can play against Maia yourself on Lichess! You can play [Maia 1100](https://lichess.org/@/maia1), [Maia 1500](https://lichess.org/@/maia5), and [Maia 1900](https://lichess.org/@/maia9).
124+
125+
Maia is an ongoing research project using chess as a case study for how to design better human-AI interactions. We hope Maia becomes a useful learning tool and is fun to play against. Our research goals include personalizing Maia to individual players, characterizing the kinds of mistakes that are made at each rating level, running Maia on your games and spotting repeated, predictable mistakes, and more.
126+
127+
This is work in progress and we'd love to hear what you think. Please let us know if you have any feedback or questions by [email](mailto:[email protected]) or [Twitter](https://twitter.com/maiachess).
128+
95129
# Acknowledgments
96130

97131
Many thanks to [Lichess.org](https://lichess.org/) for providing the human games that we trained on, and hosting our Maia models that you can play against. Ashton Anderson was supported in part by an NSERC grant, a Microsoft Research Award, and a CFI grant. Jon Kleinberg was supported in part by a Simons Investigator Award, a Vannevar Bush Faculty Fellowship, a MURI grant, and a MacArthur Foundation grant.
98132

99133
# Further Information
100134

101-
You can read the full paper [here](http://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf), and the view the code on [GitHub](https://github.com/CSSLab/maia-chess).
135+
You can read the full paper [here](http://www.cs.toronto.edu/~ashton/pubs/maia-kdd2020.pdf), and view the code on [GitHub](https://github.com/CSSLab/maia-chess).
136+
137+
If you want to see some more examples of Maia's predictions, we have a tool [here](https://csslab.github.io/Maia-Agreement-Visualizer/) to see where the different models disagree.
138+
139+
All our data is from the wonderful archive at [database.lichess.org](https://database.lichess.org/). We converted the raw PGN raw data dumps into CSV, and have made the CSV we used for testing available at [csslab.cs.toronto.edu/datasets](http://csslab.cs.toronto.edu/datasets/#maia_kdd).
140+
141+
If you're interested in being notified about beta versions of learning tools, teaching aids, and experiments based on Maia (analyses of your games, personalized puzzles, Turing tests, etc.), you can sign up for our email list [here](https://forms.gle/jaQfzSGmaeMcu2UA7).
102142

103143
### Citation
104144

0 commit comments

Comments
 (0)