Skip to content

Commit 8d41b74

Browse files
KyubyongKyubyong
authored
version 2
1 parent d5e40bf commit 8d41b74

File tree

9 files changed

+637
-875
lines changed

9 files changed

+637
-875
lines changed

‎README.md

Lines changed: 67 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,84 +1,86 @@
11
# Can Convolutional Neural Networks Crack Sudoku Puzzles?
22

3-
Sudoku is a popular number puzzle that requires you to fill blanks in a 9X9 grid with digits so that each column, each row, and each of the nine 3×3 subgrids contains all of the digits from 1 to 9. There have been various approaches to that, including computational ones. In this pilot project, we show that convolutional neural networks have the potential to crack Sukoku puzzles without any other rule-based post-processing.
3+
Sudoku is a popular number puzzle that requires you to fill blanks in a 9X9 grid with digits so that each column, each row, and each of the nine 3×3 subgrids contains all of the digits from 1 to 9. There have been various approaches to solving that, including computational ones. In this project, I show that simple convolutional neural networks have the potential to crack Sukoku without any rule-based postprocessing.
44

55
## Requirements
6-
* numpy >= 1.11.1
7-
* sugartensor >= 0.0.1.8 (pip install sugartensor)
6+
* NumPy >= 1.11.1
7+
* TensorFlow == 1.1
88

9-
## Research Question
10-
Can Convolutional Neural Networks Crack Sudoku Puzzles?
11-
129
## Background
1310
* To see what Sudoku is, check the [wikipedia](https://en.wikipedia.org/wiki/Sudoku)
1411
* To investigate this task comprehensively, read through [McGuire et al. 2013](https://arxiv.org/pdf/1201.0749.pdf).
1512

13+
## Dataset
14+
* 1M games were generated using `generate_sudoku.py` for training. I've uploaded them on the Kaggle dataset storage. They are available [here](https://www.kaggle.com/bryanpark/sudoku/downloads/sudoku.zip).
15+
* 30 authentic games were collected from http://1sudoku.com.
16+
17+
## Model description
18+
* 10 blocks of convolution layers of kernel size 3.
19+
20+
## File description
21+
* `generate_sudoku.py` create sudoku games. You don't have to run this. Instead, download [pre-generated games](https://www.kaggle.com/bryanpark/sudoku/downloads/sudoku.zip).
22+
* `hyperparams.py` includes all ajustable hyper parameters.
23+
* `data_load.py` loads data and put them in queues so multiple mini-bach data are generated in parallel.
24+
* `modules.py` contains some wrapper functions.
25+
* `train.py` is for training.
26+
* `test.py` is for test.
27+
28+
1629
## Training
17-
* STEP 1. Generate 1 million Sudoku games. (See `generate_sudoku.py`). The pre-generated games are available [here](https://www.kaggle.com/bryanpark/sudoku/downloads/sudoku.zip).
18-
* STEP 2. Construct convolutional networks as follows. (See `Graph` in `train.py`)<br/>
19-
![graph](graph.png?raw=true)
20-
* STEP 3. Train the model, feeding X (quizzes) and Y (solutions). Note that only the predictions for the position of the blanks count when computing loss. (See `train.py`)<br/>
30+
* STEP 1. Download and extract [training data](https://www.kaggle.com/bryanpark/sudoku/downloads/sudoku.zip).
31+
* STEP 2. Run `python train.py`. Or download the [pretrained file](https://u42868014.dl.dropboxusercontent.com/u/42868014/sudoku/logdir.tar.gz).
32+
33+
## Test
34+
* Run `python test.py`.
2135

22-
## Evaluation
23-
We test the performance of the final model against 30 real Sudoku puzzles and their solutions, which vary from the easy to evil level. Metrics are the following two.
24-
* Accuracy: the number of blanks where our prediction matched to the solution.
25-
* Success rate: the number of games where 100% accurately matched to our prediction.
36+
## Evaluation Metric
37+
38+
Accuray is defined as
39+
40+
Number of blanks where the prediction matched the solution / Number of blanks.
2641

2742
## Results
28-
After 4 epochs, we got [the best model file](https://drive.google.com/open?id=0B0ZXk88koS2KV1VIT2RYUGhuOEU). We designed two test methods.
2943

30-
* Test method 1: Predict the numbers in blanks all at once.
31-
* Test method 2: Predict the numbers sequentially the most confident one at a time.
44+
Ater a couple of hours of training, the training curve seems to reach the optimum.
45+
46+
<img src="fig/training_curve.png">
47+
I use a simple trick in inference. Instead of cracking the whole blanks all at once, I fill in a single blank where the prediction is the most probable among the all predictions. As can be seen below, my model scored 0.86 in accuracy. Details are avaiable in the `results` folder.
48+
3249

3350

34-
| Level | Test1 <br/>(#correct/#blanks=acc.)| Test2 <br/>(#correct/#blanks=acc.) |
35-
| --- |--- |--- |
36-
|Easy|43/47=0.91|**47/47=1.00**|
37-
|Easy|37/45=0.82|**45/45=1.00**|
38-
|Easy|40/47=0.85|**47/47=1.00**|
39-
|Easy|33/45=0.73|**45/45=1.00**|
40-
|Easy|37/47=0.79|**47/47=1.00**|
41-
|Easy|39/46=0.85|**46/46=1.00**|
42-
|Medium|27/53=0.51|32/53=0.60|
43-
|Medium|27/55=0.49|27/55=0.49|
44-
|Medium|32/55=0.58|36/55=0.65|
45-
|Medium|28/53=0.53|**53/53=1.00**|
46-
|Medium|27/52=0.52|33/52=0.63|
47-
|Medium|29/56=0.52|39/56=0.70|
48-
|Hard|30/56=0.54|41/56=0.73|
49-
|Hard|31/55=0.56|28/55=0.51|
50-
|Hard|33/55=0.60|**55/55=1.00**|
51-
|Hard|33/57=0.58|**57/57=1.00**|
52-
|Hard|27/55=0.49|50/55=0.91|
53-
|Hard|28/56=0.50|27/56=0.48|
54-
|Expert|32/56=0.57|22/56=0.39|
55-
|Expert|32/55=0.58|**55/55=1.00**|
56-
|Expert|37/54=0.69|**54/54=1.00**|
57-
|Expert|33/55=0.60|**55/55=1.00**|
58-
|Expert|30/55=0.55|23/55=0.42|
59-
|Expert|25/54=0.46|**54/54=1.00**|
60-
|Evil|32/50=0.64|**50/50=1.00**|
61-
|Evil|33/50=0.66|**50/50=1.00**|
62-
|Evil|34/49=0.69|**49/49=1.00**|
63-
|Evil|33/53=0.62|**53/53=1.00**|
64-
|Evil|35/51=0.69|**51/51=1.00**|
65-
|Evil|34/51=0.67|**51/51=1.00**|
66-
|Total Accuracy| 971/1568=0.62| **1322/1568=0.84**|
67-
|Success Rate| 0/30=0| **19/30=0.63**|
68-
69-
## Conclusions
70-
* I also tested fully connected layers, to no avail.
71-
* Up to some point, it seems that CNNs can learn to solve Sudoku.
72-
* For most problems, the second method was outperform the fist one.
73-
* Humans cannot predict all numbers simultaneously. Probably so do CNNs.
74-
75-
## Furthery Study
76-
* Reinforcement learning would be more appropriate for Sudoku solving.
77-
78-
## Notes for reproducibility
79-
* Download pre-generated Sudoku games [here](https://www.kaggle.com/bryanpark/sudoku/downloads/sudoku.zip) and extract it to `data/` folder.
80-
* Download the pre-trained model file [here](https://drive.google.com/open?id=0B0ZXk88koS2KV1VIT2RYUGhuOEU) and extract it to `asset/train/ckpt` folder.
81-
51+
| Level | Accuracy (#correct/#blanks=acc.) |
52+
| --- |--- |
53+
|Easy|**47/47 = 1.00**|
54+
|Easy|**45/45 = 1.00**|
55+
|Easy|**47/47 = 1.00**|
56+
|Easy|**45/45 = 1.00**|
57+
|Easy|**47/47 = 1.00**|
58+
|Easy|**46/46 = 1.00**|
59+
|Medium|33/53 = 0.62|
60+
|Medium|**55/55 = 1.00**|
61+
|Medium|**55/55 = 1.00**|
62+
|Medium|**53/53 = 1.00**|
63+
|Medium|33/52 = 0.63|
64+
|Medium|51/56 = 0.91|
65+
|Hard|29/56 = 0.52|
66+
|Hard|**55/55 = 1.00**|
67+
|Hard|27/55 = 0.49|
68+
|Hard|**57/57 = 1.00**|
69+
|Hard|35/55 = 0.64|
70+
|Hard|15/56 = 0.27|
71+
|Expert|**56/56 = 1.00**|
72+
|Expert|**55/55 = 1.00**|
73+
|Expert|**54/54 = 1.00**|
74+
|Expert|**55/55 = 1.00**|
75+
|Expert|17/55 = 0.31|
76+
|Expert|**54/54 = 1.00**|
77+
|Evil|**50/50 = 1.00**|
78+
|Evil|**50/50 = 1.00**|
79+
|Evil|**49/49 = 1.00**|
80+
|Evil|28/53 = 0.53|
81+
|Evil|**51/51 = 1.00**|
82+
|Evil|**51/51 = 1.00**|
83+
|Total Accuracy| 1345/1568 = _0.86_|
8284

8385

8486

‎data/test.csv

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
quizzes,solutions
2+
080032001703080002500007030050001970600709008047200050020600009800090305300820010,489532761713486592562917834258341976631759248947268153125673489876194325394825617
3+
000009007060000800789062350430600590090508020018004073043210786005000040100400000,351849267264753819789162354432671598697538421518924673943215786875396142126487935
4+
900401007047508000010700408002100003309000206400003700706009050000305690200604001,928461537647538129513792468872156943359847216461923785736219854184375692295684371
5+
100000203078026401340107000050002806080000010601700020000801042804530160506000009,165498273978326451342157698457912836289643715631785924793861542824539167516274389
6+
600200400015000020024100003102090305030501060507040208200009630050000870009008002,673285491815934726924176583162897345438521967597643218281759634356412879749368152
7+
021400080400150000605038400503607200000090000008504906006710509000043007040005120,321479685489156372675238491593687214264391758718524936836712549152943867947865123
8+
500901008008000500060040090600104007009000800700209003090010070007000600300708005,574961238918327546263845791682134957439576812751289463895612374127453689346798125
9+
400006080003040500000300000809401300100000007005709102000004000006030200030800001,452916783683247519791358624879421356124563897365789142218674935946135278537892461
10+
000050340003400009000700012900060005040000090700030004150007000200001800074020000,617259348823416579495783612932164785546872193781935264158397426269541837374628951
11+
000080100050039080109700300060000200720000016005000030002003804070910020003050000,237485169456139782189726345361547298724398516895261437912673854578914623643852971
12+
007008000030901620060000008140307060000090000070605091500000040013804050000500200,927468135835971624461253978149387562356192487278645391592716843613824759784539216
13+
010305600000200091070000200400010000007080300000060005001000030860007000003106080,912345678346278591578691243435712869627589314189463725751824936864937152293156487
14+
200100004000800500070050900690205000000070000000604015006080030004003000900002008,259137684461829573378456921693215847145378296782694315526781439814963752937542168
15+
471002008208001000000000002090018000004000500000320090700000000000600907300200465,471932658268451379935876142693518724824769531157324896746195283582643917319287465
16+
210500000570942000008000000400000280000605000063000001000000100000214097000006053,214538769576942318938167425459371286721685934863429571697853142385214697142796853
17+
004029000000006703000000050100700036900000008380004002050000000603200000000310200,734529681519846723862173459145782936927631548386954172251468397673295814498317265
18+
000050310400130600000008004065000003080000050300000970700800000006071009048060000,879654312452139687613728594965487123187392456324516978791843265536271849248965731
19+
000040003400008020030900500509000006004090700700000409003001060010500002600070000,195247683476358921238916574589734216364192758721685439953421867817569342642873195
20+
000300029008050006013000080800070000060090050000060001030000290900010600570002000,657348129298157436413629587849571362361294758725863941134786295982415673576932814
21+
000309800400010020560008001830000100000000000002000086600700039070050002001203000,217369845489517623563428971835672194196834257742195386624781539378956412951243768
22+
000609000209000000700010380000800041028090650670001000093020006000000405000905000,381649572259783164746512389935876241128394657674251938593428716862137495417965823
23+
080400300000009004053700090800000045001000900320000008070004610900300000002008070,789415326216839754453726891897163245641582937325947168578294613964371582132658479
24+
000004380800000002000895700000080073007000500410050000009513000300000007085600000,971264385854731692263895741592186473637429518418357269749513826326948157185672934
25+
060000050307000800800200007106008000004375100000400903600002009003000702090000030,469783251327651894815294367136928475984375126752416983678132549543869712291547638
26+
050400000070350014001890200018000000005186900000000580002075100560013090000008020,253461879879352614641897253718539462425186937936724581382975146564213798197648325
27+
090600000012080000087951000860190000030020080000038091000317560000060270000005010,593672148612483957487951326865194732139726485724538691248317569351869274976245813
28+
900006007006014000010900460060350002029000750800029040038007020000840300600100009,943586217786214935215973468467351892329468751851729643538697124192845376674132589
29+
000008015103000800509010004020509000300000009000601050400020508002000107930700000,247368915163954872589217364624539781315872649798641253476123598852496137931785426
30+
000910003803075200020000900070400102050000040204007030005000060006730501400068000,547912683893675214621843975379486152158329746264157839735291468986734521412568397
31+
397050000000001300006030905000006052060507030950300000705040600008200000000070423,397654281542981376186732945873496152264517839951328764725143698438269517619875423

‎data_load.py

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# -*- coding: utf-8 -*-
2+
#/usr/bin/python2
3+
'''
4+
By kyubyong park. [email protected].
5+
https://www.github.com/kyubyong/sudoku
6+
'''
7+
import tensorflow as tf
8+
import numpy as np
9+
from hyperparams import Hyperparams as hp
10+
11+
def load_data(type="train"):
12+
'''Loads training / test data.
13+
14+
Args
15+
type: A string. Either `train` or `test`.
16+
17+
Returns:
18+
X: A 3-D array of float. Entire quizzes.
19+
Has the shape of (# total games, 9, 9)
20+
Y: A 3-D array of int. Entire solutions.
21+
Has the shape of (# total games, 9, 9)
22+
'''
23+
fpath = hp.train_fpath if type=="train" else hp.test_fpath
24+
lines = open(fpath, 'r').read().splitlines()[1:]
25+
nsamples = len(lines)
26+
27+
X = np.zeros((nsamples, 9*9), np.float32)
28+
Y = np.zeros((nsamples, 9*9), np.int32)
29+
30+
for i, line in enumerate(lines):
31+
quiz, solution = line.split(",")
32+
for j, (q, s) in enumerate(zip(quiz, solution)):
33+
X[i, j], Y[i, j] = q, s
34+
35+
X = np.reshape(X, (-1, 9, 9))
36+
Y = np.reshape(Y, (-1, 9, 9))
37+
return X, Y
38+
39+
def get_batch_data():
40+
'''Returns batch data.
41+
42+
Returns:
43+
A Tuple of x, y, and num_batch
44+
x: A `Tensor` of float. Has the shape of (batch_size, 9, 9, 1).
45+
y: A `Tensor` of int. Has the shape of (batch_size, 9, 9).
46+
num_batch = A Python int. Number of batches.
47+
'''
48+
X, Y = load_data(type="train")
49+
50+
# Create Queues
51+
input_queues = tf.train.slice_input_producer([tf.convert_to_tensor(X, tf.float32),
52+
tf.convert_to_tensor(Y, tf.int32)])
53+
54+
# create batch queues
55+
x, y = tf.train.shuffle_batch(input_queues,
56+
num_threads=8,
57+
batch_size=hp.batch_size,
58+
capacity=hp.batch_size*64,
59+
min_after_dequeue=hp.batch_size*32,
60+
allow_smaller_final_batch=False)
61+
# calc total batch count
62+
num_batch = len(X) // hp.batch_size
63+
64+
return x, y, num_batch # (N, 9, 9), (N, 9, 9), ()

‎fig/training_curve.png

25.1 KB
Loading

‎hyperparams.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
class Hyperparams:
2+
'''Hyper parameters'''
3+
# data
4+
train_fpath = '../v2/data/sudoku.csv'
5+
test_fpath = '../v2/data/test.csv'
6+
7+
# model
8+
num_blocks = 10
9+
num_filters = 512
10+
filter_size = 3
11+
12+
# training scheme
13+
lr = 0.0001
14+
logdir = "logdir"
15+
batch_size = 64
16+
num_epochs = 3
17+

0 commit comments

Comments
 (0)