1
- <a name =" tokenizethis " ></a >
2
1
# TokenizeThis
3
- < a name = " tokenizethis-what-is-it " ></ a >
2
+
4
3
## What is it?
5
4
It turns a string into tokens!.
6
5
@@ -54,17 +53,13 @@ equals(tokens, [
54
53
]);
55
54
```
56
55
57
- <a name =" tokenizethis-installation " ></a >
58
- ## Installation
59
- ` npm install tokenize-this ` .
56
+ ## Installation ` npm install tokenize-this ` .
60
57
61
58
``` js
62
59
// or if in the browser: <script src="tokenize-this/tokenize-this.min.js"></script>
63
60
```
64
61
65
- <a name =" tokenizethis-usage " ></a >
66
- ## Usage
67
- ` require ` it, create a new instance, then call ` tokenize ` .
62
+ ## Usage ` require ` it, create a new instance, then call ` tokenize ` .
68
63
69
64
``` js
70
65
// var TokenizeThis = require('tokenize-this');
@@ -78,11 +73,11 @@ tokenizer.tokenize(str, function(token) {
78
73
equals (tokens, [' Hi' , ' !' , ' ,' , ' I' , ' want' , ' to' , ' add' , 5 , ' +' , 6 ]);
79
74
```
80
75
81
- <a name =" tokenizethis-advanced-usage " ></a >
82
76
## Advanced Usage
83
- <a name =" tokenizethis-advanced-usage-supplying-a-config-object-to-the-constructor-is-also-possible-see-defaultconfig-in-the-api-section-for-all-options " ></a >
84
- ### Supplying a config object to the constructor is also possible (see ".defaultConfig" in the #API section for all options)
85
- This can be used to parse many forms of data, like JSON into key-value pairs.
77
+
78
+ ### Supplying a config object to the constructor is also possible (see [ here] ( #defaultconfigobject ) for all options)
79
+
80
+ This can be used to parse other forms of data, like JSON into key-value pairs.
86
81
87
82
``` js
88
83
var jsonConfig = {
@@ -99,10 +94,10 @@ tokenizer.tokenize(str, function(token) {
99
94
equals (tokens, [' [' , ' {' , ' name' , ' Shaun Persad' , ' id' , 5 , ' }' , ' {' , ' gender' , null , ' }' , ' ]' ]);
100
95
```
101
96
102
- <a name =" tokenizethis-api " ></a >
103
97
## API
104
- <a name =" tokenizethis-api-tokenizestrstring-foreachtokentoken-surroundedbystringfunction " ></a >
105
- ### #tokenize(str: String , forEachToken(token:* , surroundedBy: String ): Function )
98
+
99
+ ### #tokenize(str: String , forEachToken(token:* , surroundedBy: String ): Function )
100
+
106
101
sends each token to the ` forEachToken ` callback.
107
102
108
103
``` js
@@ -128,8 +123,8 @@ tokenizer.tokenize(str, function(token, surroundedBy) {
128
123
equals (tokens, [true , false , null , true , false , null , 1 , 2 , 3.4 , 5.6789 ]);
129
124
```
130
125
131
- <a name =" tokenizethis-api-defaultconfigobject " ></a >
132
126
### .defaultConfig: Object
127
+
133
128
The default config object used when no config is supplied.
134
129
135
130
``` js
0 commit comments