Skip to content

Commit 2ab2860

Browse files
jonathantnealjonathantneal
committed
0.0.0
0 parents  commit 2ab2860

30 files changed

+1567
-0
lines changed

Diff for: ‎.editorconfig

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
root = true
2+
3+
[*]
4+
charset = utf-8
5+
end_of_line = lf
6+
indent_style = tab
7+
insert_final_newline = true
8+
trim_trailing_whitespace = true
9+
10+
[*.md]
11+
trim_trailing_whitespace = false
12+
13+
[*.{json,md,yml}]
14+
indent_size = 2
15+
indent_style = space

Diff for: ‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
coverage
2+
dist
3+
node_modules
4+
package-lock.json

Diff for: ‎.npmrc

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
package-lock=false
2+
tag-version-prefix=""

Diff for: ‎.travis.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# https://docs.travis-ci.com/user/travis-lint
2+
3+
language: node_js
4+
5+
env:
6+
- CODECOV_TOKEN="f6419e1e-62fa-4bad-838a-285c9abb2b43"
7+
8+
node_js:
9+
- 12
10+
11+
script:
12+
- npm test
13+
14+
after_success:
15+
- npx codecov

Diff for: ‎CHANGELOG.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Changes to CSS Tokenizer
2+
3+
### 1.0.0 (September 26, 2020)
4+
5+
Initial version

Diff for: ‎LICENSE.md

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# CC0 1.0 Universal
2+
3+
## Statement of Purpose
4+
5+
The laws of most jurisdictions throughout the world automatically confer
6+
exclusive Copyright and Related Rights (defined below) upon the creator and
7+
subsequent owner(s) (each and all, an “owner”) of an original work of
8+
authorship and/or a database (each, a “Work”).
9+
10+
Certain owners wish to permanently relinquish those rights to a Work for the
11+
purpose of contributing to a commons of creative, cultural and scientific works
12+
(“Commons”) that the public can reliably and without fear of later claims of
13+
infringement build upon, modify, incorporate in other works, reuse and
14+
redistribute as freely as possible in any form whatsoever and for any purposes,
15+
including without limitation commercial purposes. These owners may contribute
16+
to the Commons to promote the ideal of a free culture and the further
17+
production of creative, cultural and scientific works, or to gain reputation or
18+
greater distribution for their Work in part through the use and efforts of
19+
others.
20+
21+
For these and/or other purposes and motivations, and without any expectation of
22+
additional consideration or compensation, the person associating CC0 with a
23+
Work (the “Affirmer”), to the extent that he or she is an owner of Copyright
24+
and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and
25+
publicly distribute the Work under its terms, with knowledge of his or her
26+
Copyright and Related Rights in the Work and the meaning and intended legal
27+
effect of CC0 on those rights.
28+
29+
1. Copyright and Related Rights. A Work made available under CC0 may be
30+
protected by copyright and related or neighboring rights (“Copyright and
31+
Related Rights”). Copyright and Related Rights include, but are not limited
32+
to, the following:
33+
1. the right to reproduce, adapt, distribute, perform, display, communicate,
34+
and translate a Work;
35+
2. moral rights retained by the original author(s) and/or performer(s);
36+
3. publicity and privacy rights pertaining to a person’s image or likeness
37+
depicted in a Work;
38+
4. rights protecting against unfair competition in regards to a Work,
39+
subject to the limitations in paragraph 4(i), below;
40+
5. rights protecting the extraction, dissemination, use and reuse of data in
41+
a Work;
42+
6. database rights (such as those arising under Directive 96/9/EC of the
43+
European Parliament and of the Council of 11 March 1996 on the legal
44+
protection of databases, and under any national implementation thereof,
45+
including any amended or successor version of such directive); and
46+
7. other similar, equivalent or corresponding rights throughout the world
47+
based on applicable law or treaty, and any national implementations
48+
thereof.
49+
50+
2. Waiver. To the greatest extent permitted by, but not in contravention of,
51+
applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and
52+
unconditionally waives, abandons, and surrenders all of Affirmer’s Copyright
53+
and Related Rights and associated claims and causes of action, whether now
54+
known or unknown (including existing as well as future claims and causes of
55+
action), in the Work (i) in all territories worldwide, (ii) for the maximum
56+
duration provided by applicable law or treaty (including future time
57+
extensions), (iii) in any current or future medium and for any number of
58+
copies, and (iv) for any purpose whatsoever, including without limitation
59+
commercial, advertising or promotional purposes (the “Waiver”). Affirmer
60+
makes the Waiver for the benefit of each member of the public at large and
61+
to the detriment of Affirmer’s heirs and successors, fully intending that
62+
such Waiver shall not be subject to revocation, rescission, cancellation,
63+
termination, or any other legal or equitable action to disrupt the quiet
64+
enjoyment of the Work by the public as contemplated by Affirmer’s express
65+
Statement of Purpose.
66+
67+
3. Public License Fallback. Should any part of the Waiver for any reason be
68+
judged legally invalid or ineffective under applicable law, then the Waiver
69+
shall be preserved to the maximum extent permitted taking into account
70+
Affirmer’s express Statement of Purpose. In addition, to the extent the
71+
Waiver is so judged Affirmer hereby grants to each affected person a
72+
royalty-free, non transferable, non sublicensable, non exclusive,
73+
irrevocable and unconditional license to exercise Affirmer’s Copyright and
74+
Related Rights in the Work (i) in all territories worldwide, (ii) for the
75+
maximum duration provided by applicable law or treaty (including future time
76+
extensions), (iii) in any current or future medium and for any number of
77+
copies, and (iv) for any purpose whatsoever, including without limitation
78+
commercial, advertising or promotional purposes (the “License”). The License
79+
shall be deemed effective as of the date CC0 was applied by Affirmer to the
80+
Work. Should any part of the License for any reason be judged legally
81+
invalid or ineffective under applicable law, such partial invalidity or
82+
ineffectiveness shall not invalidate the remainder of the License, and in
83+
such case Affirmer hereby affirms that he or she will not (i) exercise any
84+
of his or her remaining Copyright and Related Rights in the Work or (ii)
85+
assert any associated claims and causes of action with respect to the Work,
86+
in either case contrary to Affirmer’s express Statement of Purpose.
87+
88+
4. Limitations and Disclaimers.
89+
1. No trademark or patent rights held by Affirmer are waived, abandoned,
90+
surrendered, licensed or otherwise affected by this document.
91+
2. Affirmer offers the Work as-is and makes no representations or warranties
92+
of any kind concerning the Work, express, implied, statutory or
93+
otherwise, including without limitation warranties of title,
94+
merchantability, fitness for a particular purpose, non infringement, or
95+
the absence of latent or other defects, accuracy, or the present or
96+
absence of errors, whether or not discoverable, all to the greatest
97+
extent permissible under applicable law.
98+
3. Affirmer disclaims responsibility for clearing rights of other persons
99+
that may apply to the Work or any use thereof, including without
100+
limitation any person’s Copyright and Related Rights in the Work.
101+
Further, Affirmer disclaims responsibility for obtaining any necessary
102+
consents, permissions or other rights required for any use of the Work.
103+
4. Affirmer understands and acknowledges that Creative Commons is not a
104+
party to this document and has no duty or obligation with respect to this
105+
CC0 or use of the Work.
106+
107+
For more information, please see
108+
http://creativecommons.org/publicdomain/zero/1.0/.

Diff for: ‎README.md

+126
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# CSS Tokenizer
2+
3+
[<img alt="npm version" src="https://img.shields.io/npm/v/@csstools/tokenizer.svg" height="20">](https://www.npmjs.com/package/@csstools/tokenizer)
4+
[<img alt="build status" src="https://img.shields.io/travis/csstools/tokenizer/master.svg" height="20">](https://travis-ci.org/github/csstools/tokenizer)
5+
[<img alt="code coverage" src="https://img.shields.io/codecov/c/github/csstools/tokenizer" height="20">](https://codecov.io/gh/csstools/tokenizer)
6+
[<img alt="issue tracker" src="https://img.shields.io/github/issues/csstools/tokenizer.svg" height="20">](https://github.com/csstools/tokenizer/issues)
7+
[<img alt="pull requests" src="https://img.shields.io/github/issues-pr/csstools/tokenizer.svg" height="20">](https://github.com/csstools/tokenizer/pulls)
8+
[<img alt="support chat" src="https://img.shields.io/badge/support-chat-blue.svg" height="20">](https://gitter.im/postcss/postcss)
9+
10+
This tools lets you tokenize CSS according to the [CSS Syntax Specification](https://drafts.csswg.org/css-syntax/).
11+
It works by separating a string of CSS into its smallest, semantic parts.
12+
This is intended for use by other tools, in either the backend or the frontend, and it will contribute roughly 1kB of code.
13+
14+
## Usage
15+
16+
Add this [CSS tokenizer](https://github.com/csstools/tokenizer) to your project:
17+
18+
```sh
19+
npm install @csstools/tokenizer
20+
```
21+
22+
Use this tool to tokenizer your CSS in JavaScript:
23+
24+
```js
25+
import cssTokenizer from '@csstools/tokenizer'
26+
27+
const tokens = Array.from(cssTokenizer(cssText))
28+
```
29+
30+
Use this tool in classical NodeJS:
31+
32+
```js
33+
const cssTokenizer = require('@csstools/tokenizer')
34+
35+
let iterator = cssTokenizer(cssText), iteration
36+
37+
while (!(iteration = iterator()).done) {
38+
console.log(iteration.value) // logs the token
39+
}
40+
```
41+
42+
Use this tool in scripts:
43+
44+
```html
45+
<script type="module">
46+
import cssTokenizer from "https://unpkg.com/@csstools/tokenizer?module"
47+
cssTokenizer(cssText)
48+
</script>
49+
```
50+
51+
Use this tool in classical scripts:
52+
53+
```html
54+
<script src="http://unpkg.com/@csstools/tokenizer"></script>
55+
<script>
56+
cssTokenizer(cssText)
57+
</script>
58+
```
59+
60+
## How it works
61+
62+
As an example, the string `@media` would become a **Name** token where `@` and `media` are recognized as distinct parts of that token. As another example, the string `5px` would become a **Number** token where `5` and `px` are recognized as distinct parts of that token. As a final example, the string `5px 10px` would become 3 tokens; the **Number** as mentioned before (`5px`), a **Space** token that represents a single space (` `), and then another **Number** token (`10px`).
63+
64+
An actual token is represented in a series of 5 items;
65+
66+
```js
67+
[ 0, -9, '', '100', '%' ] // CSS with a value of "100%"
68+
```
69+
70+
The **first** number represents the position at which the token was read. The **second** number represents the type id of the token. The **third**, **fourth**, and **fifth** strings represent the text prefix, value, and suffix of the token.
71+
72+
## Benchmarks
73+
74+
As of September 26, 2020, these benchmarks were averaged from my local machine:
75+
76+
```
77+
Benchmark: Tailwind CSS
78+
┌──────────────────────────────────────────────────┬───────┬────────┬────────┐
79+
│ (index) │ ms │ ms/50k │ tokens │
80+
├──────────────────────────────────────────────────┼───────┼────────┼────────┤
81+
│ PostCSS x 12.38 ops/sec ±3.33% (35 runs sampled) │ 80.79 │ 6.97 │ 579896 │
82+
│ Develop x 12.07 ops/sec ±1.54% (34 runs sampled) │ 82.86 │ 6.17 │ 670972 │
83+
└──────────────────────────────────────────────────┴───────┴────────┴────────┘
84+
Benchmark: Bootstrap
85+
┌────────────────────────────────────────────────┬──────┬────────┬────────┐
86+
│ (index) │ ms │ ms/50k │ tokens │
87+
├────────────────────────────────────────────────┼──────┼────────┼────────┤
88+
│ PostCSS x 230 ops/sec ±0.50% (89 runs sampled) │ 4.35 │ 4.37 │ 49807 │
89+
│ Develop x 142 ops/sec ±0.76% (80 runs sampled) │ 7.04 │ 5.92 │ 59502 │
90+
└────────────────────────────────────────────────┴──────┴────────┴────────┘
91+
```
92+
93+
## Contributing: Local Scripts
94+
95+
You want to take a deeper dive? Awesome! Here are a few useful development commands.
96+
97+
### npm run build
98+
99+
The **build** command creates all the files needed to run this tool in many different JavaScript environments.
100+
101+
```sh
102+
npm run build
103+
```
104+
105+
### npm run benchmark
106+
107+
The **benchmark** command builds the project and then tests its performance as compared to [PostCSS].
108+
These benchmarks are run against [Boostrap] and [Tailwind CSS].
109+
110+
```sh
111+
npm run benchmark
112+
```
113+
114+
### npm run test
115+
116+
The **test** command tests the coverage and accuracy of the tokenizer.
117+
118+
As of September 26, 2020, this tokenizer has 100% test coverage:
119+
120+
```sh
121+
npm run test
122+
```
123+
124+
[Bootstrap]: https://getbootstrap.com
125+
[PostCSS]: https://postcss.org
126+
[Tailwind CSS]: https://tailwindcss.com

Diff for: ‎babel.config.cjs

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
module.exports = {
2+
presets: [
3+
['@babel/preset-env', {
4+
targets: {
5+
node: 'current'
6+
}
7+
}],
8+
['@babel/preset-typescript']
9+
]
10+
}

Diff for: ‎cmd/_.mjs

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import * as fs from 'fs'
2+
import * as path from 'path'
3+
import * as process from 'child_process'
4+
import * as readline from 'readline'
5+
import * as url from 'url'
6+
7+
export const __dirname = path.resolve(url.fileURLToPath(import.meta.url), '..', '..')
8+
export const __bin = path.resolve('node_modules', '.bin')
9+
export const __dist = path.resolve('dist')
10+
export const __modules = path.resolve('node_modules')
11+
12+
export const resolve = (...paths) => path.resolve(__dirname, ...paths)
13+
14+
export const spawn = (cmd, args, opts) => process.spawnSync(cmd, args, { cwd: __dirname, stdio: 'inherit', ...Object(opts) })
15+
export const spawnNode = (cmd, args, opts) => spawn('node', [path.resolve(cmd), ...Array.from(Object(args))], opts)
16+
export const spawnTsc = (args, opts) => spawnNode('node_modules/typescript/lib/tsc.js', args, opts)
17+
export const spawnTscWatch = (args, opts) => spawnNode('node_modules/tsc-watch/lib/tsc-watch.js', args, opts)
18+
19+
export const rmdir = (...paths) => fs.rmdirSync(resolve(...paths), { recursive: true })
20+
21+
export const question = query => new Promise(resolve => {
22+
const rl = readline.createInterface(question.options)
23+
rl.question('major / minor / patch? ', answer => {
24+
rl.close()
25+
resolve(answer)
26+
})
27+
})
28+
29+
question.options = {}

Diff for: ‎cmd/benchmark.mjs

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import * as _ from './_.mjs'
2+
3+
const isWatch = process.argv.includes('--watch')
4+
5+
console.log(`Starting compilation...`)
6+
_.rmdir('dist')
7+
8+
if (isWatch) _.spawnTscWatch([ '--build', '--onSuccess', 'node cmd/benchmark.post.mjs' ])
9+
else {
10+
_.spawnTsc([ '--build' ])
11+
_.spawnNode('cmd/benchmark.post.mjs')
12+
}

Diff for: ‎cmd/benchmark.post.mjs

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
import * as _ from './_.mjs'
2+
3+
_.rmdir('dist/lib')
4+
_.spawnNode('dist/src/index.benchmark.js')

Diff for: ‎cmd/build.mjs

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import * as _ from './_.mjs'
2+
3+
const isWatch = process.argv.includes('--watch')
4+
5+
console.log(`Starting compilation...`)
6+
_.rmdir('dist')
7+
8+
if (isWatch) _.spawnTscWatch([ '--build', 'tsconfig.json.build.json', '--onSuccess', 'node cmd/build.post.mjs' ])
9+
else {
10+
_.spawnTsc([ '--build', 'tsconfig.json.build.json' ])
11+
_.spawnNode('cmd/build.post.mjs')
12+
}

Diff for: ‎cmd/build.post.mjs

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
import * as _ from './_.mjs'
2+
3+
console.log()
4+
_.rmdir('dist/lib')
5+
_.spawnNode('node_modules/rollup/dist/bin/rollup', ['--config', '--silent'])

Diff for: ‎cmd/publish.mjs

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
import * as _ from './_.mjs'
2+
import './test.mjs'
3+
import './build.mjs'
4+
5+
const isDryRun = Boolean(process?.env?.npm_config_dryRun)
6+
7+
console.log(), console.log('Starting publishing...')
8+
9+
_.question.options = { input: process.stdin, output: process.stdout }
10+
11+
_.question('major/minor/path? ').then(answer => {
12+
answer = answer.trim().toLowerCase()
13+
if (answer === 'major' || answer === 'minor' || answer === 'patch') {
14+
if (!isDryRun) _.spawn('npm', ['version', answer])
15+
}
16+
})

Diff for: ‎cmd/test.mjs

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import * as _ from './_.mjs'
2+
3+
const isWatch = process.argv.includes('--watch')
4+
5+
const jestBin = 'node_modules/jest/bin/jest.js'
6+
7+
console.log(`Starting testing...`)
8+
console.log()
9+
10+
_.rmdir('coverage')
11+
_.spawnNode(jestBin, [
12+
'--colors',
13+
'--passWithNoTests',
14+
...(isWatch ? ['--watchAll'] : [])
15+
])

0 commit comments

Comments
 (0)