Skip to content

Commit 8ef6110

Browse files
ThomasTNOThomasTNO
committed
Initial Coverage Agent release for Javascript
0 parents  commit 8ef6110

7 files changed

+1380
-0
lines changed

‎README.md

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# WuppieFuzz
2+
3+
TNO developed WuppieFuzz, a coverage-guided REST API fuzzer developed on top of
4+
LibAFL, targeting a wide audience of end-users, with a strong focus on
5+
ease-of-use, explainability of the discovered flaws and modularity. WuppieFuzz
6+
supports all three settings of testing (black box, grey box and white box).
7+
8+
## WuppieFuzz-Javascript
9+
10+
This is a fairly diverse environment, there are:
11+
12+
- various flavors of JavaScript (CommonJS, ES2015==ES6, more but these are the
13+
main ones)
14+
- NodeJS vs browser-based code.
15+
- async and non-async functions
16+
- modules and non-modules
17+
18+
### NodeJS agents
19+
20+
We currently supply two agents for NodeJS. You can incorporate both of these
21+
into your projects by adding a `require`-statement for the respective agent at
22+
the top of your `app.ts`, `app.js` or whatever is the entry point of your
23+
application. See the examples directory for concrete examples.
24+
25+
#### Usage:
26+
27+
```javascript
28+
require("./coverage_agent");
29+
30+
// Your code
31+
```
32+
33+
It's that simple. The coverage agent exposes a server that accepts commands from
34+
the `coverage_client`. Note that when the node process exits, the coverage
35+
server goes down with it. Check the examples to see the coverage agent in
36+
action.
37+
38+
#### NodeJS agent 1: v8 (`coverage_agent.js`)
39+
40+
This uses
41+
[node-specific built-in functionality of v8](https://v8.dev/blog/javascript-code-coverage)
42+
for taking coverage. It works like this:
43+
44+
1. **Coverage tracking**: Generate coverage files with the
45+
[minimal Node-API: `v8.takeCoverage()`](https://nodejs.org/api/v8.html#v8takecoverage).
46+
This writes a file to disk (~100KB-10MB) for every input that we send. **No
47+
customization is possible**, so the huge amount of node_modules also gets
48+
instrumented and reported on. Hopefully we can circumvent both the
49+
file-writing and the instrument-everything problems at some point, but it
50+
seems like a daunting task to dig deeply into v8 and use/customize its
51+
inspector.
52+
2. **Conversion into LCOV**: Read in the coverage files written so far and
53+
process them into coverage reports using a c8-reporter (in particular we use
54+
the `lcovonly`-reporter). At this stage we can **exclude e.g. the
55+
node_modules**. Read in the LCOV coverage report
56+
(`./coverage_report/lcov.info`) and return its contents to WuppieFuzz.
57+
58+
Optional (and working): Use c8 to also create an HTML-report out of all the
59+
coverage files. This takes a second or so, but is extremely useful for
60+
identifying bottlenecks for the fuzzer. Prerequisite is a directory with
61+
coverage data as written by `v8.takeCoverage()` named "node_coverage" in your
62+
working directory. You can then generate the html report with
63+
`node <path-to-create_coverage_report.mjs>`, which will be written to
64+
`./coverage_report` just like the `lcov.info` during fuzzing. You can view the
65+
report by opening `index.html` in a browser.
66+
67+
#### NodeJS Agent 2: runtime-coverage (`nodejs_agent_simple.js`)
68+
69+
This agent can be used both in CommonJS and ES2015, is synchronous (its
70+
`start_coverage` and `get_coverage` functions are async, but must always be
71+
`await`ed on), and is a non-module.
72+
73+
This agent uses the npm module `runtime-coverage`, with the following
74+
(simplified) dependency tree:
75+
76+
- runtime-coverage
77+
78+
- collect-v8-coverage
79+
80+
- inspector
81+
82+
- v8-to-istanbul
83+
84+
- istanbul-lib-coverage
85+
86+
- istanbul-lib-report
87+
88+
- istanbul-reports
89+
90+
This agent listens for coverage requests and has a very simple behaviour:
91+
92+
1. On the first coverage request, start tracking coverage.
93+
2. On any subsequent coverage request, return coverage recorded since the last
94+
coverage request.
95+
96+
Resetting of coverage is automatic, it is up to the client to calculate total
97+
coverage from multiple inputs if that is desired.
98+
99+
#### Which agent to pick
100+
101+
For simplicity try out agent 2, and if this does not work try with agent 1.
102+
Agent 1 has the added complexity of requiring an environment variable and
103+
intermediate file-writes (also possibly impacting performance), but gives more
104+
accurate coverage in some scenarios (see the feathers example).
105+
106+
#### Limitations
107+
108+
1. Note that the agent goes down with the target, so for **microservices** that
109+
live only as long as it takes to process an application request this is
110+
inadequate.
111+
2. The agent works for CommonJS (default for nodejs), it is unclear whether and
112+
how it can be used with **ES2016** modules.

‎coverage_agent.js

+141
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
const v8 = require('node:v8');
2+
const c8 = require('c8');
3+
4+
// Coverage server
5+
const LCOV_HEADER = new Uint8Array([0xc1, 0xc0]);
6+
const BlockType = {
7+
Header: 0x01,
8+
SessionInfo: 0x10,
9+
CoverageInfo: 0x11,
10+
CmdOk: 0x20,
11+
CmdDump: 0x40,
12+
}
13+
14+
const fs = require('node:fs');
15+
16+
function concatUInt8Arrays(a1, a2) {
17+
// sum of individual array lengths
18+
let mergedArray = new Uint8Array(a1.length + a2.length);
19+
mergedArray.set(a1);
20+
mergedArray.set(a2, a1.length);
21+
return mergedArray;
22+
}
23+
24+
class Block {
25+
constructor(block_type, block_data) {
26+
this.block_type = block_type;
27+
this.block_data = block_data;
28+
}
29+
30+
to_buffer() {
31+
let buf
32+
if (this.block_type == BlockType.CoverageInfo) {
33+
buf = Buffer.alloc(1 + 4 + this.block_data.length)
34+
buf.writeUInt32LE(this.block_data.length, 1);
35+
buf.fill(this.block_data, 5);
36+
} else if (this.block_type == BlockType.Header) {
37+
buf = Buffer.alloc(1 + LCOV_HEADER.length);
38+
buf.fill(this.block_data, 1);
39+
}
40+
buf[0] = this.block_type;
41+
return buf;
42+
}
43+
}
44+
45+
const COVERAGE_PORT = 3001;
46+
let coverage_started = false;
47+
48+
let net = require('net');
49+
50+
let server = net.createServer();
51+
52+
server.on('connection', function (socket) {
53+
console.log('Coverage connection established.');
54+
55+
// The server can also receive data from the client by reading from its socket.
56+
socket.on('data', async function (buf) {
57+
// buf.subarray(0, 5) = REQUEST_HEADER, unused by us for the time being.
58+
// buf[5] = command, we only send BLOCK_CMD_DUMP, so this is ignored.
59+
// buf[6] = boolean that indicates whether or not to retrieve coverage (ignore).
60+
// Read RESET PARAMETER
61+
let reset_byte = buf[7];
62+
if (reset_byte == 0) {
63+
console.log("Warning: coverage agent does not support getting coverage without reset.");
64+
}
65+
let reset = true;
66+
// Fetch and return coverage data
67+
if (coverage_started) {
68+
let cov_data = await get_coverage();
69+
if (cov_data != null) {
70+
let header_block = new Block(BlockType.Header, LCOV_HEADER);
71+
let header_block_buf = header_block.to_buffer();
72+
socket.write(header_block_buf);
73+
let coverage_block = new Block(BlockType.CoverageInfo, cov_data)
74+
let cov_block_buf = coverage_block.to_buffer();
75+
socket.write(cov_block_buf);
76+
}
77+
}
78+
if (reset) {
79+
await start_coverage();
80+
}
81+
socket.write(Buffer.from([BlockType.CmdOk]));
82+
});
83+
84+
// When the client requests to end the TCP connection with the server, the server
85+
// ends the connection.
86+
socket.on('end', function () {
87+
console.log('Closing connection with the client');
88+
});
89+
90+
socket.on('error', function (err) {
91+
console.log(`Error: ${err}`);
92+
});
93+
});
94+
95+
server.on('error', (e) => {
96+
if (e.code === 'EADDRINUSE') {
97+
console.error('Address in use, retrying...');
98+
setTimeout(() => {
99+
server.close();
100+
server.listen(COVERAGE_PORT, '127.0.0.1', function () {
101+
console.log(`Coverage agent listening on port ${COVERAGE_PORT}`)
102+
});
103+
}, 1000);
104+
}
105+
});
106+
server.listen(COVERAGE_PORT, '127.0.0.1', function () {
107+
console.log(`Coverage agent listening on port ${COVERAGE_PORT}`)
108+
});
109+
110+
async function start_coverage() {
111+
if (!coverage_started) {
112+
v8.takeCoverage();
113+
coverage_started = true;
114+
return true;
115+
} else {
116+
return false;
117+
}
118+
}
119+
120+
// Return binary-encoded coverage information
121+
async function get_coverage() {
122+
if (!coverage_started) {
123+
console.log("Getting coverage, but haven't started!");
124+
return false;
125+
}
126+
v8.takeCoverage();
127+
coverage_started = false;
128+
const opts = {
129+
all: true,
130+
return: true,
131+
reporter: ['lcovonly'],
132+
deleteCoverage: false,
133+
tempDirectory: "./node_coverage",
134+
reportsDirectory: "./coverage_report",
135+
excludeNodeModules: true
136+
};
137+
let myrep = c8.Report(opts);
138+
await myrep.run();
139+
// Return the LCOV-formatted coverage report
140+
return fs.readFileSync("coverage_report/lcov.info");
141+
}

‎coverage_agent.ts

+138
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
2+
// Coverage server
3+
4+
import {Buffer} from 'buffer/';
5+
6+
const JACOCO_HEADER = [0xc0, 0xc0];
7+
// const RAW_HEADER = [0xc1, 0x0];
8+
const RAW_HEADER = new Uint8Array([0xc1, 0xc0]);
9+
const REQUEST_HEADER = [0x01, 0xc0, 0xc0, 0x10, 0x07];
10+
const BLOCK_CMD_DUMP = 0x40;
11+
const FORMAT_VERSION = new Uint8Array([0x10, 0x07]);
12+
const BlockType = {
13+
Header: 0x01,
14+
SessionInfo: 0x10,
15+
CoverageInfo: 0x11,
16+
CmdOk: 0x20,
17+
CmdDump: 0x40,
18+
}
19+
20+
function concatUInt8Arrays(a1, a2) {
21+
// sum of individual array lengths
22+
let mergedArray = new Uint8Array(a1.length + a2.length);
23+
mergedArray.set(a1);
24+
mergedArray.set(a2, a1.length);
25+
return mergedArray;
26+
}
27+
28+
class Block {
29+
block_type: any;
30+
block_data: any;
31+
constructor(block_type, block_data) {
32+
this.block_type = block_type;
33+
this.block_data = block_data;
34+
}
35+
36+
to_buffer() {
37+
let buf
38+
if (this.block_type == BlockType.CoverageInfo) {
39+
buf = Buffer.alloc(1 + 4 + this.block_data.length)
40+
buf.writeUInt32LE(this.block_data.length, 1);
41+
buf.fill(this.block_data, 5);
42+
} else if (this.block_type == BlockType.Header) {
43+
buf = Buffer.alloc(1 + RAW_HEADER.length + FORMAT_VERSION.length);
44+
buf.fill(this.block_data, 1);
45+
}
46+
buf[0] = this.block_type;
47+
return buf;
48+
}
49+
}
50+
51+
const runtimeCoverage = require('runtime-coverage');
52+
const COVERAGE_PORT = 3001;
53+
let coverage_started = false;
54+
55+
let net = require('net');
56+
const { resolve } = require("path");
57+
const { binary } = require('@hapi/joi');
58+
59+
60+
let server = net.createServer();
61+
62+
server.on('connection', function(socket) {
63+
console.log('Coverage connection established.');
64+
65+
// The server can also receive data from the client by reading from its socket.
66+
socket.on('data', async function(buf) {
67+
// buf.subarray(0, 5) = REQUEST_HEADER, unused by us for the time being.
68+
// buf[5] = command, we only send BLOCK_CMD_DUMP, so this is ignored.
69+
// buf[6] = boolean that indicates whether or not to retrieve coverage (ignore).
70+
// Read RESET PARAMETER
71+
let reset_byte = buf[7];
72+
if (reset_byte == 0) {
73+
console.log("Warning: coverage agent does not support getting coverage without reset.");
74+
}
75+
let reset = true;
76+
// Fetch and return coverage data
77+
if (coverage_started) {
78+
let cov_data = await get_coverage();
79+
if (cov_data != null) {
80+
let header_data = concatUInt8Arrays(RAW_HEADER, FORMAT_VERSION);
81+
// let header_data = new Uint8Array([...RAW_HEADER, ...FORMAT_VERSION]);
82+
let header_block = new Block(BlockType.Header, header_data);
83+
let header_block_buf = header_block.to_buffer();
84+
socket.write(header_block_buf);
85+
let coverage_block = new Block(BlockType.CoverageInfo, cov_data)
86+
let cov_block_buf = coverage_block.to_buffer();
87+
socket.write(cov_block_buf);
88+
}
89+
}
90+
if (reset) {
91+
await start_coverage();
92+
}
93+
socket.write(Buffer.from([BlockType.CmdOk]));
94+
});
95+
96+
// When the client requests to end the TCP connection with the server, the server
97+
// ends the connection.
98+
socket.on('end', function() {
99+
console.log('Closing connection with the client');
100+
});
101+
102+
socket.on('error', function(err) {
103+
console.log(`Error: ${err}`);
104+
});
105+
});
106+
107+
server.listen(COVERAGE_PORT, '127.0.0.1', function() {
108+
console.log(`Coverage agent listening on port ${COVERAGE_PORT}`)
109+
});
110+
111+
async function start_coverage() {
112+
if (!coverage_started) {
113+
runtimeCoverage.startCoverage();
114+
coverage_started = true;
115+
}
116+
}
117+
118+
// Return binary-encoded coverage information
119+
async function get_coverage() {
120+
const options = {
121+
all: true,
122+
return: true,
123+
reporters: ['lcovonly'],
124+
};
125+
coverage_started = false; // runtime-coverage automatically reset every time you get coverage
126+
let coverage_promise = runtimeCoverage.getCoverage(options)
127+
.then((response) => response)
128+
.catch(error => {
129+
console.log(error.message);
130+
start_coverage();
131+
return "coverage not started";
132+
});
133+
let coverage = await coverage_promise;
134+
if (coverage === "coverage not started") {
135+
return null;
136+
}
137+
return coverage["lcovonly"];
138+
}

0 commit comments

Comments
 (0)