Skip to content

Commit 755a298

Browse files
csgregorianxkrogen
authored andcommitted
Fix misimplemented CountTimeWritable setter and update the README docs regarding the output file (PR #96)
1 parent 8ba72ae commit 755a298

File tree

3 files changed

+17
-12
lines changed

3 files changed

+17
-12
lines changed

README.md

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -176,19 +176,23 @@ At this point, a workload job (map-only MapReduce job) can be launched, e.g.:
176176
```
177177
./bin/start-workload.sh
178178
-Dauditreplay.input-path=hdfs:///dyno/audit_logs/
179+
-Dauditreplay.output-path=hdfs:///dyno/results/
179180
-Dauditreplay.num-threads=50
180181
-nn_uri hdfs://namenode_address:port/
181182
-start_time_offset 5m
182183
-mapper_class_name AuditReplayMapper
183184
```
184185
The type of workload generation is configurable; AuditReplayMapper replays an audit log trace as discussed previously.
185-
The AuditReplayMapper is configured via configurations; `auditreplay.input-path` and `auditreplay.num-threads` are
186-
required to specify the input path for audit log files and the number of threads per map task. A number of map tasks
187-
equal to the number of files in `input-path` will be launched; each task will read in one of these input files and
188-
use `num-threads` threads to replay the events contained within that file. A best effort is made to faithfully replay
189-
the audit log events at the same pace at which they originally occurred (optionally, this can be adjusted by
190-
specifying `auditreplay.rate-factor` which is a multiplicative factor towards the rate of replay, e.g. use 2.0 to
191-
replay the events at twice the original speed).
186+
The AuditReplayMapper is configured via configurations; `auditreplay.input-path`, `auditreplay.output-path` and
187+
`auditreplay.num-threads` are required to specify the input path for audit log files, the output path for the results,
188+
and the number of threads per map task. A number of map tasks equal to the number of files in `input-path` will be
189+
launched; each task will read in one of these input files and use `num-threads` threads to replay the events contained
190+
within that file. A best effort is made to faithfully replay the audit log events at the same pace at which they
191+
originally occurred (optionally, this can be adjusted by specifying `auditreplay.rate-factor` which is a multiplicative
192+
factor towards the rate of replay, e.g. use 2.0 to replay the events at twice the original speed).
193+
194+
The AuditReplayMapper will output the benchmark results to a file `part-r-00000` in the output directory in CSV format.
195+
Each line is in the format `user,type,operation,numops,cumulativelatency`, e.g. `hdfs,WRITE,MKDIRS,2,150`.
192196

193197
#### Integrated Workload Launch
194198

@@ -203,6 +207,7 @@ launch an integrated application with the same parameters as were used above, th
203207
-block_list_path hdfs:///dyno/blocks
204208
-workload_replay_enable
205209
-workload_input_path hdfs:///dyno/audit_logs/
210+
-workload_output_path hdfs:///dyno/results/
206211
-workload_threads_per_mapper 50
207212
-workload_start_delay 5m
208213
```

dynamometer-workload/src/main/java/com/linkedin/dynamometer/workloadgenerator/audit/CountTimeWritable.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,11 @@ public long getTime() {
4545
}
4646

4747
public void setCount(long count) {
48-
this.count.set(getCount() + count);
48+
this.count.set(count);
4949
}
5050

5151
public void setTime(long time) {
52-
this.time.set(getTime() + time);
52+
this.time.set(time);
5353
}
5454

5555
@Override

dynamometer-workload/src/test/java/com/linkedin/dynamometer/workloadgenerator/TestWorkloadGenerator.java

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -118,10 +118,10 @@ private void testAuditWorkloadWithOutput(String auditOutputPath) throws Exceptio
118118
try (FSDataInputStream auditOutputFile = dfs.open(new Path(auditOutputPath, "part-r-00000"))) {
119119
String auditOutput = IOUtils.toString(auditOutputFile);
120120
LOG.info(auditOutput);
121-
assertTrue(auditOutput.matches(".*(hdfs,WRITE,[A-Z]+,[17]+,[0-9]+\\n){3}.*"));
121+
assertTrue(auditOutput.matches(".*(hdfs,WRITE,[A-Z]+,[13]+,[0-9]+\\n){3}.*"));
122122
// Matches three lines of the format "hdfs,WRITE,name,count,time"
123-
// Using [17] for the count group because each operation is run either
124-
// 1 or 7 times but the output order isn't guaranteed
123+
// Using [13] for the count group because each operation is run either
124+
// 1 or 3 times but the output order isn't guaranteed
125125
}
126126
}
127127
}

0 commit comments

Comments
 (0)