Skip to content

Commit 0f97c99

Browse files
mikemiles-devmikemiles-dev
mikemiles-dev
and
mikemiles-dev
authored
Exporting (#68)
* V5 Re-export * V7 Export * Re-export IPFix * comment * set to flowsets * IPFix Parsing. * Rename * Removed unix timestamp * Update Readme * fix readme * Re-Exporting * Readme --------- Co-authored-by: mikemiles-dev <[email protected]>
1 parent b095c9d commit 0f97c99

26 files changed

+1323
-466
lines changed

‎Cargo.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@ serde = { version = "1.0.166", features = ["derive"] }
1717

1818
[features]
1919
default = ["parse_unknown_fields"]
20-
unix_timestamp = []
2120
parse_unknown_fields = []
2221

2322
[dev-dependencies]

‎README.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,24 @@ let parsed = NetflowParser::default().parse_bytes(&v5_packet);
4848
let v5_parsed: Vec<NetflowPacketResult> = parsed.iter().filter(|p| p.is_v5()).map(|p| p.clone()).collect();
4949
```
5050

51+
## Re-Exporting flows
52+
53+
Netflow Parser now supports parsed V5, V7, V9, IPFix can be re-exported back into bytes.
54+
```rust
55+
let packet = [
56+
0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3,
57+
4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1,
58+
2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7,
59+
];
60+
if let NetflowPacketResult::V5(v5) = NetflowParser::default()
61+
.parse_bytes(&packet)
62+
.first()
63+
.unwrap()
64+
{
65+
assert_eq!(v5.to_be_bytes(), packet);
66+
}
67+
```
68+
5169
## V9/IPFix notes:
5270

5371
Parse the data ('&[u8]' as any other versions. The parser (NetflowParser) holds onto already parsed templates, so you can just send a header/data flowset combo and it will use the cached templates.) To see cached templates simply use the parser for the correct version (v9_parser for v9, ipfix_parser for IPFix.)
@@ -63,8 +81,7 @@ To access templates flowset of a processed V9/IPFix flowset you can find the `fl
6381

6482
## Features
6583

66-
* unix_timestamp - When enabled a field `unix_time` is provided that uses the flow unix time as a count since 0000 UTC 1970 as Duration.
67-
* parse_unknown_fields - When enabled fields not listed in this library will attempt to be parsed as a Vec of bytes and the field_number listed. When disabled an error is thrown when attempting to parse those fields. Enabled by default.
84+
* `parse_unknown_fields` - When enabled fields not listed in this library will attempt to be parsed as a Vec of bytes and the field_number listed. When disabled an error is thrown when attempting to parse those fields. Enabled by default.
6885

6986
## Examples
7087

‎RELEASES.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# 0.3.3
22
* Renamed Sets to FlowSets for IPFIX for consistency.
33
* Concrete error type for parsing
4+
* V5, V7, V9, IPFix now supports exporting back into bytes with `to_be_bytes`.
5+
* V9,IPFix field maps are now keyed by order.
6+
* Removed unix timestamp feature. May re-implement in the future.
47

58
# 0.3.2
69
* Readme changes

‎src/lib.rs

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,25 @@
4848
//! let v5_parsed: Vec<NetflowPacketResult> = parsed.iter().filter(|p| p.is_v5()).map(|p| p.clone()).collect();
4949
//! ```
5050
//!
51+
//! ## Re-Exporting flows
52+
//! Netflow Parser now supports parsed V5, V7, V9, IPFix can be re-exported back into bytes.
53+
//! ```rust
54+
//! use netflow_parser::{NetflowParser, NetflowPacketResult};
55+
//!
56+
//! let packet = [
57+
//! 0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3,
58+
//! 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1,
59+
//! 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7,
60+
//! ];
61+
//! if let NetflowPacketResult::V5(v5) = NetflowParser::default()
62+
//! .parse_bytes(&packet)
63+
//! .first()
64+
//! .unwrap()
65+
//! {
66+
//! assert_eq!(v5.to_be_bytes(), packet);
67+
//! }
68+
//! ```
69+
//!
5170
//! ## V9/IPFix notes:
5271
//!
5372
//! Parse the data (`&[u8]` as any other versions. The parser (NetflowParser) holds onto already parsed templates, so you can just send a header/data flowset combo and it will use the cached templates.) To see cached templates simply use the parser for the correct version (v9_parser for v9, ipfix_parser for IPFix.)
@@ -61,8 +80,7 @@
6180
//!
6281
//! ## Features
6382
//!
64-
//! * unix_timestamp - When enabled a field `unix_time` is provided that uses the flow unix time as a count since 0000 UTC 1970 as Duration.
65-
//! * parse_unknown_fields - When enabled fields not listed in this library will attempt to be parsed as a Vec of bytes and the field_number listed. When disabled an error is thrown when attempting to parse those fields. Enabled by default.
83+
//! * `parse_unknown_fields` - When enabled fields not listed in this library will attempt to be parsed as a Vec of bytes and the field_number listed. When disabled an error is thrown when attempting to parse those fields. Enabled by default.
6684
//!
6785
//! ## Examples
6886
//! Some examples has been included mainly for those who want to use this parser to read from a Socket and parse netflow. In those cases with V9/IPFix it is best to create a new parser for each router. There are both single threaded and multi-threaded examples in the examples directory.

‎src/snapshots/netflow_parser__tests__base_tests__it_doesnt_parse_0_length_fields_ipfix.snap

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,4 +54,3 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
5454
- 2
5555
- 3
5656
- 4
57-

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_ipfix.snap

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
1313
observation_domain_id: 16909060
1414
flowsets:
1515
- header:
16-
id: 2
16+
header_id: 2
1717
length: 20
1818
body:
1919
template:
@@ -30,20 +30,26 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
3030
field_type: PacketDeltaCount
3131
field_length: 4
3232
- header:
33-
id: 256
33+
header_id: 256
3434
length: 28
3535
body:
3636
data:
3737
data_fields:
38-
- PacketDeltaCount:
39-
DataNumber: 16909058
40-
SourceIpv4address:
41-
Ip4Addr: 1.2.3.4
42-
DestinationIpv4address:
43-
Ip4Addr: 1.2.3.3
44-
- PacketDeltaCount:
45-
DataNumber: 67438087
46-
SourceIpv4address:
47-
Ip4Addr: 0.2.0.2
48-
DestinationIpv4address:
49-
Ip4Addr: 0.1.2.3
38+
- 0:
39+
- SourceIpv4address
40+
- Ip4Addr: 1.2.3.4
41+
1:
42+
- DestinationIpv4address
43+
- Ip4Addr: 1.2.3.3
44+
2:
45+
- PacketDeltaCount
46+
- DataNumber: 16909058
47+
- 0:
48+
- SourceIpv4address
49+
- Ip4Addr: 0.2.0.2
50+
1:
51+
- DestinationIpv4address
52+
- Ip4Addr: 0.1.2.3
53+
2:
54+
- PacketDeltaCount
55+
- DataNumber: 67438087

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_ipfix_and_re_exports.snap

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
---
2+
source: src/tests.rs
3+
expression: ipfix.to_be_bytes()
4+
---
5+
- 0
6+
- 10
7+
- 0
8+
- 64
9+
- 1
10+
- 2
11+
- 3
12+
- 4
13+
- 0
14+
- 0
15+
- 0
16+
- 0
17+
- 1
18+
- 2
19+
- 3
20+
- 4
21+
- 0
22+
- 2
23+
- 0
24+
- 20
25+
- 1
26+
- 0
27+
- 0
28+
- 3
29+
- 0
30+
- 8
31+
- 0
32+
- 4
33+
- 0
34+
- 12
35+
- 0
36+
- 4
37+
- 0
38+
- 2
39+
- 0
40+
- 4
41+
- 1
42+
- 0
43+
- 0
44+
- 28
45+
- 1
46+
- 2
47+
- 3
48+
- 4
49+
- 1
50+
- 2
51+
- 3
52+
- 3
53+
- 1
54+
- 2
55+
- 3
56+
- 2
57+
- 0
58+
- 2
59+
- 0
60+
- 2
61+
- 0
62+
- 1
63+
- 2
64+
- 3
65+
- 4
66+
- 5
67+
- 6
68+
- 7

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_ipfix_data_cached_template.snap

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,14 @@ expression: parser.parse_bytes(&packet)
1313
observation_domain_id: 0
1414
flowsets:
1515
- header:
16-
id: 258
16+
header_id: 258
1717
length: 10
1818
body:
1919
data:
2020
data_fields:
21-
- PacketDeltaCount:
22-
DataNumber: 8
23-
SourceIpv4address:
24-
Ip4Addr: 0.0.1.1
21+
- 0:
22+
- PacketDeltaCount
23+
- DataNumber: 8
24+
1:
25+
- SourceIpv4address
26+
- Ip4Addr: 0.0.1.1

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_ipfix_options_template.snap

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
1313
observation_domain_id: 2
1414
flowsets:
1515
- header:
16-
id: 3
16+
header_id: 3
1717
length: 28
1818
body:
1919
options_template:

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_ipfix_options_template_with_data.snap

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
1313
observation_domain_id: 2
1414
flowsets:
1515
- header:
16-
id: 3
16+
header_id: 3
1717
length: 28
1818
body:
1919
options_template:
@@ -32,20 +32,26 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
3232
field_type: ExportedFlowRecordTotalCount
3333
field_length: 2
3434
- header:
35-
id: 260
35+
header_id: 260
3636
length: 20
3737
body:
3838
options_data:
3939
data_fields:
40-
- ExportedMessageTotalCount:
41-
DataNumber: 276
42-
ExportedFlowRecordTotalCount:
43-
DataNumber: 5140
44-
Enterprise:
45-
DataNumber: 1
46-
- ExportedMessageTotalCount:
47-
DataNumber: 5140
48-
ExportedFlowRecordTotalCount:
49-
DataNumber: 7710
50-
Enterprise:
51-
DataNumber: 2
40+
- 0:
41+
- Enterprise
42+
- DataNumber: 1
43+
1:
44+
- ExportedMessageTotalCount
45+
- DataNumber: 276
46+
2:
47+
- ExportedFlowRecordTotalCount
48+
- DataNumber: 5140
49+
- 0:
50+
- Enterprise
51+
- DataNumber: 2
52+
1:
53+
- ExportedMessageTotalCount
54+
- DataNumber: 5140
55+
2:
56+
- ExportedFlowRecordTotalCount
57+
- DataNumber: 7710

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_multiple_packets.snap

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -31,10 +31,12 @@ expression: "NetflowParser::default().parse_bytes(&all)"
3131
body:
3232
data:
3333
data_fields:
34-
- InBytes:
35-
DataNumber: 151126788
36-
Ipv4SrcAddr:
37-
Ip4Addr: 9.9.9.8
34+
- 0:
35+
- InBytes
36+
- DataNumber: 151126788
37+
1:
38+
- Ipv4SrcAddr
39+
- Ip4Addr: 9.9.9.8
3840
- V5:
3941
header:
4042
version: 5
@@ -48,7 +50,7 @@ expression: "NetflowParser::default().parse_bytes(&all)"
4850
engine_type: 6
4951
engine_id: 7
5052
sampling_interval: 2057
51-
sets:
53+
flowsets:
5254
- src_addr: 0.1.2.3
5355
dst_addr: 4.5.6.7
5456
next_hop: 8.9.0.1
@@ -85,7 +87,7 @@ expression: "NetflowParser::default().parse_bytes(&all)"
8587
unix_nsecs: 134807553
8688
flow_sequence: 33752069
8789
reserved: 101124105
88-
sets:
90+
flowsets:
8991
- src_addr: 0.1.2.3
9092
dst_addr: 4.5.6.7
9193
next_hop: 8.9.0.1
@@ -141,10 +143,12 @@ expression: "NetflowParser::default().parse_bytes(&all)"
141143
body:
142144
data:
143145
data_fields:
144-
- InBytes:
145-
DataNumber: 151126788
146-
Ipv4SrcAddr:
147-
Ip4Addr: 9.9.9.8
146+
- 0:
147+
- InBytes
148+
- DataNumber: 151126788
149+
1:
150+
- Ipv4SrcAddr
151+
- Ip4Addr: 9.9.9.8
148152
- IPFix:
149153
header:
150154
version: 10
@@ -156,7 +160,7 @@ expression: "NetflowParser::default().parse_bytes(&all)"
156160
observation_domain_id: 16909060
157161
flowsets:
158162
- header:
159-
id: 2
163+
header_id: 2
160164
length: 20
161165
body:
162166
template:
@@ -173,20 +177,26 @@ expression: "NetflowParser::default().parse_bytes(&all)"
173177
field_type: PacketDeltaCount
174178
field_length: 4
175179
- header:
176-
id: 256
180+
header_id: 256
177181
length: 28
178182
body:
179183
data:
180184
data_fields:
181-
- PacketDeltaCount:
182-
DataNumber: 16909058
183-
SourceIpv4address:
184-
Ip4Addr: 1.2.3.4
185-
DestinationIpv4address:
186-
Ip4Addr: 1.2.3.3
187-
- PacketDeltaCount:
188-
DataNumber: 67438087
189-
SourceIpv4address:
190-
Ip4Addr: 0.2.0.2
191-
DestinationIpv4address:
192-
Ip4Addr: 0.1.2.3
185+
- 0:
186+
- SourceIpv4address
187+
- Ip4Addr: 1.2.3.4
188+
1:
189+
- DestinationIpv4address
190+
- Ip4Addr: 1.2.3.3
191+
2:
192+
- PacketDeltaCount
193+
- DataNumber: 16909058
194+
- 0:
195+
- SourceIpv4address
196+
- Ip4Addr: 0.2.0.2
197+
1:
198+
- DestinationIpv4address
199+
- Ip4Addr: 0.1.2.3
200+
2:
201+
- PacketDeltaCount
202+
- DataNumber: 67438087

‎src/snapshots/netflow_parser__tests__base_tests__it_parses_v5.snap

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ expression: "NetflowParser::default().parse_bytes(&packet)"
1515
engine_type: 6
1616
engine_id: 7
1717
sampling_interval: 2057
18-
sets:
18+
flowsets:
1919
- src_addr: 0.1.2.3
2020
dst_addr: 4.5.6.7
2121
next_hop: 8.9.0.1

0 commit comments

Comments
 (0)