Skip to content

Commit 60af72a

Browse files
fntlnzfntlnz
authored
bpftrace chapter (#1)
bpftrace chapter
2 parents ecb30d2 + b38951e commit 60af72a

24 files changed

+359
-48
lines changed

Diff for: ‎content/10-welcome.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,5 @@ Linux <i><b>e</b>xtended <b>B</b>erkeley <b>P</b>acket <b>F</b>ilters</i>
1212
*Be kind with others*<br/>
1313
*Thank you!*
1414

15-
**Slides: https://bpf.sh/workshop**
15+
**Slides: https://workshop.bpf.sh/**
1616
]

Diff for: ‎content/30-pre-requirements.md

+1
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,4 @@ class: extra-details
1616

1717
- a little bit of bash-fu (environment variables, loops)
1818

19+

Diff for: ‎content/bpftrace/501-bpftrace.md

+14-4
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,19 @@ weight: 501
77

88
On GitHub https://github.com/iovisor/bpftrace
99

10-
- Built from the ground-up for BPF and Linux
11-
- Used in production at Netflix, Facebook, etc
12-
- Custom one-liners
13-
- Tools
10+
*What it is*:
11+
12+
- Higher level language to write eBPF programs;
13+
- Built from the ground-up for BPF and Linux;
14+
- Used in production at Netflix, Facebook, etc;
15+
- Custom one-liners;
16+
- Comes with tools;
17+
- It is just for tracing;
18+
19+
*What it is NOT*:
20+
21+
- A framework to build your loaders;
22+
- You can't do classic bpf with it (like seccomp programs or socket probe types);
23+
- It does not support traffic control and XDP;
1424

1525

Diff for: ‎content/bpftrace/504-bpftrace-oneliners.md

-28
This file was deleted.

Diff for: ‎content/bpftrace/517-bpftrace-functions.md

+16-15
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,20 @@
11
---
2-
title: "bpftrace-actions"
3-
weight: 516
2+
title: "bpftrace-functions"
3+
weight: 517
44
---
55

6-
# bpftrace: Actions
7-
8-
**Per-event output**
9-
10-
- printf()
11-
- system()
12-
- join()
13-
- time()
14-
15-
**Map Summaries**
16-
17-
- @ = count() or @++
18-
- @ = hist()
6+
# bpftrace: Functions
197

8+
| function | description |
9+
|------------------------------------------|--------------------------------------------------------|
10+
| hist(int n) | Produce a log2 histogram of values of n |
11+
| lhist(int n# int min# int max# int step) | Produce a linear histogram of values of n |
12+
| count() | Count the number of times this function is called |
13+
| sum(int n) | Sum this value |
14+
| min(int n) | Record the minimum value seen |
15+
| max(int n) | Record the maximum value seen |
16+
| avg(int n) | Average this value |
17+
| stats(int n) | Return the count# average# and total for this value |
18+
| delete(@x) | Delete the map element passed in as an argument |
19+
| str(char *s [# int length]) | Returns the string pointed to by s |
20+
| printf(char *fmt# ...) | Print formatted to stdout |

Diff for: ‎content/bpftrace/518-bpftrace-functions-cont.md

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: "bpftrace-functions-contd"
3+
weight: 518
4+
---
5+
6+
# bpftrace: Functions (cont'd)
7+
8+
| function | description |
9+
|------------------------------------------|--------------------------------------------------------|
10+
| print(@x[# int top [# int div]]) | Print a map# with optional top entry count and divisor |
11+
| clear(@x) | Delete all key/values from a map |
12+
| sym(void *p) | Resolve kernel address |
13+
| usym(void *p) | Resolve user space address |
14+
| ntop([int af# ]int|char[4|16] addr) | Resolve ip address |
15+
| kaddr(char *name) | Resolve kernel symbol name |
16+
| uaddr(char *name) | Resolve user space symbol name |
17+
| reg(char *name) | Returns the value stored in the named register |
18+
| join(char *arr[] [# char *delim]) | Prints the string array |
19+
| time(char *fmt) | Print the current time |
20+
| cat(char *filename) | Print file content |
21+
| system(char *fmt) | Execute shell command |
22+
| exit() | Quit bpftrace |

Diff for: ‎content/bpftrace/519-variable-types.md

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
title: "bpftrace-variable-types"
3+
weight: 519
4+
---
5+
6+
# bpftrace: Variable types
7+
8+
**Basic Variables**
9+
10+
- @global
11+
- @thread_local[tid]
12+
- $scratch
13+
14+
**Associative Arrays**
15+
16+
- @array[key] = value
17+
18+
**Buitins**
19+
20+
- pid
21+
- ...

Diff for: ‎content/bpftrace/520-builtin-variables.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
title: "bpftrace-builtin-variables"
3+
weight: 520
4+
---
5+
6+
# bpftrace: Builtin Variables
7+
8+
| variable | description |
9+
|----------------------|----------------------------------------------------|
10+
| tid | Thread ID (kernel pid) |
11+
| cgroup | Cgroup ID of the current process |
12+
| uid | User ID |
13+
| gid | Group ID |
14+
| nsecs | Nanosecond timestamp |
15+
| elapsed | Nanosecond timestamp since bpftrace initialization |
16+
| cpu | Processor ID |
17+
| comm | Process name |

Diff for: ‎content/bpftrace/521-builtin-variables-contd.md

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
title: "bpftrace-builtin-variables-contd"
3+
weight: 521
4+
---
5+
6+
# bpftrace: Builtin Variables (cont'd)
7+
8+
| variable | description |
9+
|----------------------|----------------------------------------------------|
10+
| pid | Process ID (kernel tgid) |
11+
| stack | Kernel stack trace |
12+
| ustack | User stack trace |
13+
| arg0, arg1, ... etc. | Arguments to the function being traced |
14+
| retval | Return value from function being traced |
15+
| func | Name of the function currently being traced |
16+
| probe | Full name of the probe |
17+
| curtask | Current task_struct as a u64 |
18+
| rand | Random number of type u32 |
19+
| $1, $2, ... etc. | Positional parameters to the bpftrace program |

Diff for: ‎content/bpftrace/523-bpftrace-exercise-tools.md

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: "bpftrace-exercise-tools"
3+
weight: 523
4+
class: extra-details
5+
---
6+
7+
## bpftrace hands on: Tools!
8+
9+
1. We will clone the bpftrace repository in our Linux machine;
10+
2. We are not cloning it to install bpftrace itself, but to get all the tools under the `tools` folder
11+
12+
.exercise[
13+
- Clone the bpftrace repo
14+
```bash
15+
git clone https://github.com/iovisor/bpftrace.git
16+
cd bpftrace/tools
17+
```
18+
]

Diff for: ‎content/bpftrace/524-bpftrace-exercise-tools-tcpretrans.md

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: "bpftrace-exercise-tools-tcpretrans"
3+
weight: 524
4+
class: extra-details
5+
---
6+
7+
## bpftrace hands on: Trace or count TCP retransmits
8+
9+
- In the bpftrace tools folder, there's a tool called `tcpretrans.bt`;
10+
- TCP wants to make sure that your packet is received with the *guarantee* that all the received bytes will be identical and in the same order as those sent,
11+
this technique is called **positive acknowledgement with re-transmission**;
12+
- What happens when there are many retransmits is that your system can have a significant overhead, then you want to know when a retransmit occurs, `tcpretrans.bt` does just that
13+
- Retransmits are usually a sign of poor network health, and this tool is
14+
useful for their investigation. Unlike using tcpdump, this tool has very
15+
low overhead, as it only traces the retransmit function. It also prints
16+
additional kernel details: the state of the TCP session at the time of the
17+
retransmit.
18+

Diff for: ‎content/bpftrace/525-bpftrace-exercise-tools-tcpretrans-contd.md

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
title: "bpftrace-exercise-tools-tcpretrans-contd"
3+
weight: 525
4+
class: extra-details
5+
---
6+
7+
## bpftrace hands on: Trace or count TCP retransmits (cont'd)
8+
9+
.exercise[
10+
- In the `bpftrace/tools` folder;
11+
- With root permissions;
12+
- Execute the `tcpretrans.bt` tool;
13+
14+
```bash
15+
bpftrace tcpretrans.bt
16+
```
17+
- Once it's started, the best way to trigger some retransmits is to try to connect to a closed port;
18+
- Try it on a new terminal while leaving `tcpretrans.bt` active!
19+
20+
```bash
21+
telnet bpf.sh 9090
22+
```
23+
]

Diff for: ‎content/bpftrace/526-bpftrace-oneliner-read-kernel-tracing-vfs.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
---
3+
title: "bpftrace-oneliners-read-kernel-tracing-vfs"
4+
weight: 526
5+
class: extra-details
6+
---
7+
8+
# bpftrace hands on: tracing read bytes using a kretprobe
9+
10+
- We will use the capability of bpftrace to instrument the `vfs_read` function in the kernel using a `kretprobe`;
11+
- We will create an array called `bytes` that will dump a linear histogram where the arguments are: value, min, max, step. The first argument (retval) of vfs_read() is the return value: the number of bytes read;
12+
13+
.exercise[
14+
- Execute this one liner using bpftrace, then let it run for a while then use `Ctrl-C` to dump the results
15+
```bash
16+
bpftrace -e 'kretprobe:vfs_read { @bytes = lhist(retval, 0, 2000, 200); }'
17+
```
18+
]
19+
20+
.footnote[.smaller[
21+
In Linux, all files are accessed through the Virtual Filesystem Switch, or VFS, a layer of code which implements generic filesystem actions and vectors requests to the correct specific code to handle the request.
22+
]]
23+
24+

Diff for: ‎content/bpftrace/527-bpftrace-oneliner-read-syscall.md

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
2+
---
3+
title: "bpftrace-oneliners-read-syscall"
4+
weight: 527
5+
class: extra-details
6+
---
7+
8+
# bpftrace hands on: tracing read bytes using a tracepoint
9+
10+
- We want to do the same thing we did with the `kretprobe` in the previous exercise
11+
12+
.exercise[
13+
- Execute this one liner using bpftrace
14+
```bash
15+
bpftrace -e 'tracepoint:syscalls:sys_exit_read { @bytes = lhist(args->ret, 0, 2000, 200); }'
16+
```
17+
- Let it run for a while then use `Ctrl-C` to dump the results
18+
]
19+
20+
**What's the difference?**
21+
While being very powerful (it can trace any kernel function), `kretprobe` approach can't be considered "stable", because internal
22+
kernel functions can change between kernels. On the other hand using a tracepoint is a much more stable approach because tracepoints
23+
are considered as a user facing feature and not an internal one by kernel developers.
24+
Whenever possible use tracepoints instead of kprobe/kretprobe.
25+
26+

Diff for: ‎content/bpftrace/528-bpftrace-oneliner-uretprobe.md

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
2+
---
3+
title: "bpftrace-oneliners-uretprobe"
4+
weight: 528
5+
class: extra-details
6+
---
7+
8+
# bpftrace hands on: reading userspace returns
9+
10+
We have a Go program that prints a random number every second.
11+
```go
12+
package main
13+
import(
14+
"time"
15+
"fmt"
16+
"math/rand"
17+
)
18+
func main() {
19+
for {
20+
time.Sleep(time.Second * 1)
21+
fmt.Printf("%d\n", giveMeNumber())
22+
}
23+
}
24+
func giveMeNumber() int {
25+
return rand.Intn(100) + rand.Intn(900)
26+
}
27+
```
28+
We want to get the random number out of it using a bpftrace program.

Diff for: ‎content/bpftrace/529-bpftrace-oneliner-uretprobe-contd.md

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
2+
---
3+
title: "bpftrace-oneliners-uretprobe-contd"
4+
weight: 528
5+
class: extra-details
6+
---
7+
8+
# bpftrace hands on: reading userspace returns
9+
.exercise[
10+
- Create a file named `main.go` with the code from [previous slide](#bpftrace-oneliners-uretprobe);
11+
- Then, compile it with:
12+
```bash
13+
go build -o randomnumbers main.go
14+
```
15+
- This will create a binary named `randomnumbers` in the current folder;
16+
- Once that is done, we just start the program `./randomnumbers`;
17+
- Now, in a new terminal, we instrument the program using bpftrace and a `uretprobe`:
18+
19+
```bash
20+
bpftrace -e \
21+
'uretprobe:./randomnumbers:"main.giveMeNumber"
22+
{ printf("%d\n", retval) }'
23+
```
24+
Bonus point, try to do an `objdump -t randomnumbers | grep -i giveMe`, what do you notice?
25+
]
26+

Diff for: ‎content/bpftrace/590-bpftrace-internals.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
2+
---
3+
title: "bpftrace-internals"
4+
weight: 590
5+
---
6+
7+
# bpftrace: Internals
8+
9+
.pic[
10+
![supported bpf probe types](/img/bpftrace-internals.png)
11+
]

Diff for: ‎content/bpftrace/598-takeaways.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
2+
---
3+
title: "bpftrace-credits"
4+
weight: 599
5+
---
6+
7+
# Takeaways
8+
9+
- There's an higher level language to use eBPF, called `bpftrace`;
10+
- `bpftrace` can be used only for eBPF based tracing;
11+
- It's pretty magic;
12+
- There are a **LOT** of premade tools you can use in the `bpftrace/tools` folder, saves a lot of time;

Diff for: ‎content/bpftrace/599-bpftrace-credits.md

+3
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,6 @@ weight: 599
88

99
- [Brendan Gregg's presentation on bpftrace](https://github.com/iovisor/bpf-docs/blob/master/bpftrace_public_template_jun2019.odp)
1010
- [IOVisor's bpftrace reference guide](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md)
11+
- [IOVisor's bpftrace repository](https://github.com/iovisor/bpftrace)
12+
- [Wikipedia article on TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
13+
- [IOVisor's bpftrace one liner's tutorial](https://github.com/iovisor/bpftrace/blob/b6c4136fabf2527fc736bc08ee875625156b5431/docs/tutorial_one_liners.md)

Diff for: ‎environment/.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*.log
2+
.vagrant/

0 commit comments

Comments
 (0)