New Map API

[dave-tucker]

Following the discussion on #45 and since I started work on #70 it's become clear that the current Map API needs some work.
We have 3 use cases I can think of from userspace

1. I am loading an eBPF file which describes maps and I wish to be able to get a mutable/immutable reference to them in my program (implemented)
2. I wish to create a map from userspace, and register this definition with the BpfLoader such that any loaded programs use the map that I've created (not implemented)
3. I wish to load a map from a pinned location on bpffs for use in my userspace code (not implemented)
   a. As (3) but register the map with the BpfLoader so it can be used when loading my program, which was implemented in Add support for map and program pinning #45

Creating a Map from Userspace

The simple case would be to re-implement what we have in aya-bpf

let mymap = HashMap<u32,u32>::with_max_entries(1024, 0);
let my_pinned_map = HashMap<u32,u32>::pinned(1024, 0);

We could however implement a builder style api, which I think is a little more expressive:

let mymap : HashMap<u32, u32> = MapBuilder<HashMap<u32, u32>>::new().max_entries(1024).pinned().build();

Registering Maps to a BpfLoader

use aya::{BpfLoader, Btf};
use std::fs;

let bpf = BpfLoader::new()
    // load the BTF data from /sys/kernel/btf/vmlinux
    .btf(Btf::from_sys_fs().ok().as_ref())
    // load map
    .map(mymapp)
    // finally load the code
    .load_file("file.o")?;

In order for .map() to work, our HashMap<u32,u32> needs to be able to be inserted in to a Vec<Map>.

Given that #70, might need to implement a special case of create - because i believe that we may also need to create dummy inner maps to make the verifier happy - I'm wondering whether it makes sense for Map to become a trait instead.

Loading a Pinned Map from Userspace

let mymap = HashMap<u32,u32>::load("/path/to/map/on/bpffs");

This is a little trickier to implement in practice because we don't have the bpf_map_def ahead of time.

For load:

• We can use bpf_obj_get to get the fd
• We can then use bpf_obj_get_info_by_fd to get the bpf_map_info struct, which appears to be a superset of bpf_map_def

Moving all Map code to a new aya-map module

Seeing as creating maps from userspace will offer an almost identical API to the aya-bpf module, I wonder whether it might be worth moving all map code to it's own module called aya-map 🤔

This would allow us to provide a consistent API from both userspace and kernel space - we can use attributes #[cfg(feature = "user")] or #[cfg(feature = "ebpf")] to implement functions where the implementation needs to differ. In some ways this would make #36 easier, since I believe we already have some mocks in place for maps which could be re-used when testing eBPF code 🤔

[alessandrod]

Following the discussion on #45 and since I started work on #70 it's become clear that the current Map API needs some work. We have 3 use cases I can think of from userspace

1. I am loading an eBPF file which describes maps and I wish to be able to get a mutable/immutable reference to them in my program (implemented)

2. I wish to create a map from userspace, and register this definition with the `BpfLoader` such that any loaded programs use the map that I've created (not implemented)

What would be the purpose of registering the map with BpfLoader? When and from where would the registered map be accessed?

3. I wish to load a map from a pinned location on bpffs for use in my userspace code (not implemented)
   a. As (3) but register the map with the `BpfLoader` so it can be used when loading my program, which was implemented in [Add support for map and program pinning #45](https://github.com/aya-rs/aya/issues/45)

Creating a Map from Userspace

The simple case would be to re-implement what we have in aya-bpf

let mymap = HashMap<u32,u32>::with_max_entries(1024, 0);
let my_pinned_map = HashMap<u32,u32>::pinned(1024, 0);

We could however implement a builder style api, which I think is a little more expressive:

let mymap : HashMap<u32, u32> = MapBuilder<HashMap<u32, u32>>::new().max_entries(1024).pinned().build();

Without going into API details yet, generally speaking whenever there's a need for a builder, I think we should have both the builder and a simplified API for the common cases implemented on top of the builder. Having to type the whole builder thing every time where one could use HashMap::with_max_entries(1024) seems not ideal.

Registering Maps to a BpfLoader

use aya::{BpfLoader, Btf};
use std::fs;

let bpf = BpfLoader::new()
    // load the BTF data from /sys/kernel/btf/vmlinux
    .btf(Btf::from_sys_fs().ok().as_ref())
    // load map
    .map(mymapp)
    // finally load the code
    .load_file("file.o")?;

Can you explain what's the use case for this? I can see how we could implement pinned maps this way, but I think I prefer the current approach to this, where the user only specifies a path, and then loading happens transparently as with non-pinned maps.

Loading a Pinned Map from Userspace

let mymap = HashMap<u32,u32>::load("/path/to/map/on/bpffs");

This is a little trickier to implement in practice because we don't have the bpf_map_def ahead of time.

For load:

* We can use `bpf_obj_get` to get the fd

* We can then use `bpf_obj_get_info_by_fd` to get the `bpf_map_info` struct, which appears to be a superset of `bpf_map_def`

I think this would work yes

Moving all Map code to a new aya-map module

Seeing as creating maps from userspace will offer an almost identical API to the aya-bpf module, I wonder whether it might be worth moving all map code to it's own module called aya-map 🤔

I don't think this would work. The only thing we'd be able to share is the API signatures, but then we'd have two completely different implementations. Maps in aya-bpf are glorified sugar for building bpf_map_def instances. Maps in aya userspace do syscalls, rwlocks, etc, all stuff that doesn't exist bpf side.

[dave-tucker]

What would be the purpose of registering the map with BpfLoader? When and from where would the registered map be accessed?

So the map created from userspace could be used in the eBPF program (and not recreated by the call to load).

Registering Maps to a BpfLoader

use aya::{BpfLoader, Btf};
use std::fs;

let bpf = BpfLoader::new()
    // load the BTF data from /sys/kernel/btf/vmlinux
    .btf(Btf::from_sys_fs().ok().as_ref())
    // load map
    .map(mymapp)
    // finally load the code
    .load_file("file.o")?;

Can you explain what's the use case for this? I can see how we could implement pinned maps this way, but I think I prefer the current approach to this, where the user only specifies a path, and then loading happens transparently as with non-pinned maps.

I could have an eBPF program that wants to reuse a pinned map, while creating some maps that get pinned to its mao_pin_path.

Moving all Map code to a new aya-map module

Seeing as creating maps from userspace will offer an almost identical API to the aya-bpf module, I wonder whether it might be worth moving all map code to it's own module called aya-map 🤔

I don't think this would work. The only thing we'd be able to share is the API signatures, but then we'd have two completely different implementations. Maps in aya-bpf are glorified sugar for building bpf_map_def instances. Maps in aya userspace do syscalls, rwlocks, etc, all stuff that doesn't exist bpf side.

Yeah maybe this would be too much

[alessandrod]

What would be the purpose of registering the map with BpfLoader? When and from where would the registered map be accessed?

So the map created from userspace could be used in the eBPF program (and not recreated by the call to load).

But used how by the eBPF program? Maps for which we have relocation entries in program code are discovered and loaded automatically. Maps inside other maps are looked up using get() on the outer map (for which we'll have a relocation entry). How would maps registered in the loader be used?

Registering Maps to a BpfLoader

use aya::{BpfLoader, Btf};
use std::fs;

let bpf = BpfLoader::new()
    // load the BTF data from /sys/kernel/btf/vmlinux
    .btf(Btf::from_sys_fs().ok().as_ref())
    // load map
    .map(mymapp)
    // finally load the code
    .load_file("file.o")?;

Can you explain what's the use case for this? I can see how we could implement pinned maps this way, but I think I prefer the current approach to this, where the user only specifies a path, and then loading happens transparently as with non-pinned maps.

I could have an eBPF program that wants to reuse a pinned map, while creating some maps that get pinned to its mao_pin_path.

Don't we already support that today?

[MatteoNardi]

use aya::{BpfLoader, Btf};
use std::fs;

let bpf = BpfLoader::new()
    // load the BTF data from /sys/kernel/btf/vmlinux
    .btf(Btf::from_sys_fs().ok().as_ref())
    // load map
    .map(mymapp)
    // finally load the code
    .load_file("file.o")?;

Currently it's not possible to decide from user-space that a map should be pinned, you must specify it in the eBPF-side map definition.
I admit our use-case for wanting this is a little convoluted, but here it is:

• We have several mostly independent tracing modules, each with its own eBPF program
• They share a pinned map indicating what processes must be tracked
• When running our test-suite, we'd like pinning to be disabled (to avoid modules interfering with each other)

[dave-tucker]

Closing as this is stale.