Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate a skewed workload at shard granularity in mako #7566

Open
dlambrig opened this issue Jul 11, 2022 · 5 comments
Open

Generate a skewed workload at shard granularity in mako #7566

dlambrig opened this issue Jul 11, 2022 · 5 comments
Assignees
Labels
testing simulation, real cluster, and unit tests.

Comments

@dlambrig
Copy link
Contributor

dlambrig commented Jul 11, 2022

Mako's -z option can generate a skewed workload based on Zipf distribution, but this is at the record level. It would be useful to skew workloads over shards, rather than records, so some are busier than others.

A new option -zs could divide the record count by the default shard size. While running, zipfian_next() could select a shard from that set. Then a random record could be selected from that shard.

@dlambrig dlambrig added the testing simulation, real cluster, and unit tests. label Jul 11, 2022
@dlambrig dlambrig self-assigned this Jul 11, 2022
@sfc-gh-xwang
Copy link
Collaborator

I have written a FDB workload SkewedReadWriteWorkload (#7087 ) doing the similar thing.
If mako is not a requirement for your case, it can be helpful.

@dlambrig
Copy link
Contributor Author

dlambrig commented Jul 18, 2022

I will try it, thank you. The reason I use mako is it allows me to create a custom configuration with many ss. I think the workloads only can be used in a simulation environment, if so getting workloads to work outside simulation would be a nice enhancement.

@jzhou77
Copy link
Contributor

jzhou77 commented Jul 18, 2022

I think the workloads only can be used in a simulation environment, if so getting workloads to work outside simulation would be a nice enhancement.

Many workloads can be used outside simulation, e.g., ConsistencyCheck.

@sfc-gh-xwang
Copy link
Collaborator

sfc-gh-xwang commented Jul 18, 2022

This workload can be used outside simulation test.
I used it for many DD rebalance tests on a real cluster, and the commented setting in the toml file is the setting I used in real cluster. To use workload outside simulation, you need some fdbserver -r multitest as described here.

@dlambrig
Copy link
Contributor Author

dlambrig commented Jul 18, 2022

this will help, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing simulation, real cluster, and unit tests.
Projects
None yet
Development

No branches or pull requests

3 participants