Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete for replot #870

Open
wants to merge 5 commits into
base: development
Choose a base branch
from
+33 −0
Open

Delete for replot #870

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fb848ca
Add target def.n for replot single file removal
graemes Jul 15, 2021
6496461
Changelog
graemes Jul 15, 2021
cbe5d6c
Correct double quotes (too helpful editor)
graemes Jul 15, 2021
e7c8032
Use site_root and change example
graemes Jul 15, 2021
db02d4e
Merge branch 'development' into delete_for_replot
graemes Jul 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Allows explicit specification of the plotter executable path if this is preferred over setting the `PATH` environment variable to find the program.
Presently does not support executables other than the expected names (`chia`, and `chia_plot`).
([#823](https://github.com/ericaltendorf/plotman/pull/823))
- New rsyncd target definition which removes a single file after archiving based on a file glob (for replotting)

## [0.5] - 2021-07-07
### Fixed
32 changes: 32 additions & 0 deletions src/plotman/resources/target_definitions.yaml
View file
Original file line number Diff line number Diff line change
@@ -58,6 +58,38 @@ target_definitions:
"${command}" ${options} "${source}" "${url_root}/${relative_path}/"
transfer_process_name: "{command}"
transfer_process_argument_prefix: "rsync://{user}@{host}:{rsync_port}/{site}"
rsyncd_replot:
env:
# A value of null indicates a mandatory option
command: rsync
options: --bwlimit=80000 --preallocate --remove-source-files --skip-compress plot --whole-file
rsync_port: 873
ssh_port: 22
user: null
host: null
site_root: null
site: null
path_suffix: ""
# Provide the file glob to determine the next file to be deleted e.g. "*/plot-k32-2021-0[3-6]-*"
replot_glob: null
disk_space_script: |
#!/bin/bash
set -evx
site_root_stripped=$(echo "${site_root}" | sed 's;/\+$;;')
# printf with %.0f used to handle mawk such as in Ubuntu Docker images
# otherwise it saturates and you get saturated sizes like 2147483647
ssh -p "${ssh_port}" "${user}@${host}" "df -aBK | grep \" $(echo "${site_root_stripped}" | sed 's;/\+$;;')/\" | awk '{ gsub(/K\$/,\"\",\$4); printf \"%s:%.0f\n\", \$6, \$4*1024 }'"
transfer_script: |
#!/bin/bash
set -evx
echo Launching transfer activity
full_destination=$(realpath --canonicalize-missing "${destination}/${path_suffix}")
relative_path=$(realpath --canonicalize-missing --relative-to="${site_root}" "${full_destination}")
url_root="rsync://${user}@${host}:${rsync_port}/${site}"
"${command}" ${options} "${source}" "${url_root}/${relative_path}/"
ssh -p "${ssh_port}" "${user}@${host}" "ls -1U ${site_root}/${replot_glob}.plot | head -n 1 | xargs rm -fv"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line the only change? Could it just be added to the existing targets with replot_glob being optional (and probably a different name) and execute only if specified? Is there a reason to use xargs over command substitution?

Not that this is a problem, but to make sure I understand, the expectation is to delete one plot from any archive drive and that the user will have consistent and separate directories across all archival drives for to-be-deleted plots vs. being-created plots. For example, I have */original/*.plot that I will end up deleting and I am now plotting to */pool.xch<contract address here>/*.plot. So, my replot_glob would be */original/*. Also, in a decade when we all need k33 and our drives just refuse to die we can use this with the plot-k32- as in your example and replace with k33. And in the end, the user just has to think about and be sure to setup the cross-archive-drive consistency or hazard deleting the plots they are creating etc.

Just to write it out, not to say there's a problem, this does require the user to provide a single plot worth of free space prior to this mechanism "working". That does avoid the complexity of the space check having to account for a plot that could be deleted. This seems like a good thing to avoid and it comes at a very low cost, though it does need to be made clear.

Anyways, a good bit of text but mostly just me thinking out loud so you can make sure I'm understanding properly.

As always, thanks for the work and the contribution.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've understood the intent precisely.

The additional 'deletion' line is the only extra - created a separate target for simplicity and was adapted from a comment in the plotting forum without too much consideration.

You're right - the user will need to be aware of what it is doing re: deleting initial plot and construction of the glob but as you've pointed out it is a fairly 'simple' solution.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is easy, and other solutions would probably be hard, I wouldn't push back on this. However, it seems far from intuitive, so I'd like it if everything that's been written up here explaining what's going on is converted into documentation somewhere nearby...?

As for longer-term solutions, I think there are a variety of plot-management tasks which would be cool to eventually support -- replacing OG plots with NFT plots, replacing plots that have won blocks with new plots, migrating from k32 to larger, "defragging" empty space on plot drives, migrating plots from specified drives elsewhere to enable them to be decommissioned, etc. This would really be a "plot management" rather than the "plotting management" system plotman has been, but it would need to interface closely with the traditional plotman.

transfer_process_name: "{command}"
transfer_process_argument_prefix: "rsync://{user}@{host}:{rsync_port}/{site}"
# external_script:
# env:
# some_common_value_with_a_default: /a/path