"Clean Up Files" Feature #1023
Description
Make sure these boxes are checked before submitting your issue -- thanks for reporting issues back to Parse Server!
- You've met the prerequisites.You're running the latest version of Parse Server.You've searched through existing issues. Chances are that your issue has been reported or resolved before.
One of the features that I liked on the hosted Parse was, in the settings, the button Clean Up Files. This way, every file stored in S3 for example, that wasn't anymore referenced from a PFFile, would be deleted. I liked it specially because it allowed us to save on unused/unneeded resources.
Maybe a Rest call using the master key would be initially enough? In the future, with possible integration with the parse-dashboard?
I know it's lower priority compared to the features/fixes that are being developed, but that would be great to have.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Participants
Activity
[-]Allow deleting unused files[/-][+]"Clean Up Files"[/+][-]"Clean Up Files"[/-][+]"Clean Up Files" Feature[/+]gfosco commentedon Mar 17, 2016
This would be pretty difficult actually, and would need to be built for each specific Files adapter. Right now, there's no 'listing' of what files exist through the adapter.
natario1 commentedon Mar 17, 2016
+1 , agree with the need.
ckarmy commentedon Jul 18, 2016
It's possible to clean the unused files stored in GridStore now?
yorkwang commentedon Sep 28, 2016
+1, It's a very useful feature.
Lokiitzz commentedon Oct 3, 2016
+1, It would be nice.
umair6 commentedon Oct 19, 2016
+1
abdulwasayabbasi commentedon Oct 19, 2016
+1 very much needed
JoseVigil commentedon Nov 21, 2016
+1
55 remaining items
davimacedo commentedon Nov 11, 2020
That's why I'd not go with the script in the api. It will be only a matter of time for people to start complaining about the script not working. The same happened with the push notifications system. It took a long time to have a scalable process because previously it was a single parse server instance trying to handle all pushes.
For this to be scalable in the api, we'd need to to a similar approach to the one in push notifications. Break the files in small sets, put those sets on a queue and run multiple processes consuming the sets and processing one by one. Even though we are talking about something that will be complex to be written and also to be deployed.
mtrezza commentedon Nov 11, 2020
Good points. @dblythy can you find anything reusable in the files utils repo that has been mentioned before?
dblythy commentedon Nov 11, 2020
I had a quick look through it and it seems to use a similar search algorithm as I wrote (lookup schema and look for “File”). I can have a more detailed look at that and also how the push notifications approach is done and work towards a cleanup feature similar to that.
c0sm1cdus7 commentedon May 12, 2021
Was this ever implemented?
dblythy commentedon May 12, 2021
The main reason this stalled was because figuring out whether a file is an "orphan" (as in it is not associated to any parent object) is entirely dependant on the way that files are associated with objects. As a file can be set to
object.field,object.field[2].key.field[2], can be associated by relations, pointers, etc, it becomes rather difficult for an inbuilt algorithm to decide whether a file is to be deleted or not.If you're familiar with how your Parse Server determines file associations, you can do something similar to this:
dblythy commentedon Aug 20, 2021
I think the main conceptual challenge here is:
I'm thinking:
Parse.Cloud.checkFileParentswhich the developer is responsible for querying for the file in existing data.mtrezza commentedon Aug 20, 2021
Good analysis @dblythy. Could we break this down into a minimum viable feature with some limitations?
dblythy commentedon Aug 20, 2021
I think that’s a good point. Perhaps for most users, being able to have a collection of their files (uploaded by, view count, etc) visible in the dashboard would probably be an improvement.
We could add that the counter will only be accurate for simple data schemas, and leave the deleting of files up to the developer.
There could be the potential to bake in some common use cases, such as unique profile picture management.
mtrezza commentedon Aug 20, 2021
I like that very much, like with hashes to not upload the same file multiple times, but reference the existing file? I think that would be a very practical use case. Yes, we can see this has a lot of potential, so getting a very basic first feature version released would be a good start.