Saving & Loading to third party sites

[Envian]

I'm currently looking into writing guides and sharing more plans, and have need of shorter share links. To serve this end, I'm looking into adding support for (one or more) third party sites. I do see Gists on your roadmap, so I would like to build that feature out.

I will add further details over time, and feel free to ask any questions.

TOS considerations

First and foremost, we need to consider the TOS's of the site(s) we want to use. Making use of Github to host our apps backend is sketchy at best, but isn't explicitly forbidden. For the initial implementation, I will start with Github gists, but other sites may be considered down the line.

File Format

The easiest file format would simply be the full plan link URL. This has the benefit of including the URL of the site within the file itself. Alternatively, a pretty printed JSON file is more likely to be acceptable by github.

Can I get your thoughts on file format?

GUI Implementation

• Authorize with Github button will be added to both the Save & Load menus under a "GitHub Gist" tab. When authorized, this is replaced with a label showing the logged in user, and a Deauthorize button.
• The save menu will allow you to name the gist. Gists will be saved with the MIME type of application/x-xivplan for identification later. Gists will be private by default.
• The load menu will show all gists with the MIME type application/x-xivplan.
• The load menu will also include a text box "Load from URL" to load public plans*.
• The share menu will be updated to include a Share Gist tab, which is only visible when working on a gist. This tab will have a button to allow the user to make the gist public, and when public the share URL will be shown.

* I've been floating the idea of a generic "Load from URL" option that simply performs a GET to that URL and loads what it finds, but it feels like a bit too much of a power user feature, when plan codes exist.

Link Format

Currently planning on only adding a hash based URL. It will look something like this:

https://xivplan.netlify.app/#/gist/github-gist-id

GitHub Authorization Quirks

Github authorization is fairly straight forward - however one quirk is that we need to navigate away from the page in order to authorize, then we will be redirected back. This will clear any unsaved state, so I plan on adding a temporary save for the scene. It will be stored on local storage.

[Envian]

Also, while I am fully ready to implement this completely, as you may be aware node & react is not my area of expertise. If you'd prefer to limit my contributions, I would not be offended.

[joelspadin]

TOS

Definitely something we should check. I hadn't looked into this yet, but I figured gists are meant for sharing code snippets, notes, etc. which isn't far off from how this would use them.

File Format

I figured pretty-printed JSON would probably be best for this. If it's possible to get the ID of a new gist before creating it, we could include the xivplan URL to that plan in the data, either as a header that we strip out before parsing, like

// https://xivplan.netlify.app/gist/<id>
{ 
  ... 
}

or as a field in the data like

{
  "$url": "https://xivplan.netlify.app/gist/<id>",
  ...
}

so that it remains standard JSON.

GUI
Authorize with Github button will be added to both the Save & Load menus under a "GitHub Gist" tab.

This seems reasonable to me. Alternatively, we could have a sign in button at the top of the Save and Load menus and then only show the relevant tabs if you're signed in, but I'm not sure if that's easier to use or just more confusing.

Gists will be saved with the MIME type of application/x-xivplan for identification later.

I haven't checked what the specification for MIME types says, but it looks like a lot of specialized JSON formats tack +json onto the name, so maybe application/x-xivplan+json?

The load menu will show all gists with the MIME type application/x-xivplan.

Exactly what I was thinking (aside from not having decided a good way to identify which gists should be shown; I like the idea of using the MIME type).

The load menu will also include a text box "Load from URL" to load public plans*.

I'm not sure how necessary this is when you can just open the link in a new tab.

The share menu will be updated to include a Share Gist tab, which is only visible when working on a gist. This tab will have a button to allow the user to make the gist public, and when public the share URL will be shown.

Seems reasonable to me. Maybe the load menu should also display the public/private status and have a share/copy link button as well?

Link Format

That pretty much matches what I was thinking. From what I understand, non-hash-based URLs are generally preferred, though since xivplan is a static site with no other pages currently, using a hash is fine (and consistent with the share links, where I had to use a hash to work around server URL limits).

GitHub Authorization Quirks

It is possible to do OAuth authentication in a popup window. Not sure this is the best possible example of it, but I've used code similar to https://github.com/jlumbroso/react-github-login for this before.

Also note that this requires having a separate server set up to relay the OAuth messages with the app's private API key (otherwise there's no way to keep said key private). I'd need to look into whether that could be done using Netlify's serverless functions instead, which should be free (as long as I don't get more than 125k sign ins per month).

[Envian]

TOS

Github's API TOS doesn't explicitly forbid much. Just illegal activity, spam, advertisements. etc.

File Format

I'm a fan of the $url field. while we won't know what a file's URL will be until its created, we can create the file empty, and get the URL before posting the real content.

we could have a sign in button at the top of the Save and Load menus and then only show the relevant tabs if you're signed in, but I'm not sure if that's easier to use or just more confusing

I like your idea of having an "Authenticate with Github" option somewhere, and hiding the gist tab unless authenticated. Couple that with selecting only a single third party site to integrate with, will keep the UI more clean.

I haven't checked what the specification for MIME types says, but it looks like a lot of specialized JSON formats tack +json onto the name, so maybe application/x-xivplan+json?

Adding +json is an optional suffix we can add - and in theory we can use this to differentiate between application/x-xivplan+json and application/x-xivplan+gzip which would be the URL's format. I do like this plan.

I'm not sure how necessary this is when you can just open the link in a new tab.

The idea behind the "Open URL" input was if you had the link to someone else's gist, but didn't have the XIV Plan link. There wouldn't be an intuitive way to open that gist without constructing the url yourself. However, if we add $url to the plan JSON, that solves this problem.

Seems reasonable to me. Maybe the load menu should also display the public/private status and have a share/copy link button as well?

I did think about that, but that would clutter the load menu. I can see the use case for making a plan public, but private is much more niche. That's why I suggested making publishing a plan a one way operation.

https://github.com/jlumbroso/react-github-login

I took a look at the source code for this project - this would work well so long as the popup is allowed. Unfortunately an in-app popup (Similar to how Open File is a "popup"), would require an IFrame, but there's a good chance that GitHub blocks IFrames for security.

Also note that this requires having a separate server set up to relay the OAuth messages with the app's private API key

Ah, I didn't realize there wasn't a serverless authentication flow. But, in hindsight, this makes sense.

With a relay, we would want to verify the API requests on the server, otherwise we could be forwarding questionable requests. This doesn't necessarily need to be a day 1 feature, but something that should be strongly considered.

[joelspadin]

I took a look at the source code for this project - this would work well so long as the popup is allowed. Unfortunately an in-app popup (Similar to how Open File is a "popup"), would require an IFrame, but there's a good chance that GitHub blocks IFrames for security.

In a previous project, I used an actual popup window rather than an iframe. It worked pretty well. (Code could probably be written much more cleanly though.)

[Daendaralus]

Heya, not sure what the state of this currently is, but I've recently forked and made some changes for myself. Among them sharing via the paste.myst.rs API. From what I can tell, this seems to comply with their TOS. Although it's realistically not nearly as future proof as something like github.

My changes are very quick and dirty (Not helped by 0 experience with TS or react) and as such various parts of the code are a bit mangled. Not particularly valuable as a contribution I believe, but maybe still of use to you.

It's currently hosted on a github pages, here's an example plan: https://daendaralus.github.io/xivplan/#/plan/ttz3wr4o

[Envian]

Hello fellow criterion enjoyer!

Thanks for posting this. I haven't made any notable progress on this ticket due to a change in my professional life - that and the complexity of adding authentication, session management, backend validation, and all the other processes required for secure server to server communication left me demotivated.

paste.myst.rs solves a lot of those problems though. I've reviewed your saving/loading code and the fact that the server allows for creating pastes without authentication is a game changer. I did review their site and agree that using them for storing plans is technically not against their TOS, and can serve as a great starting point for URL based sharing.

I'll see if I can find some time to add a GUI around this which would work with the main package.

[Envian]

Hey @joelspadin, I did some research on pastemyst and, from a technical perspective, I can see two options for implementation. I'd like your feedback on which, if either, approach you feel is more appropriate.

Authenticated API

Authentication with the pastemyst API is a simple bearer token. Users would go to the paste.myst.rs site, connect their github account, and receive an API token. This token can then be copied into the xivplan app to allow us to fully read/write/query pastes on the user's account.

When authenticated, we could have pastemyst be an alternative to local storage, and also double as a sharing system. Plans will be set private by default, with a checkbox to toggle to public in the Share dialog. Unauthenticated users would have the ability to open plans that have been shared by other users using the unauthenticated API.

To mitigate the risk that pastemyst becomes unavailable, I can also add a local storage copy of the plan (Separate from the Local Storage) tab so plans can be restored if we lose access to pastemyst.

Unauthenticated API

Pastemyst also has a public, unauthenticated API. This only allows for reading public pastes, and creating new pastes - existing pastes cannot be edited in any form, and are permanent once uploaded.

If we implement this approach, in addition to a simple "Share" functionality, I would add logic to keep track of a plan's current share link, and add checks so we only upload a new copy when necessary, and a plan is only uploaded when explicitly requested. A quick mockup of the GUI, with appropriate disclaimers, looks like this:

In both versions of the API, we are limited to 5 request per second, which is plenty to work with. Each of the API requests will be coming from the user's browser rather than a central server.

[joelspadin]

Sorry, I forgot about this until recently. I don't think I'm going to have time to look into this before the expansion launches (I blame From Software), but I did just finish implementing some support for a feature that I hope will make it a bit easier to share large plans.

The site can now be installed as an app, and on Chromium-based browsers (Chrome, Edge, etc.) it now supports opening and saving JSON files with an .xivplan file extension. You can now save a file and send it to someone (e.g. via Discord), and if they have the site installed as an app, they can directly open the file in the app. I plan to allow for drag and drop of .xivplan files onto the page and saving them via download to at least get some support for Firefox next.

That doesn't solve the problem of having a single location for a plan that you know is the latest version though, so I would still like to work on this eventually.

[Envian]

Hey Joel, I've taken a look at the changes, and the file support and this is all great! it has certainly helped with sharing plans within a community.

I just want to clarify one thing though: my goal with opening this issue was to enable frictionless sharing with strangers, such as in party finder, or in a strats pastebin. In theory, the links serve this purpose, but having a plan link longer than the discord character limit is a bit... awkward.

Currently I'm working around this by using tinyurl to shorten plan links - and that works great, but people don't trust tinyurl, so a built in solution would be preferred. Unfortunately I don't think this will be possible without a backend of some sort.

Even though i proposed this change, I'm not 100% comfortable relying on the generosity of strangers for hosting plans - hence the disclaimers that I included in my mockups. Maybe the solution here is to fork the project and add support user login and a backend, allowing for a traditional server structure. This is not a priority for me though, so I do not plan to work on it any time soon.

[joelspadin]

Now that I have support for storing plans as files, I wonder if a better approach to sharing them would be to use a service that's explicitly for cloud file storage and sharing, like dropbox, google drive, etc. For that to work, you would need to authenticate with the service so XIVPlan can write files, and then sharing would give you a link like https://xivplan.netlify.app/dropbox/<id>, which would allow anyone else to view the file (as long as it has the correct permissions to be publicly viewable).

That would also work well with #32 if these links opened in a read-only mode by default (with an option to switch back to the editor interface and save a copy of it yourself if you want).

[NinjaTertel]

I feel like the Dropbox/gDrive would be a better approach that doesnt include setting up your own server to store created raidplans.
Maybe to make it easier: XIVPlan wouldn't need to have write access to the service. If XIVPlan adds support through loading the url like in your example (https://xivplan.netlify.app/dropbox/<id>), then XIVPlan can just download the file from the ID, parse and automatically import it the same way as drag-and-drop works. The only requirement a user/raidplan creator would need is the file would need to be publicly available.

Great tool BTW 👍