Wallet v2

[ekzyis]

Description

close #1495 fix #2234

DX for supporting a new wallet: 926c706

TODOs / Overview of Changes

• ✅ refactor vault (Migrate vault entries to new schema #2092)
  • ℹ️ new Vault table no longer contains foreign keys to wallets or users
  • ℹ️ wallets now point to the vault within their config, not separated from it
• ✅ support multiple receive and send protocols per wallet on the backend (Wallet schema v2 #2146)
  • ℹ️ new wallet schema consists of the following tables:
    • WalletTemplate
    • Wallet
    • WalletProtocol
    • table for each send and recv protocol (WalletSendNWC, WalletRecvNWC, ...)
• ✅ store send and receive config in separate tables for easier integrity checks (Wallet schema v2 #2146)
• ✅ JSDoc for protocols
  • ✅ wallets/lib (shared by client and server)
  • ✅ wallets/client
  • ✅ wallets/server
• ✅ separate wallets from protocols in frontend for better UX
  • ℹ️ we now show cards of wallets which have protocols inside of them
  • ℹ️ users can easier find a wallet to attach by just looking at their names/logos, instead of having to know if they have a wallet that supports a protocol that we also support
• ✅ create a list of wallets with the protocols they support
  • ✅ insert wallets with protocol support into static WalletTemplate table
  • ✅ double-check protocol support per wallet and if I missed any
  • ❌ add placeholder wallets like Phoenix?
  • ✅ proper input labels, placeholder, help etc. (compare with master)
• ✅ restructure wallets/ folder
  • ℹ️ client/server wallet imports are now split on the highest level: client only imports stuff from wallets/client/ and server only from wallets/server/
  • ℹ️ everything (exclusively) wallet related, including validation, was moved to the wallets/ folder
• ✅ wallet I/O
  • ✅ load plaintext wallets
  • ✅ load encrypted wallets
  • ✅ save plaintext wallets
  • ✅ save encrypted wallets
  • ✅ client-side form validation
  • ✅ server-side form validation
  • ✅ wallet network tests
    • ✅ payments (Use test payments with HODL invoices for wallet validation #1287)
    • ✅ invoices
  • ✅ save LNC (requires generating other fields)
  • ✅ create new wallets from template
  • ✅ fix LND gRPC macaroon transform to invalid value 🤔
• ✅ wallet API
  • ✅ payments
  • ✅ invoices
• ✅ update existing code to use wallet v2 API
  • ✅ send p2p zap
  • ✅ receive p2p zap
  • ✅ lightning address
  • ✅ autowithdrawals
  • ✅ manual withdrawals
• ✅ let users search for their wallet
  • ℹ️ (fuzzy) search should have autocomplete for wallet names
• ❓ include wizard / multi-step forms probably not in this PR
• ✅ wallet encryption
  • ✅ remove the requirement to have device sync enabled for a gun
    • ℹ️ wallets are now always encrypted and saved on server
  • ✅ generate key by default
  • ✅ show locked wallets if wrong key and prompt to enter passphrase
  • ✅ encrypt wallets with new key on passphrase export
  • ✅ reset option if passphrase lost
• ✅ wallet migration
  • ✅ migration of old CryptoKey
  • ✅ migrate local wallets (device sync was not enabled)
  • ✅ merge local wallets with existing wallets during migration
    • ℹ️ local wallets currently create a wallet separate from existing receiving credentials
• ✅ wallet status
  • ℹ️ set wallet send/recv status via trigger instead of deriving from logs
• ✅ wallet logs
  • ✅ always store all wallet logs on the server
    • ℹ️ no code to manage different sources
    • ℹ️ consistent logs across all devices
  • ❌ show status of wallet network tests on attach in dedicated UI instead of relying on wallet logs
  • ❌ don't show "logs", show wallet events like a wallet
    • ℹ️ this means that there won't be multiple log messages for the same payment. we will just show the current status of a payment, just like a wallet would do.
  • ✅ wallet logs for templates
  • ✅ pagination
  • ✅ log message context
• ✅ wallet settings
  • ✅ detach
  • ✅ enabled
  • ✅ priority via DnD
  • ✅ priority via DnD on mobile
  • ✅ (global) limits (dust, fee, balance)
• ✅ create another wallet from a template when there's already one
• ✅ make sure state is updated when a wallet is updated (WALLETS vs WALLET query)
  • ✅ refetch WALLET on save or detach
  • ✅ refetch WALLETS if wallets changed
• ✅ handle device-dependant wallets like WebLN
• ✅ append and validate domain in lightning address input
• ✅ make sure this fixes duplicate lost hat/horse when wallet is disabled #2234
• ✅ update lightning address form in wallet receive prompt
• ✅ update code to pay QR code with WebLN
• ✅ split CUSTOM wallet into NWC and LN_ADDR wallet
• ✅ handle IndexedDB not available
• ✅ Q&A:
  • ❓ can key hash and wallet encryption get out of sync?
    • ℹ️ in that case, the passphrase prompt would ask for the wrong key
    • ✅ update: we now always compare key hashes on the server
  • ✅ payments
  • ✅ migration

Test instructions

• a wallet seed (docker/db/wallet-seed.sql) is loaded into the database on init
• wallet seed is also used to test migration
• wallet seed contains multiple wallets for user test_wallet_v2 (id 21001)
• login as user: sndev login test_wallet_v2
• passphrase to unlock wallets:

media fit youth secret combine live cupboard response enable loyal kitchen angle

Screenshots

we now show wallets instead of protocols:

each wallet can now support multiple protocols for send or receive:

Checklist

Are your changes backwards compatible? Please answer below:

no

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:

8

• migration was tested with wallet seed and prod db dump, manually inspecting the database to make sure the migrated data makes sense and logging in as various users (even though I can't unlock their wallets since they are encrypted)
• everything else was tested manually via the frontend (payments, wallet I/O, withdrawals, encryption & decryption, passphrase reset etc.)

For frontend changes: Tested on mobile, light and dark mode? Please answer below:

yes

Did you introduce any new environment variables? If so, call them out explicitly here:

no

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[ekzyis]

I refactored the wallet reducer in 8be0b30.

Instead of storing page + loading in the state, I noticed what I actually wanted was to have status. This meant I could also remove router events to "reset the page" when we left /wallets. I can just use a regular button to go from 'attach wallets' to the list of wallets instead of "changing the page."

I also fixed the flicker of 'attach wallets' -> 'unlock wallets' or 'attach wallets' -> list of wallets in 8be0b30 by adding a loading spinner while we're still determining what we want to show:

With this refactor, I can now easily add another status when the wallets are unavailable instead of thinking in pages in the state.

[ekzyis]

I now show this message to the user if IndexedDB is not supported:

[ekzyis]

Okay, I’m done here now. As far as I am concerned, this can be merged!

[huumn]

Edit: I fixed this.

When I attempt to migrate prod seed:

Migration name: 20250702000001_wallet_v2

Database error code: 22P02

Database error:
ERROR: invalid input value for enum "LogLevel": "WARN"

DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState(E22P02), message: "invalid input value for enum \"LogLevel\": \"WARN\"", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("enum.c"), line: Some(129), routine: Some("enum_in") }

[huumn]

On brand new accounts, I'm being asked to unlock wallets

[ekzyis]

On brand new accounts, I'm being asked to unlock wallets

Ohhh, good catch! There is a race condition in useKeyInit. The useEffect is called twice (not sure yet if that's also caused by strict mode), so if there's no key yet, we might call generateRandomKeytwice. The server saves the first key, whereas the client overwrites it with the second key, leading to a key mismatch.

---

Update

Pretty sure it's because of strict mode because the second console.log is dimmed:

If you have React DevTools installed, any console.log calls during the second render call will appear slightly dimmed. React DevTools also offers a setting (off by default) to suppress them completely.

diff --git a/wallets/client/context/hooks.js b/wallets/client/context/hooks.js
index 99280325..025feb9b 100644
--- a/wallets/client/context/hooks.js
+++ b/wallets/client/context/hooks.js
@@ -121,6 +121,8 @@ export function useKeyInit () {
   const loadKey = useLoadKey()
   const loadOldKey = useLoadOldKey()

+  console.log('useKeyInit')
+
   useEffect(() => {
     if (!me?.id) return

I read this now. As mentioned in that link, this is how we can deal with strict mode issues in development:

useEffect(() => {
  let mounted = true
  async function effect() {
    await Promise.resolve()
    if (!mounted) return
    // the following code will only run after first cleanup in strict mode
    ...
  }
  effect()
  return () => { mounted = false }
}, dependencies)

I've tested this by logging mounted before and after await Promise.resolve() and mounted does indeed always change from true to false in the first effect.

So by calling await Promise.resolve() in the effect before checking mounted, we can be sure that we've waited for an immediate cleanup.

This looks like a hack / working against React Strict Mode instead of with it, but I think this is basically Fetching data with Effects with waiting for the cleanup before triggering the request, instead of only ignoring the response. So imo, doing this is fine when a bug is really only caused by strict mode and we know that there's no bug it's trying to reveal.

---

Update

Mhh, it actually is trying to reveal a bug: the code is prone to race conditions. To fix that, we would need to read and write the key in one transaction in a way that the second transaction does not read the key before we wrote it. Will check if IndexedDB has such functionality.

[huumn]

IME: let strict mode help you. Do not hack around it.

It's particularly tricky with this stateful/sideeffect initialization stuff. We need to learn a decent pattern around it imo.

[ekzyis]

IME: let strict mode help you. Do not hack around it.

It's particularly tricky with this stateful/sideeffect initialization stuff. We need to learn a decent pattern around it imo.

Yeah, I agree. I am using a single transaction in readwrite mode to read and write the value now in 9141050.

Since I open and close a database for each call in my refactored useIndexedDB hook, I actually couldn't use it because each transaction would run in a separate connection so readwrite mode wouldn't block each other as mentioned here. So I wrote custom IndexedDB code in useKeyInit so I can have the same database and run two queries within the same transaction ...

But the issue doesn't happen now anymore for another reason: Since I now have to check if the db is set before I run the queries and since afaict, the queries only run after strict mode went through "init db -> cleanup -> init db", the queries don't run twice at the same time anymore.

I think I will have to rethink how to wrap IndexedDB properly in a hook so it doesn't limit us in important ways. Or just use a library at this point.

[huumn]

The original useIndexedDB was basically a light version of what a library would provide, scoped for our needs.