Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add implementation of unified vault format (UVF). #16623

Draft
wants to merge 39 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
d1995d7
First ideas for uvf imple.
chenkins Dec 6, 2024
dee75a2
Switch to uvf draft snapshot dependency.
ylangisc Jan 6, 2025
61000a2
Migrate API changes.
ylangisc Jan 18, 2025
010a4b9
Extract filename cryptor and provider.
ylangisc Jan 18, 2025
007d66d
Intermediate interface for Cryptomator stuff.
ylangisc Jan 20, 2025
f6550ea
More extraction.
ylangisc Jan 20, 2025
b93c1b4
More extraction.
ylangisc Jan 20, 2025
e5af13a
Review.
ylangisc Feb 6, 2025
1330c34
Fix UnsupportedOperationException for fileNameCryptor() without revis…
chenkins Feb 6, 2025
1eb0db6
Bump version to 9.1.3.uvfdraft-SNAPSHOT.
chenkins Feb 8, 2025
aedf1bd
Temporarily skip SFTPCryptomatorInteroperabilityTest because of depen…
chenkins Feb 11, 2025
4ed0381
Switch to byte array for directory ids to avoid lossy data conversion.
ylangisc Feb 24, 2025
5baa253
Directory id for root is derived from key material.
ylangisc Feb 25, 2025
6d2ce9f
Fix cryptolib uvfdraft version.
chenkins Feb 25, 2025
e977086
Fix typo.
ylangisc Feb 26, 2025
29adf98
Metadata file/directory extension is .uvf.
ylangisc Feb 27, 2025
ac3512d
Fix compile.
chenkins Feb 27, 2025
ad5c784
Fix base64 URL pattern.
ylangisc Feb 27, 2025
8f8437d
Fix pattern.
ylangisc Feb 27, 2025
13f3fa9
Delete unused.
dkocher Mar 6, 2025
04eeba6
Require to pass UVF metadata payload in password callback.
dkocher Mar 6, 2025
278165b
Destroy on close.
dkocher Mar 6, 2025
354f2eb
Add UVF test listing files from vault created with cryptolib.
chenkins Mar 7, 2025
dba0ec7
Add category TestcontainerTest.
chenkins Mar 7, 2025
9907923
Fix hostname.
chenkins Mar 7, 2025
a865048
Simplify setup for uvf integration test.
chenkins Mar 8, 2025
e7fb181
Use expected files.
chenkins Mar 8, 2025
01539ae
Fix rootDirId handling.
ylangisc Mar 9, 2025
f486498
Implement loading dir.uvf using file content encryption instead of ra…
chenkins Mar 9, 2025
4d804f3
Verify root dir ID.
chenkins Mar 9, 2025
edf148d
Use revision from file header in CryptoDirectoryUVFProvider.
chenkins Mar 9, 2025
ca9b7bf
Implement decrypt respecting revision.
chenkins Mar 11, 2025
7d7f454
Rename.
chenkins Mar 11, 2025
894d524
Read file in subdir with current revision.
chenkins Mar 11, 2025
4684dbb
Add test write and read file UVF.
chenkins Mar 13, 2025
485f079
Add test write UVF ile to subdir.
chenkins Mar 13, 2025
bf67c96
Add test move UVF file.
chenkins Mar 13, 2025
44d8e2a
Use set equals.
chenkins Mar 13, 2025
703d9b7
Cleanup.
chenkins Mar 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Read file in subdir with current revision.
  • Loading branch information
chenkins committed Mar 11, 2025
commit 894d5248402ef2fc21d88933cf6725168100a912
Original file line number Diff line number Diff line change
@@ -176,6 +176,69 @@ public Path decrypt(final Session<?> session, final Path file) throws Background
}
}

public Path encrypt(Session<?> session, Path file, byte[] directoryId, boolean metadata) throws BackgroundException {
final Path encrypted;
if(file.isFile() || metadata) {
if(file.getType().contains(Path.Type.vault)) {
log.warn("Skip file {} because it is marked as an internal vault path", file);
return file;
}
if(new SimplePathPredicate(file).test(this.getHome())) {
log.warn("Skip vault home {} because the root has no metadata file", file);
return file;
}
final Path parent;
final String filename;
if(file.getType().contains(Path.Type.encrypted)) {
final Path decrypted = file.attributes().getDecrypted();
parent = this.getDirectoryProvider().toEncrypted(session, decrypted.getParent().attributes().getDirectoryId(), decrypted.getParent());
filename = this.getDirectoryProvider().toEncrypted(session, parent.attributes().getDirectoryId(), decrypted.getName(), decrypted.getType());
}
else {
parent = this.getDirectoryProvider().toEncrypted(session, file.getParent().attributes().getDirectoryId(), file.getParent());
// / diff to AbstractVault.encrypt
String filenameO = this.getDirectoryProvider().toEncrypted(session, parent.attributes().getDirectoryId(), file.getName(), file.getType());
filename = ((CryptoDirectoryUVFProvider) this.getDirectoryProvider()).toEncrypted(session, file.getParent(), file.getName());
// \ diff to AbstractVault.decrypt
}
final PathAttributes attributes = new PathAttributes(file.attributes());
attributes.setDirectoryId(null);
if(!file.isFile() && !metadata) {
// The directory is different from the metadata file used to resolve the actual folder
attributes.setVersionId(null);
attributes.setFileId(null);
}
// Translate file size
attributes.setSize(this.toCiphertextSize(0L, file.attributes().getSize()));
final EnumSet<Path.Type> type = EnumSet.copyOf(file.getType());
if(metadata && this.getVersion() == VAULT_VERSION_DEPRECATED) {
type.remove(Path.Type.directory);
type.add(Path.Type.file);
}
type.remove(Path.Type.decrypted);
type.add(Path.Type.encrypted);
encrypted = new Path(parent, filename, type, attributes);
}
else {
if(file.getType().contains(Path.Type.encrypted)) {
log.warn("Skip file {} because it is already marked as an encrypted path", file);
return file;
}
if(file.getType().contains(Path.Type.vault)) {
return this.getDirectoryProvider().toEncrypted(session, this.getHome().attributes().getDirectoryId(), this.getHome());
}
encrypted = this.getDirectoryProvider().toEncrypted(session, directoryId, file);
}
// Add reference to decrypted file
if(!file.getType().contains(Path.Type.encrypted)) {
encrypted.attributes().setDecrypted(file);
}
// Add reference for vault
file.attributes().setVault(this.getHome());
encrypted.attributes().setVault(this.getHome());
return encrypted;
}

private int loadRevision(final Session<?> session, final Path directory) throws BackgroundException {
// Read directory id from file
log.debug("Read directory ID from {}", directory);
Original file line number Diff line number Diff line change
@@ -37,6 +37,8 @@
import java.nio.charset.StandardCharsets;
import java.util.EnumSet;

import com.google.common.io.BaseEncoding;

public class CryptoDirectoryUVFProvider extends CryptoDirectoryV7Provider {
private static final Logger log = LogManager.getLogger(CryptoDirectoryUVFProvider.class);

@@ -47,10 +49,12 @@ public class CryptoDirectoryUVFProvider extends CryptoDirectoryV7Provider {
= new UUIDRandomStringService();
private final Path dataRoot;
private final CryptorCache filenameCryptor;
private final CryptoFilename filenameProvider;

public CryptoDirectoryUVFProvider(final AbstractVault vault, final CryptoFilename filenameProvider, final CryptorCache filenameCryptor) {
super(vault, filenameProvider, filenameCryptor);
this.filenameCryptor = filenameCryptor;
this.filenameProvider = filenameProvider;
this.home = vault.getHome();
this.vault = vault;
this.dataRoot = new Path(home, "d", home.getType());
@@ -64,6 +68,20 @@ protected byte[] toDirectoryId(final Session<?> session, final Path directory, f
return super.toDirectoryId(session, directory, directoryId);
}

// interface mismatch: we need parent path to get dirId and revision from dir.uvf
public String toEncrypted(final Session<?> session, final Path parent, final String filename) throws BackgroundException {
if(new SimplePathPredicate(home).test(parent)) {
final String ciphertextName = filenameCryptor.encryptFilename(BaseEncoding.base64Url(), filename, vault.getRootDirId()) + vault.getRegularFileExtension();
log.debug("Encrypted filename {} to {}", filename, ciphertextName);
return filenameProvider.deflate(session, ciphertextName);

}
final byte[] directoryId = load(session, parent);
final String ciphertextName = vault.getCryptor().fileNameCryptor(loadRevision(session, parent)).encryptFilename(BaseEncoding.base64Url(), filename, directoryId) + vault.getRegularFileExtension();
log.debug("Encrypted filename {} to {}", filename, ciphertextName);
return filenameProvider.deflate(session, ciphertextName);
}

@Override
public Path toEncrypted(final Session<?> session, final byte[] directoryId, final Path directory) throws BackgroundException {
if(!directory.isDirectory()) {
@@ -99,6 +117,9 @@ public Path toEncrypted(final Session<?> session, final byte[] directoryId, fina
}

protected byte[] load(final Session<?> session, final Path directory) throws BackgroundException {
if(new SimplePathPredicate(home).test(directory)) {
return vault.getRootDirId();
}
final Path parent = this.toEncrypted(session, directory.getParent().attributes().getDirectoryId(), directory.getParent());
final String cleartextName = directory.getName();
final String ciphertextName = this.toEncrypted(session, parent.attributes().getDirectoryId(), cleartextName, EnumSet.of(Path.Type.directory));
Original file line number Diff line number Diff line change
@@ -20,6 +20,7 @@
import ch.cyberduck.core.features.AttributesFinder;
import ch.cyberduck.core.features.Bulk;
import ch.cyberduck.core.features.Delete;
import ch.cyberduck.core.features.Read;
import ch.cyberduck.core.features.Write;
import ch.cyberduck.core.io.StatusOutputStream;
import ch.cyberduck.core.proxy.ProxyFactory;
@@ -49,6 +50,7 @@
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.AbstractMap;
import java.util.Arrays;
import java.util.Collections;
@@ -99,11 +101,11 @@ public void listMinio() throws BackgroundException, IOException {
new S3BucketCreateService(storage).create(bucket, "us-east-1");

final List<String> files = Arrays.asList(
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/rExOms183v5evFwgIKiW0qvbsor1Hg==.uvf/dir.uvf",
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/dir.uvf",
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/GsMMTRvsuuP_6NjgRwopmWcuof-PyRQ=.uvf",
"/d/TU/EVUUXMHY2HHNQ4BLKNE3GBLEFD4YW6/4RuVMuXcOTOfhSQZAwEV1E4XiNrMVOY=.uvf",
"/d/TU/EVUUXMHY2HHNQ4BLKNE3GBLEFD4YW6/dir.uvf"
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/rExOms183v5evFwgIKiW0qvbsor1Hg==.uvf/dir.uvf", // -> /subir
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/dir.uvf", // -> /
"/d/RZ/K7ZH7KBXULNEKBMGX3CU42PGUIAIX4/GsMMTRvsuuP_6NjgRwopmWcuof-PyRQ=.uvf", // -> /foo.txt
"/d/6L/HPWBEU3OJP2EZUCP4CV3HHL47BXVEX/5qTOPMA1BouBRhz_G7qfmKety92geI4=.uvf", // -> /subdir/bar.txt
"/d/6L/HPWBEU3OJP2EZUCP4CV3HHL47BXVEX/dir.uvf" // /subdir
);
final String jwe = "{\n" +
" \"fileFormat\": \"AES-256-GCM-32k\",\n" +
@@ -152,14 +154,32 @@ public Credentials prompt(final Host bookmark, final String title, final String
{
final AttributedList<Path> list = storage.getFeature(ListService.class).list(home, new DisabledListProgressListener());
assertEquals(2, list.size());
assertTrue(Arrays.toString(list.toArray()), list.contains(new Path("/cyberduckbucket/foo.txt", EnumSet.of(AbstractPath.Type.file, AbstractPath.Type.decrypted))));
final Path foo = new Path("/cyberduckbucket/foo.txt", EnumSet.of(AbstractPath.Type.file, AbstractPath.Type.decrypted));
assertTrue(Arrays.toString(list.toArray()), list.contains(foo));
assertTrue(Arrays.toString(list.toArray()), list.contains(new Path("/cyberduckbucket/subdir", EnumSet.of(AbstractPath.Type.directory, AbstractPath.Type.placeholder, AbstractPath.Type.decrypted))));

final byte[] buf = new byte[300];
final TransferStatus status = new TransferStatus();
try(final InputStream inputStream = storage.getFeature(Read.class).read(foo, status, new DisabledConnectionCallback())) {
int l = inputStream.read(buf);
assertEquals(9, l);
assertEquals("Hello Foo", new String(Arrays.copyOfRange(buf, 0, l)));
}
}
{
final PathAttributes subdir = storage.getFeature(AttributesFinder.class).find(new Path("/cyberduckbucket/subdir", EnumSet.of(AbstractPath.Type.directory, AbstractPath.Type.placeholder, AbstractPath.Type.decrypted)));
final AttributedList<Path> list = storage.getFeature(ListService.class).list(new Path("/cyberduckbucket/subdir", EnumSet.of(AbstractPath.Type.directory, AbstractPath.Type.placeholder, AbstractPath.Type.decrypted)).withAttributes(subdir), new DisabledListProgressListener());
assertEquals(1, list.size());
assertTrue(Arrays.toString(list.toArray()), list.contains(new Path("/cyberduckbucket/subdir/bar.txt", EnumSet.of(AbstractPath.Type.file, AbstractPath.Type.decrypted))));
final Path bar = new Path("/cyberduckbucket/subdir/bar.txt", EnumSet.of(AbstractPath.Type.file, AbstractPath.Type.decrypted));
assertTrue(Arrays.toString(list.toArray()), list.contains(bar));

final byte[] buf = new byte[300];
final TransferStatus status = new TransferStatus();
try(final InputStream inputStream = storage.getFeature(Read.class).read(bar, status, new DisabledConnectionCallback())) {
int l = inputStream.read(buf);
assertEquals(9, l);
assertEquals("Hello Bar", new String(Arrays.copyOfRange(buf, 0, l)));
}
}
}
finally {
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.