Cloning a directory tree

Let's put together some of the concepts in the other guides to write a simple tool. We'll mirror a local directory structure into manta. We'll walk the filesystem at the specified directory, create parallel directories in manta, and upload any files we see along the way. This implementation will be single-threaded since the intention is to explore basic aspects of java-manta.

Below is a listing of a local folder containing some files related to java-manta that we want to back up. It would be easy to tar these files up and upload just archive but that prevents us from being able to access the individual files without having to download the archive and extract them.

.
├── README.md
├── coverage
│   └── jacoco.68652f0639e3244b307bb8b53a2b2152d7bd4870.tar.gz
├── images
│   ├── Manta\ Client\ Architecture.draw.io.xml
│   ├── Manta\ Encrypted\ File\ Request\ Flow.xml
│   └── Manta\ Server\ Architecture.draw.io.xml
├── rendered
│   ├── Manta\ Client\ Architecture.png
│   ├── Manta\ Encrypted\ File\ Request\ Flow.png
│   └── Manta\ Server\ Architecture.png
└── scratch
    ├── JMHBenchmark.java
    ├── MBeanSupervisor.java
    ├── MCT.java
    ├── MantaCredentials.java
    ├── MantaRecursiveDirectoryCreationStrategyTest.java
    └── ScratchIT.java

Generating a project

We'll use the standard Maven quickstart archetype to generate our project:

$ mvn archetype:generate -DgroupId=com.example -DartifactId=directory-upload -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Once the project has been generated add the following dependencies in your pom file:

<dependency>
    <groupId>info.picocli</groupId>
    <artifactId>picocli</artifactId>
    <version>LATEST</version>
</dependency>
<dependency>
    <groupId>com.joyent.manta</groupId>
    <artifactId>java-manta-client</artifactId>
    <version>LATEST</version>
</dependency>

To make option parsing simpler we'll use the excellent picocli project. This project allows us to parse arguments and options using just a few easy-to-read annotations. Don't fret if you're not familiar with picocli since we'll be using the simplest possible features:

@Option for optional arguments, allowing the user to specify whether or not we should follow symlinks
@Parameters for required parameters, specifying an index and description
by implementing Callable<Void>, a standard Java interface indicating that the class has a call method returning Void, picocli will perform option/argument parsing and error handling on our behalf

Go ahead and replace the default App class with the following snippet:

public class DirectoryUpload implements Callable<Void> {

    @Parameters(index = "0", description = "local directory to mirror")
    private String source;

    @Parameters(index = "1", description = "remote path for directory root")
    private String destination;

    @Option(names = {"-l", "--follow-links"})
    private boolean followLinks;

    public static void main(final String[] args) throws Exception {
        CommandLine.call(new DirectoryUpload(), System.err, args);
    }

    @Override
    public Void call() {
        System.out.println("Source: " + source);
        System.out.println("Destination: " + destination);

        // We'll add more code here to actually perform the traversal and copy.

        return null;
    }
}

You should be able to run this main class (either through your IDE or by compiling and running the class using javac and java) and see the arguments printed with their respective labels. Try omitting the parameters and see how picocli prints an error message indicating what inputs are expected:

Missing required parameters: source, destination
Usage: <main class> [-l] <source> <destination>
      source                  local directory to upload
      destination             remote path for directory root
  -l, --follow-links

Let's prepare our client and get ready to actually traverse the local filesystem. Add the following code after the print statements in call:

final EnumSet<FileVisitOption> visitOpts;
if (followLinks) {
    visitOpts = EnumSet.of(FileVisitOption.FOLLOW_LINKS);
} else {
    visitOpts = EnumSet.noneOf(FileVisitOption.class);
}

final ConfigContext config = new ChainedConfigContext(
        new DefaultsConfigContext(),
        new EnvVarConfigContext(),
        new SystemSettingsConfigContext());

client = new MantaClient(config);

We can't send any data to manta without a client so we'll also instantiate one with a flexible configuration which reads from system settings, environment variables, and finally the defaults as described in the client configuration starter guide.

Directly under the client assignment we'll perform a check of the destination directory:

if (client.existsAndIsAccessible(destination)) {
    System.err.println(String.format("Destination directory [%s] already exists!", destination));
    System.exit(1);
}

For the purpose of this example we'll print an error and exit if the destination directory exists but in a future guide we'll get around to comparing timestamps and uploading files only if they've changed.

Relative paths

Adding a few more statements in the call method will give us the structure we need to actually begin to copy files:

final Path sourcePath = Paths.get(source);

try {
    Files.walkFileTree(
        sourcePath,
        visitOpts,
        Integer.MAX_VALUE,
        new Uploader(client, sourcePath, destination));
} catch (IOException e) {
    e.printStackTrace(System.err);
    System.exit(1);
}

System.out.println("Directory upload complete: " + source);

Since we want to transplant the directory tree specified as the source into Manta without regard for the absolute local path we'll obtain a Path object for the source directory. This will allow us to determine relative path names between the files we find and the source directory, simplifying the process of building path names in Manta.

This is all the code we'll need in this method. Now it's time to find some files to upload.

Walking the local filesystem

Finding files within the local filesystem can be achieved in a few different ways in Java, but since we'd like to keep this example simple and easy to read we'll use the built-in FileVisitor interface. This interface vastly simplifies the process of traversing a directory tree, offering four methods which return a FileVisitResult to indicate how iteration should proceed:

preVisitDirectory: called when first visiting a directory. We'll use this method to create the remote directory in Manta.
visitFile: called once for each file in the directory. We'll use this to actually upload the file content.
visitFileFailed: called if the file's attributes could not be read, e.g. due to a permissions issue.
postVisitDirectory: called when we're done iterating through a directory. This method also accepts an exception that indicates why the traversal ended prematurely, e.g. because the directory being iterated through was deleted during iteration.

Since we're only really interested in preVisitDirectory and visitFile we'll extend the SimpleFileVisitor class, an implementation of FileVisitor which either returns FileVisitResult.CONTINUE or rethrows exceptions it encounters. To keep all the code in one place we'll make our implementation a static inner class of DirectoryUpload:

public class DirectoryUpload implements Callable<Void> {

    // public Void call() { ... }

    static class Uploader extends SimpleFileVisitor<Path> {

        private final MantaClient client;

        private final Path sourceRoot;

        private final String destinationRoot;

        public Uploader(final Path source, final String destination) {
            this.sourceRoot = source;
            this.destinationRoot = destination;
        }

        @Override
        public FileVisitResult preVisitDirectory(final Path dir, 
                                                 final BasicFileAttributes attrs) throws IOException {
            final Path relativeDir = sourceRoot.relativize(dir);
            System.out.println("Visit Directory: " + relativeDir);

            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFile(final Path file, 
                                         final BasicFileAttributes attrs) throws IOException {
            final Path relativeFile = sourceRoot.relativize(file);
            System.out.println("Visit File: " + relativeFile);

            return FileVisitResult.CONTINUE;
        }
    }
}

You should be able to run your main method and see directories being traversed in depth-first order. Using the example directory results in the following output:

Source: ./docs
Destination: /tomas.celaya/stor/java-manta-docs
Visit Directory: 
Visit Directory: coverage
Visit File: coverage/jacoco.68652f0639e3244b307bb8b53a2b2152d7bd4870.tar.gz
Visit Directory: images
Visit File: images/Manta Client Architecture.draw.io.xml
Visit File: images/Manta Encrypted File Request Flow.xml
Visit File: images/Manta Server Architecture.draw.io.xml
Visit File: README.md
Visit Directory: rendered
Visit File: rendered/Manta Client Architecture.png
Visit File: rendered/Manta Encrypted File Request Flow.png
Visit File: rendered/Manta Server Architecture.png
Visit Directory: scratch
Visit File: scratch/JMHBenchmark.java
Visit File: scratch/MantaCredentials.java
Visit File: scratch/MantaRecursiveDirectoryCreationStrategyTest.java
Visit File: scratch/MBeanSupervisor.java
Visit File: scratch/MCT.java
Visit File: scratch/ScratchIT.java
Directory mirroring complete: ./docs

Creating directories

Since we need to create a directory in Manta before we can upload files to it let's expand preVisitDirectory to create remote copies of the directories we encounter:

@Override
public FileVisitResult preVisitDirectory(final Path dir, 
                                         final BasicFileAttributes attrs) throws IOException {
    final Path relativeDir = sourceRoot.relativize(dir);
    System.out.println("Visit Directory: " + relativeDir);

    final String remoteDir = String.format("%s/%s", destinationRoot, relativeDir);
    System.out.println("Creating Remote Directory: " + remoteDir);

    // Recursively create the remote directory
    client.putDirectory(remoteDir, true);

    return FileVisitResult.CONTINUE;
}

Finally, let's populate these directories with the files we encounter.

Uploading files

The visitFile method should look similar to preVisitDirectory but operating on files instead:

@Override
public FileVisitResult visitFile(final Path file, 
                                 final BasicFileAttributes attrs) throws IOException {
    final Path relativeFile = sourceRoot.relativize(file);
    System.out.println("Visit File: " + relativeFile);

    final String remoteFile = String.format("%s/%s", destinationRoot, relativeFile);
    System.out.println("Uploading File: " + remoteFile);

    // Upload the visited file
    client.put(remoteFile, file.toFile());

    return FileVisitResult.CONTINUE;
}

All together now

Your DirectoryUpload class should look like the following:

import com.joyent.manta.client.MantaClient;
import com.joyent.manta.config.ChainedConfigContext;
import com.joyent.manta.config.ConfigContext;
import com.joyent.manta.config.DefaultsConfigContext;
import com.joyent.manta.config.EnvVarConfigContext;
import com.joyent.manta.config.SystemSettingsConfigContext;
import picocli.CommandLine;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;

import java.io.IOException;
import java.nio.file.FileVisitOption;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.EnumSet;
import java.util.concurrent.Callable;

public class DirectoryUpload implements Callable<Void> {

    @Parameters(index = "0", description = "local directory to upload")
    private String source;

    @Parameters(index = "1", description = "remote path for directory root")
    private String destination;

    @Option(names = {"-l", "--follow-links"})
    private boolean followLinks;

    public static void main(final String[] args) throws Exception {
        CommandLine.call(new DirectoryUpload(), System.err, args);
    }

    @Override
    public Void call() {

        System.out.println("Source: " + source);
        System.out.println("Destination: " + destination);

        final EnumSet<FileVisitOption> visitOpts;
        if (followLinks) {
            visitOpts = EnumSet.of(FileVisitOption.FOLLOW_LINKS);
        } else {
            visitOpts = EnumSet.noneOf(FileVisitOption.class);
        }

        final ConfigContext config = new ChainedConfigContext(
                new DefaultsConfigContext(),
                new EnvVarConfigContext(),
                new SystemSettingsConfigContext());

        final MantaClient client = new MantaClient(config);

        if (client.existsAndIsAccessible(destination)) {
            System.err.println(String.format("Destination directory [%s] already exists!", destination));
            System.exit(1);
        }

        final Path sourcePath = Paths.get(source);

        try {
            Files.walkFileTree(
                    sourcePath,
                    visitOpts,
                    Integer.MAX_VALUE,
                    new Uploader(client, sourcePath, destination));
        } catch (IOException e) {
            e.printStackTrace(System.err);
            System.exit(1);
        }

        System.out.println("Directory mirroring complete: " + source);

        return null;
    }

    static class Uploader extends SimpleFileVisitor<Path> {

        private final MantaClient client;

        private final Path sourceRoot;

        private final String destinationRoot;

        public Uploader(final MantaClient client, final Path source, final String destination) {
            this.client = client;
            this.sourceRoot = source;
            this.destinationRoot = destination;
        }

        @Override
        public FileVisitResult preVisitDirectory(final Path dir, final BasicFileAttributes attrs) throws IOException {
            final Path relativeDir = sourceRoot.relativize(dir);
            System.out.println("Visit Directory: " + relativeDir);

            final String remoteDir = String.format("%s/%s", destinationRoot, relativeDir);
            System.out.println("Creating Remote Directory: " + remoteDir);

            // Recursively create the remote directory
            client.putDirectory(remoteDir, true);

            System.out.println("Created Directory: " + remoteDir);

            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFile(final Path file, final BasicFileAttributes attrs) throws IOException {
            final Path relativeFile = sourceRoot.relativize(file);
            System.out.println("Visit File: " + relativeFile);

            final String remoteFile = String.format("%s/%s", destinationRoot, relativeFile);
            System.out.println("Uploading File: " + remoteFile);

            // Upload the visited file
            client.put(remoteFile, file.toFile());

            return FileVisitResult.CONTINUE;
        }
    }
}

Running `DirectoryUpload`

Here's the output from uploading the example directory mentioned in the first section:

$ java com.joyent.manta.cli.DirectoryUpload ./docs /tomas.celaya/stor/java-manta-docs
Source: ./docs
Destination: /tomas.celaya/stor/java-manta-docs
Visit Directory: 
Creating Remote Directory: /tomas.celaya/stor/java-manta-docs/
Created Directory: /tomas.celaya/stor/java-manta-docs/
Visit Directory: coverage
Creating Remote Directory: /tomas.celaya/stor/java-manta-docs/coverage
Created Directory: /tomas.celaya/stor/java-manta-docs/coverage
Visit File: coverage/jacoco.68652f0639e3244b307bb8b53a2b2152d7bd4870.tar.gz
Uploading File: /tomas.celaya/stor/java-manta-docs/coverage/jacoco.68652f0639e3244b307bb8b53a2b2152d7bd4870.tar.gz
Visit Directory: images
Creating Remote Directory: /tomas.celaya/stor/java-manta-docs/images
Created Directory: /tomas.celaya/stor/java-manta-docs/images
Visit File: images/Manta Client Architecture.draw.io.xml
Uploading File: /tomas.celaya/stor/java-manta-docs/images/Manta Client Architecture.draw.io.xml
Visit File: images/Manta Encrypted File Request Flow.xml
Uploading File: /tomas.celaya/stor/java-manta-docs/images/Manta Encrypted File Request Flow.xml
Visit File: images/Manta Server Architecture.draw.io.xml
Uploading File: /tomas.celaya/stor/java-manta-docs/images/Manta Server Architecture.draw.io.xml
Visit File: README.md
Uploading File: /tomas.celaya/stor/java-manta-docs/README.md
Visit Directory: rendered
Creating Remote Directory: /tomas.celaya/stor/java-manta-docs/rendered
Created Directory: /tomas.celaya/stor/java-manta-docs/rendered
Visit File: rendered/Manta Client Architecture.png
Uploading File: /tomas.celaya/stor/java-manta-docs/rendered/Manta Client Architecture.png
Visit File: rendered/Manta Encrypted File Request Flow.png
Uploading File: /tomas.celaya/stor/java-manta-docs/rendered/Manta Encrypted File Request Flow.png
Visit File: rendered/Manta Server Architecture.png
Uploading File: /tomas.celaya/stor/java-manta-docs/rendered/Manta Server Architecture.png
Visit Directory: scratch
Creating Remote Directory: /tomas.celaya/stor/java-manta-docs/scratch
Created Directory: /tomas.celaya/stor/java-manta-docs/scratch
Visit File: scratch/JMHBenchmark.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/JMHBenchmark.java
Visit File: scratch/MantaCredentials.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/MantaCredentials.java
Visit File: scratch/MantaRecursiveDirectoryCreationStrategyTest.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/MantaRecursiveDirectoryCreationStrategyTest.java
Visit File: scratch/MBeanSupervisor.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/MBeanSupervisor.java
Visit File: scratch/MCT.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/MCT.java
Visit File: scratch/ScratchIT.java
Uploading File: /tomas.celaya/stor/java-manta-docs/scratch/ScratchIT.java
Directory mirroring complete: ./docs

Process finished with exit code 0

Check with `mfind`

We can verify our files were downloaded by downloading the files with getAsInputStream or getToTempFile. Alternatively, the mfind command from node-manta can be used to inspect the

$ mfind ~~/stor/java-manta-docs
/tomas.celaya/stor/java-manta-docs/README.md
/tomas.celaya/stor/java-manta-docs/coverage
/tomas.celaya/stor/java-manta-docs/images
/tomas.celaya/stor/java-manta-docs/rendered
/tomas.celaya/stor/java-manta-docs/scratch
/tomas.celaya/stor/java-manta-docs/scratch/JMHBenchmark.java
/tomas.celaya/stor/java-manta-docs/scratch/MBeanSupervisor.java
/tomas.celaya/stor/java-manta-docs/scratch/MCT.java
/tomas.celaya/stor/java-manta-docs/scratch/MantaCredentials.java
/tomas.celaya/stor/java-manta-docs/scratch/MantaRecursiveDirectoryCreationStrategyTest.java
/tomas.celaya/stor/java-manta-docs/scratch/ScratchIT.java
/tomas.celaya/stor/java-manta-docs/images/Manta Client Architecture.draw.io.xml
/tomas.celaya/stor/java-manta-docs/images/Manta Encrypted File Request Flow.xml
/tomas.celaya/stor/java-manta-docs/images/Manta Server Architecture.draw.io.xml
/tomas.celaya/stor/java-manta-docs/coverage/jacoco.68652f0639e3244b307bb8b53a2b2152d7bd4870.tar.gz
/tomas.celaya/stor/java-manta-docs/rendered/Manta Client Architecture.png
/tomas.celaya/stor/java-manta-docs/rendered/Manta Encrypted File Request Flow.png
/tomas.celaya/stor/java-manta-docs/rendered/Manta Server Architecture.png

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cloning a directory tree

Generating a project

Relative paths

Walking the local filesystem

Creating directories

Uploading files

All together now

Running `DirectoryUpload`

Check with `mfind`

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Cloning a directory tree

Generating a project

Relative paths

Walking the local filesystem

Creating directories

Uploading files

All together now

Running DirectoryUpload

Check with mfind

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Running `DirectoryUpload`

Check with `mfind`