Skip to content

Commit

Permalink
Improve VFS simple's discovery of registered backends (#7)
Browse files Browse the repository at this point in the history
* Added the ability to have vfs simple route to file systems based on file URI host (volume) and file

* cleaned up vfssimple doc

* more doc fixes

* cleaned up doc a bit more
  • Loading branch information
Chris Roush authored and funkyshu committed Feb 15, 2019
1 parent dd4b284 commit e54f524
Show file tree
Hide file tree
Showing 3 changed files with 100 additions and 24 deletions.
73 changes: 51 additions & 22 deletions docs/vfssimple.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,45 +14,74 @@ supported backend filesystem by using full URI's:

Just import vfssimple.

package main

import(
"github.com/c2fo/vfs/vfssimple"
)

...

func DoSomething() error {
package main
import(
"github.com/c2fo/vfs/vfssimple"
)
...
func DoSomething() error {
myLocalDir, err := vfssimple.NewLocation("file:///tmp/")
if err != nil {
return err
}

myS3File, err := vfssimple.NewFile("s3://mybucket/some/path/to/key.txt")
if err != nil {
return err
}

localFile, err := myS3File.MoveToLocation(myLocalDir)
if err != nil {
return err
}

}
}


### Authentication and Options

vfssimple is largely an example of how to initialize a set of backend
filesystems. It only provides a default initialization of the individual file
systems. See backend docs for specific authentication info for each backend but
generally speaking, most backends can use Environment variables to set
credentials or client options.
vfssimple is largely an example of how to initialize a set of backend filesystems. It only provides a default
initialization of the individual file systems. See backend docs for specific authentication info for each backend but
generally speaking, most backends can use Environment variables to set credentials or client options.

File systems can only use one set of options. If you would like to configure more than one file system of the same type/schema with separate credentials,
you can register and map file system options to locations or individual objects. The vfssimple library will automatically try to
resolve the provided URI in NewFile() or NewLocation() to the registered file system.

package main

import(
"github.com/c2fo/vfs/vfssimple"
"github.com/c2fo/vfs/backend"
"github.com/c2fo/vfs/backend/s3"
)

...

func DoSomething() error {
bucketAuth := s3.NewFileSystem().WithOptions(s3.Options{
AccessKeyID: "key1",
SecretAccessKey: "secret1,
Region: "us-west-2",
})
fileAuth := s3.NewFileSystem().WithOptions(s3.Options{
AccessKeyID: "key2",
SecretAccessKey: "secret2,
Region: "us-west-2",
})
backend.Register("s3://bucket1/, bucketAuth)
backend.Register("s3://bucket2/file.txt, fileAuth)
secureFile, _ := vfssimple.NewFile("s3://bucket2/file.txt")
publicLocation, _ := vfssimple.NewLocation("s3://bucket1/")
secureFile.CopyToLocation(publicLocation)
}

To do more, especially if you need to pass in specific [vfs.Options](../README.md#type-options)'s via
WithOption() or perhaps a mock client for testing via WithClient() or something
else, you'd need to implement your own factory. See [backend](backend.md)
for more information.

## Functions

Expand Down
38 changes: 36 additions & 2 deletions vfssimple/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,41 @@ vfssimple is largely an example of how to initialize a set of backend filesystem
initialization of the individual file systems. See backend docs for specific authentication info for each backend but
generally speaking, most backends can use Environment variables to set credentials or client options.
To do more, especially if you need to pass in specific vfs.Option's via WithOption() or perhaps a mock client for testing via
WithClient() or something else, you'd need to implement your own factory. See github.com/c2fo/vfs/backend for more information.
File systems can only use one set of options. If you would like to configure more than one file system of the same type/schema with separate credentials,
you can register and map file system options to locations or individual objects. The vfssimple library will automatically try to
resolve the provided URI in NewFile() or NewLocation() to the registered file system.
package main
import(
"github.com/c2fo/vfs/vfssimple"
"github.com/c2fo/vfs/backend"
"github.com/c2fo/vfs/backend/s3"
)
...
func DoSomething() error {
bucketAuth := s3.NewFileSystem().WithOptions(s3.Options{
AccessKeyID: "key1",
SecretAccessKey: "secret1,
Region: "us-west-2",
})
fileAuth := s3.NewFileSystem().WithOptions(s3.Options{
AccessKeyID: "key2",
SecretAccessKey: "secret2,
Region: "us-west-2",
})
backend.Register("s3://bucket1/, bucketAuth)
backend.Register("s3://bucket2/file.txt, fileAuth)
secureFile, _ := vfssimple.NewFile("s3://bucket2/file.txt")
publicLocation, _ := vfssimple.NewLocation("s3://bucket1/")
secureFile.CopyToLocation(publicLocation)
}
*/
package vfssimple
13 changes: 13 additions & 0 deletions vfssimple/vfssimple.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"errors"
"fmt"
"net/url"
"strings"

"github.com/c2fo/vfs"
"github.com/c2fo/vfs/backend"
Expand Down Expand Up @@ -46,6 +47,18 @@ func parseSupportedURI(uri string) (vfs.FileSystem, string, string, error) {

var fs vfs.FileSystem
for _, backendScheme := range backend.RegisteredBackends() {
// Object-level backend
if strings.Index(uri, backendScheme) > 1 {
fs = backend.Backend(backendScheme)
break
}
// Bucket-level backend
volume := fmt.Sprintf("%s://%s/", u.Scheme, u.Host)
if volume == backendScheme {
fs = backend.Backend(backendScheme)
break
}
// Scheme-level backend
if u.Scheme == backendScheme {
fs = backend.Backend(backendScheme)
}
Expand Down

0 comments on commit e54f524

Please sign in to comment.