C2FO/vfs: Pluggable, extensible virtual file system for Go

Installation

go get github.com/c2fo/vfs/v7

Quick Start

package main
import (
    "fmt"
    "strings"
    "io"
    "log"
"github.com/c2fo/vfs/v7"
    "github.com/c2fo/vfs/v7/vfssimple"
)
func main() {
    // Create local OS file from a URI
    osFile, err := vfssimple.NewFile("file:///tmp/example.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer osFile.Close()
// Write to the file
    _, err = io.Copy(osFile, strings.NewReader("Hello from vfs!"))
    if err != nil {
        log.Fatal(err)
    }
if err := osFile.Close(); err != nil {
        log.Fatal(err)
    }
    fmt.Println("File created and written:", osFile.URI())
}

Listing

list.Entries streams the contents of any location as vfs.Entry values, in byte order, fetching from the backend only as you iterate:

for e, err := range list.Entries(ctx, loc, list.Glob("/.csv"), list.FilesOnly()) {
    if err != nil {
        return err
    }
    fmt.Println(e.RelPath(), e.Size())
}

• Options such as list.Prefix, list.Recursive, list.MaxDepth, list.FilesOnly, list.DirsOnly, list.Glob, list.Match, list.Filter and list.DirFilter choose what is listed. list.Limit with list.After pages through a listing, and each option may be given once. A list.Glob implies the depth it names, so it is not combined with list.Recursive.
• An entry carries its relative path, and the size and modification time the backend listed with it, so reading them costs no extra request. Info() and InfoContext(ctx) fetch them when a backend did not list them.
• An error for one unreadable subdirectory is yielded with that directory's entry and the listing continues, so a recursive listing can report which folders were and were not accessible.
• list.Collect and list.TopN (with list.BySize, list.ByModTime and list.ByName) gather a listing when you do want it in memory.
• A backend without native listing still works, listing files only. See list for the full API.

| Before | After | |---|---| | loc.List() | list.Collect(list.Entries(ctx, loc, list.FilesOnly())), or range over list.Entries to stream | | loc.ListByPrefix("rep") | list.Entries(ctx, loc, list.Prefix("rep"), list.FilesOnly()) | | loc.ListByRegex(re) | list.Entries(ctx, loc, list.Match(re), list.FilesOnly()) |

Each entry's RelPath() is the name List returned, and list.Entries also reports directories unless list.FilesOnly() is given.

Advanced Usage

• See the GoDoc for deeper technical details, backend-specific options, and additional APIs.
• Specialized operations like copying between S3 locations or advanced configs (e.g. Azure credentials) are documented in each backend doc.

• vfs - core FileSystem, Location, and File interfaces
• vfscp
• vfssimple
• backend

• list - streaming listing of a location
• options

• utils

Community Contributed Backends

These backends are maintained in contrib/backend/ as separate Go modules:

• Dropbox backend - Dropbox cloud storage
• OneDrive backend - Microsoft OneDrive/SharePoint via Graph API
• Google Drive backend - Google Drive (My Drive and shared drives) via the Drive v3 API
• SQL FS backend - Files stored in PostgreSQL or SQLite through database/sql (registered explicitly, not on import)

Additional Tools

• lockfile: Provides distributed locking mechanisms using vfs backends.
• vfsevents: Production-ready file event monitoring with support for real-time notifications, retry logic, and multiple watcher types (VFS polling, local fsnotify, S3 events, GCS events).
• testcontainers: Runs the shared backend conformance suites against real servers in disposable Docker containers.

FAQ

Q: Why am I seeing an empty file when using io.Copy on some backends if my source is empty? A: An empty Reader often means the backend doesn't write a file until data is actually written. Use utils.TouchCopy if you need to ensure a zero-byte file is created.

Q: Will vfs v6 still be supported? A: Yes and no. We will continue to provide security patches and bug fixes for v6, but new features and enhancements will be added to v7.

Q: How long will v6 be supported? A: We will support v6 until the next major release, v8, is released. Then v7 will be supported until v9 is released.

Upgrading

Upgrading from v6 to v7

Please review these changes and update your code accordingly to ensure compatibility with v7.

S3 Backend

The project now uses the aws-sdk-go-v2 library instead of the deprecated, EOL aws-sdk-go. This update necessitated these changes to the S3 backend:

• The S3 backend's s3fs.Client() function now returns an s3.Client which is a subset of AWS's sdk v2 functionality.

• The Option.Retry field is now an aws.Retryer instead of a request.Retry. Ensure that your Option logic is

Azure Backend

• Scheme for Azure has been updated from https to az. Update your code to use the new scheme.
• Authority for Azure has been updated from blob.core.windows.net to my-container-name, such that the full URI

GS Backend

• The Options.Retry field, with the now deprecated vfs.Retry type, has been moved to gs.FileSystem as the new gs.Retryer

All Backends

• location.Volume() method which returns the authority as a string has been deprecated in favor of the location.Authority() method which

• location.ChangeDir() method has been deprecated in favor of the existing location.NewLocation() method. Update your code to use the

• vfs.Options struct has been deprecated in favor of using backend-specific structs.
• filesystem.Retry() method has been deprecated in favor of using backend-specific functional options, such as gs.WithRetryer.

Upgrading from v5 to v6

With v6, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.

"keyExchanges":"diffie-hellman-group-a256"

becomes

"keyExchanges":["diffie-hellman-group-a256"]

Contributing

1. Fork it (git clone https://github.com/c2fo/vfs.git)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am "Add some fooBar")
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request

License