Documentation
¶
Overview ¶
Package fsutil provides various implementations of the fs.FS interface, primarily for working with remote files.
To support URI schemes, the package defines the Protocol interface, which takes a URI and returns an appropriate fs.FS implementation and the path to the file within the filesystem.
To support multiple URI schemes, the package provides the "Mux" protocol, which delegates the URI to the appropriate protocol based on the scheme.
Example:
mux := NewMux(map[string]ProtoFunc{
"file": func(uri *url.URL) (Protocol, error) { return NewFileProto(), nil },
"http": func(uri *url.URL) (Protocol, error) { return NewHTTPProto(context.Background()), nil },
"https": func(uri *url.URL) (Protocol, error) { return NewHTTPProto(context.Background()), nil },
})
// Get the appropriate filesystem and path for the URI.
fs, path, err := mux.FileSystem(errutil.Must(http.Parse("http://example.com/file")))
if err != nil {
log.Fatal(err)
}
b, err := fs.ReadFile(path)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
Index ¶
- func IPFSPathResolution(cid string) func(f *httpFS, name string) (*netURL.URL, error)
- func IPFSSubdomainResolution(cid string) func(f *httpFS, name string) (*netURL.URL, error)
- func NewCacheFS(fs fs.FS, opts ...CacheFSOption) (fs.FS, error)
- func NewChainFS(opts ...ChainFSOption) fs.FS
- func NewChecksumFS(fs fs.FS, opts ...ChecksumFSOption) (fs.FS, error)
- func NewGzipFS(fs fs.FS, opts ...GzipFSOption) fs.FS
- func NewHTTPFS(ctx context.Context, baseURI *netURL.URL, opts ...HTTPFSOption) (fs.FS, error)
- func NewIPFSFS(ctx context.Context, cid string, opts ...IPFSOption) (fs.FS, error)
- func NewRetryFS(ctx context.Context, fs fs.FS, attempts int, delay time.Duration) fs.FS
- func ParseURI(p Protocol, uri string) (fs.FS, string, error)
- type CacheFSOption
- type ChainFSOption
- type ChecksumFSOption
- type ChecksumFSVerifyMode
- type FileOption
- type GzipFSOption
- type HTTPFSOption
- type IPFSGateway
- type IPFSOption
- type ProtoFunc
- type Protocol
- func NewCacheProto(proto Protocol, opts ...CacheFSOption) Protocol
- func NewChainProto(opts ...ChainFSOption) Protocol
- func NewChecksumProto(proto Protocol, opts ...ChecksumFSOption) Protocol
- func NewFSProto(f fs.FS) Protocol
- func NewFileProto(opts ...FileOption) Protocol
- func NewGzipProto(proto Protocol, opts ...GzipFSOption) Protocol
- func NewHTTPProto(ctx context.Context, opts ...HTTPFSOption) Protocol
- func NewIPFSProto(ctx context.Context, opts ...IPFSOption) Protocol
- func NewMux(ps map[string]ProtoFunc) Protocol
- func NewRetryProto(ctx context.Context, proto Protocol, attempts int, delay time.Duration) Protocol
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IPFSPathResolution ¶
func IPFSSubdomainResolution ¶
func NewCacheFS ¶
NewCacheFS creates a new cache filesystem.
The cache filesystem caches the contents of the files in the cache directory. If the file is not found in the cache, it will be read from the underlying file system and cached.
func NewChainFS ¶
func NewChainFS(opts ...ChainFSOption) fs.FS
NewChainFS creates a new chain filesystem.
The chain filesystem chains multiple file systems together. It will try to open a file in the first file system. If it fails, it will try the next one, and so on. If all file systems fail, it will return an error.
func NewChecksumFS ¶
NewChecksumFS creates a new checksum file system.
The file system wraps an existing file system, computes the checksum of the file contents, and compares it with the provided checksum. The checksum must be provided in the file name as a query parameter, e.g., "file?checksum=0x1234...".
If the checksum does not match, the file system returns an error when reading the file.
func NewGzipFS ¶
func NewGzipFS(fs fs.FS, opts ...GzipFSOption) fs.FS
NewGzipFS creates a new gzip filesystem.
The gzip filesystem will wrap the given filesystem and add gzip decompression functionality.
func NewIPFSFS ¶
NewIPFSFS creates a new IPFS filesystem.
The IPFS filesystem uses IPFS gateways to resolve IPFS paths. To verify the integrity of the file contents and ensure that returned data is valid, an optional checksum hash can be provided as a "checksum" parameter in the URL.
It is important to provide a checksum, as there is no guarantee that the data returned from IPFS gateways is valid. A misconfigured or malicious gateway could return a different or corrupted file.
func NewRetryFS ¶
NewRetryFS wraps the given FS to add retry functionality.
Types ¶
type CacheFSOption ¶
type CacheFSOption func(*cacheFS)
func WithCacheDir ¶
func WithCacheDir(dir string) CacheFSOption
WithCacheDir sets the cache directory.
func WithCacheNamespace ¶ added in v0.57.1
func WithCacheNamespace(ns string) CacheFSOption
WithCacheNamespace sets the cache namespace, that is added to the hash of the file name to create a unique cache file name.
type ChainFSOption ¶
type ChainFSOption func(*chainFS)
func WithChainFilesystems ¶
func WithChainFilesystems(fs ...fs.FS) ChainFSOption
WithChainFilesystems sets the file systems to chain.
func WithChainRandOrder ¶
func WithChainRandOrder() ChainFSOption
WithChainRandOrder sets the file systems to chain in random order.
type ChecksumFSOption ¶
type ChecksumFSOption func(*checksumFS)
func WithChecksumHash ¶
func WithChecksumHash(hash func() hash.Hash) ChecksumFSOption
WithChecksumHash sets the hash function used to compute the checksum. The default hash function is LegacyKeccak256.
func WithChecksumParamName ¶
func WithChecksumParamName(name string) ChecksumFSOption
WithChecksumParamName sets the name of the URL query parameter that contains the checksum value. The default parameter name is "checksum".
func WithChecksumVerifyMode ¶
func WithChecksumVerifyMode(mode ChecksumFSVerifyMode) ChecksumFSOption
WithChecksumVerifyMode sets the mode of the checksum verification. The default mode is ChecksumFSVerifyAfterRead.
type ChecksumFSVerifyMode ¶
type ChecksumFSVerifyMode int
const ( // ChecksumFSVerifyAfterRead verifies the checksum after reading the file // contents. ChecksumFSVerifyAfterRead ChecksumFSVerifyMode = iota // ChecksumFSVerifyAfterOpen verifies the checksum immediately after // opening the file. ChecksumFSVerifyAfterOpen )
type FileOption ¶
type FileOption func(*fileProto)
func WithFileWorkingDir ¶
func WithFileWorkingDir(wd string) FileOption
WithFileWorkingDir sets the working directory for the file protocol.
type GzipFSOption ¶
type GzipFSOption func(*gzipFS)
func WithGzipCheckExtension ¶
func WithGzipCheckExtension(check bool) GzipFSOption
WithGzipCheckExtension enables or disables checking the file extension to determine whether to decompress the file. If enabled, only files with the specified extensions will be decompressed.
func WithGzipExtensions ¶
func WithGzipExtensions(exts ...string) GzipFSOption
WithGzipExtensions sets the list of file extensions that will be decompressed. The default extension is "gz". Ignored if WithGzipCheckExtension is set to false.
func WithGzipReadLimit ¶
func WithGzipReadLimit(limit int64) GzipFSOption
WithGzipReadLimit sets the maximum size of the decompressed data. If the decompressed data exceeds the limit, the io.ErrUnexpectedEOF error will be returned. The default limit is 128MiB.
type HTTPFSOption ¶
type HTTPFSOption func(*httpFS)
func WithHTTPClient ¶
func WithHTTPClient(client *http.Client) HTTPFSOption
WithHTTPClient sets the HTTP client used to perform HTTP requests.
type IPFSGateway ¶
type IPFSOption ¶
type IPFSOption func(*ipfsFS)
func WithIPFSChecksumHash ¶
func WithIPFSChecksumHash(hash func() hash.Hash) IPFSOption
WithIPFSChecksumHash sets the hash function used to compute the checksum.
func WithIPFSGateways ¶
func WithIPFSGateways(gateways ...*IPFSGateway) IPFSOption
WithIPFSGateways sets the IPFS gateways used to resolve IPFS paths.
func WithIPFSHTTPClient ¶
func WithIPFSHTTPClient(client *http.Client) IPFSOption
WithIPFSHTTPClient sets the HTTP client used to perform HTTP requests.
type Protocol ¶
Protocol defines a file system protocol. It provides a file system instance and a path within that file system for a given URI.
The returned file system always points to the highest possible level directory.
func NewCacheProto ¶
func NewCacheProto(proto Protocol, opts ...CacheFSOption) Protocol
NewCacheProto creates a new cache protocol.
The cache protocol will wrap the filesystem returned by a given protocol with a cache filesystem.
func NewChainProto ¶
func NewChainProto(opts ...ChainFSOption) Protocol
NewChainProto creates a new chain protocol.
func NewChecksumProto ¶
func NewChecksumProto(proto Protocol, opts ...ChecksumFSOption) Protocol
NewChecksumProto creates a new checksum protocol.
func NewFSProto ¶ added in v0.57.1
NewFSProto creates a new file system protocol that uses the provided file system.
func NewFileProto ¶
func NewFileProto(opts ...FileOption) Protocol
NewFileProto creates a new file protocol that uses the local filesystem. The URI scheme must be "file" and the host must be empty or "localhost". The working directory is set to "/" by default.
func NewGzipProto ¶
func NewGzipProto(proto Protocol, opts ...GzipFSOption) Protocol
NewGzipProto creates a new gzip protocol.
The gzip protocol will wrap the filesystem returned by a given protocol with a gzip filesystem.
func NewHTTPProto ¶
func NewHTTPProto(ctx context.Context, opts ...HTTPFSOption) Protocol
The HTTP protocol is used to create an HTTP file system.
func NewIPFSProto ¶
func NewIPFSProto(ctx context.Context, opts ...IPFSOption) Protocol
NewIPFSProto creates a new IPFS protocol.
The IPFS protocol is used to create an IPFS file system.
Source Files