Documentation
¶
Overview ¶
Package io provides an interface for IO implementations along with a registry for registering IO implementations for different URI schemes.
Subpackages of this package provide implementations for cloud storage providers which will register themselves if imported. For instance, adding the following import:
import _ "github.com/apache/iceberg-go/io/gocloud"
Will register cloud storage implementations for S3, GCS, Azure, and in-memory blob storage. The local filesystem (file:// and empty scheme) is registered by default.
Index ¶
- Constants
- Variables
- func GetRegisteredSchemes() []string
- func LoadFSFunc(props map[string]string, location string) func(ctx context.Context) (IO, error)
- func Register(scheme string, factory SchemeFactory)
- func Unregister(scheme string)
- type File
- type FileWriter
- type IO
- type LocalFS
- type ReadDirFile
- type ReadFileIO
- type SchemeFactory
- type WriteFileIO
Constants ¶
const ( S3Region = "s3.region" S3SessionToken = "s3.session-token" S3SecretAccessKey = "s3.secret-access-key" S3AccessKeyID = "s3.access-key-id" S3EndpointURL = "s3.endpoint" S3ProxyURI = "s3.proxy-uri" S3ConnectTimeout = "s3.connect-timeout" S3SignerURI = "s3.signer.uri" S3RemoteSigningEnabled = "s3.remote-signing-enabled" S3ForceVirtualAddressing = "s3.force-virtual-addressing" )
Constants for S3 configuration options
const ( GCSEndpoint = "gcs.endpoint" GCSKeyPath = "gcs.keypath" GCSJSONKey = "gcs.jsonkey" GCSCredType = "gcs.credtype" GCSUseJSONAPI = "gcs.usejsonapi" // set to anything to enable )
Constants for GCS configuration options
const ( ADLSSasTokenPrefix = "adls.sas-token." ADLSConnectionStringPrefix = "adls.connection-string." ADLSEndpoint = "adls.endpoint" ADLSProtocol = "adls.protocol" )
Constants for Azure configuration options
Variables ¶
var (
ErrIOSchemeNotFound = errors.New("io scheme not registered")
)
Functions ¶
func GetRegisteredSchemes ¶
func GetRegisteredSchemes() []string
GetRegisteredSchemes returns the list of registered scheme names.
func LoadFSFunc ¶
LoadFSFunc is a helper function to create IO Func factory for later usage
func Register ¶
func Register(scheme string, factory SchemeFactory)
Register adds a new scheme factory to the registry. If the scheme is already registered, it will panic.
func Unregister ¶
func Unregister(scheme string)
Unregister removes the requested scheme factory from the registry.
Types ¶
type File ¶
A File provides access to a single file. The File interface is the minimum implementation required for Iceberg to interact with a file. Directory files should also implement
type FileWriter ¶
type FileWriter interface {
io.WriteCloser
io.ReaderFrom
}
A FileWriter represents an open writable file.
type IO ¶
type IO interface {
// Open opens the named file.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "open", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not satisfy
// fs.ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
// Remove removes the named file or (empty) directory.
//
// If there is an error, it will be of type *PathError.
Remove(name string) error
}
IO is an interface to a hierarchical file system.
The IO interface is the minimum implementation required for a file system to utilize an iceberg table. A file system may implement additional interfaces, such as ReadFileIO, to provide additional or optimized functionality.
func FSPreProcName ¶
FSPreProcName wraps an io/fs.FS like FS, only if fn is non-nil then it is called to preprocess any filenames before they are passed to the underlying fsys.
func LoadFS ¶
LoadFS takes a map of properties and an optional URI location and attempts to infer an IO object from it using the registered scheme factories.
The scheme is extracted from the location URI and used to look up the appropriate factory from the registry. The local filesystem (file:// or empty scheme) is registered by default.
Additional schemes can be registered by importing subpackages. For S3, GCS, Azure and in-memory support, import:
import _ "github.com/apache/iceberg-go/io/gocloud"
type LocalFS ¶
type LocalFS struct{}
LocalFS is an implementation of IO that implements interaction with the local file system.
type ReadDirFile ¶
type ReadDirFile interface {
File
// ReadDir read the contents of the directory and returns a slice
// of up to n DirEntry values in directory order. Subsequent calls
// on the same file will yield further DirEntry values.
//
// If n > 0, ReadDir returns at most n DirEntry structures. In this
// case, if ReadDir returns an empty slice, it will return a non-nil
// error explaining why.
//
// At the end of a directory, the error is io.EOF. (ReadDir must return
// io.EOF itself, not an error wrapping io.EOF.)
//
// If n <= 0, ReadDir returns all the DirEntry values from the directory
// in a single slice. In this case, if ReadDir succeeds (reads all the way
// to the end of the directory), it returns the slice and a nil error.
// If it encounters an error before the end of the directory, ReadDir
// returns the DirEntry list read until that point and a non-nil error.
ReadDir(n int) ([]fs.DirEntry, error)
}
A ReadDirFile is a directory file whose entries can be read with the ReadDir method. Every directory file should implement this interface. (It is permissible for any file to implement this interface, but if so ReadDir should return an error for non-directories.)
type ReadFileIO ¶
type ReadFileIO interface {
IO
// ReadFile reads the named file and returns its contents.
// A successful call returns a nil error, not io.EOF.
// (Because ReadFile reads the whole file, the expected EOF
// from the final Read is not treated as an error to be reported.)
//
// The caller is permitted to modify the returned byte slice.
// This method should return a copy of the underlying data.
ReadFile(name string) ([]byte, error)
}
ReadFileIO is the interface implemented by a file system that provides an optimized implementation of ReadFile.
type SchemeFactory ¶
SchemeFactory is a function that creates an IO implementation for a given URI and properties.
type WriteFileIO ¶
type WriteFileIO interface {
IO
// Create attempts to create the named file and return a writer
// for it.
Create(name string) (FileWriter, error)
// WriteFile writes p to the named file.
WriteFile(name string, p []byte) error
}
WriteFileIO is the interface implemented by a file system that provides an optimized implementation of WriteFile
Source Files
Directories