json

package module
v0.0.0-...-4ebe608 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2025 License: Apache-2.0 Imports: 22 Imported by: 3

README

json

This go library processes JSON streams and buffers.

It is intended to provide a low-level toolkit for functionally demanding use-cases involving JSON data.

It brings JSON support for the go-openapi packages.

It is not a replacement for the standard library encoding/json. As such, it does not marshal or unmarshal your favorite go struct to or from JSON.

It does not compete with these blazingly fast JSON codec libraries, which target the marshal/unmarshal feature.

Instead, it provides structures and types that keep the dynamic nature of JSON. The resolution of individual JSON values to native go types may be deferred.

Essentially, it provides ways to work with a JSON document without the need for a pre-defined go data structure.

What's inside?

The module github.com/fredbi/core/json provides the following tools:

What is not inside?

  • functionality like json.Unmarshal or json.Marshal that shuffles JSON to/from go data structures.
  • JSON schema support is provided by github.com/fredbi/core/jsonschema. This library focuses on the low-level aspects of JSON only.
  • OpenAPI support is provided by github.com/fredbi/core/spec. This is far beyond the scope of mere JSON processing.

How does this lib compare to others?

We designed this library to serve advanced use cases such as JSON schema analysis and OpenAPI spec handling.

Besides, we wanted the core of go-openapi to avoid external dependencies for this kind of work.

Depending on your goals, you might find libraries that are a better fit for your use case.

  • It does not generate code like github.com/mailru/easyjson. However, it shares a lot of ideas with the lexer and writer from easyjson.
  • It does not provide an API to extract JSON values directly like github.com/valyala/fastjson or github.com/tidwall/gjson. However, it shares a lot of ideas with fastjson such as reusable values stored in an Arena.
  • It does not support mapping to your structs like encoding/json or drop-in replacements for the standard library like github.com/go-ccy/go-json or github.com/jsoniterator/go.
  • TODO: compare github.com/go-faster/jx
  • TODO: compare github.com/francoispqt/gojay
  • TODO: compare github.com/bytedance/sonic (a JIT JSON serializing/deserializing lib)

On the other hand, it provides a few features that you won't find in the above-mentioned libraries:

  • Accurate error reporting with context when lexing JSON
  • Support for streams as well as buffers
  • Support for JSON pointers and JSONPath expressions
  • The ability to build JSON programmatically

What is "verbatim"?

A verbatim document reproduces the original JSON unaltered, that is, including non-significant blank space used for indentation. It also leaves escaped unicode sequence unchanged.

This allows to process JSON with the ability to control the original input (as in a text editor).

In contrast, a default document focuses on JSON semantics, i.e. structure and values: non-significant blanks are ignored and the annoying escaped unicode sequences are resolved as soon a possible into their UTF-8 equivalent.

Verbatim lexer, store, document and writer should be reserved to tools like linters, text editor plugins and the like.

What can I do with a JSON "Document"?

Documents are immutable, but cheap to build or amend using shallow clones. This guards against nasty bugs and allows for a concurrent processing of documents.

Documents maintain the original ordering of keys in objects.

  • Marshal / Unmarshal to/from JSON bytes
  • Encode / Decode to/from a stream of JSON bytes
  • Build or clone & amend using the Builder type
  • Walk a document using iterators
  • TODO: resolve a JSON Pointer within the document
  • TODO: Walk a document using jsonpath expressions. See github.com/fredbi/core/json/jsonpath.

Design goals

  1. accurate representation of JSON semantics[^1].
  2. accurate error reporting [^2]
  3. memory-efficient processing of JSON. The toolkit tries its best to keep a low profile in memory [^3].
  4. immutable documents, cheap to clone and build programmatically
  5. dependency-free packages, besides test dependencies
  6. extensible interfaces[^4]

Non-goals

This library favors accuracy over speed. It may not be the fastest JSON processor available for go. In particular:

  • we trade accurate error reporting for speed

At this moment, it is not a goal to support XXL documents outside of main memory, although this could be a contributed extension of the stores.Store interface.

Likewise, it is not part of the current plan to provide writers to specialized backends such as databases or message-oriented middleware. Again, such a feature may be provided in the future as a contributed extension (while keeping our dependency-free goal).

At this moment, the scope is limited to JSON. YAML data, considered as a superset of JSON, is not part of the picture for now.

Use cases

Support go-openapi v2:

Besides, here are a few examples of where this toolkit might be useful:

  • lexer/writer: JSON linter, text editor plugin
  • Document: implement an API that deals dynamically with json
  • dynamic JSON: alternative way to implement an API that deals dynamically with json

[^1]: the default lexer is strict regarding JSON semantics. JSON types allow to distinguish undefined from null.

[^2]: the lexer does its best to report informative error messages, and capture the context of the offending part, including when dealing with streams.

[^3]: this leverages the memory pooling utilities exposed by github.com/fredbi/core/pools. By and large, the toolkit favors memory scarcity over CPU, assuming that much of the CPU-intensive processing may be deferred lazily. For instance, th default implementation for stores.Store compresses long strings.

[^4]: contrib modules are proposed to host standalone go modules that may bring their own set of dependencies

Documentation

Overview

Package json deals with JSON documents.

This library provides tools to work with raw JSON data, with no attempt to convert it to native go types until the very last moment.

This allows to abide strictly to the JSON standards. In particular, it supports JSON-specifics which are not well handled by the standard library, such as:

  • null
  • values with a zero value in go (e.g. 0, "", false)
  • very large or very high-precision numbers that overflow go native types (e.g. int64, float64)

Main memory usage

The library design is primarily focused on keeping a low memory profile, in terms of space as well as in terms of allocations:

  • few things are pointers or require allocations
  • most data structures that need a dynamic allocation may be recycled using pools, thus amortizing allocations
  • JSON values are isolated in a highly packed in-memory store
  • object keys are interned
  • lazy JSON value resolution

Immutability

Another design goal of this package is immutability: all provided objects Document, light.Node, [token.Token], values.Value etc are all immutable, and designed to be cheap to clone instead.

Mutating or constructing a JSON Document programmatically requires a Builder to carry out a series of fluent building methods. This produces a modified copy-on-write clone of the original Document.

Extensibility

There are tons of use-cases out there to play with.

Specific modules may be added to extend or improve the default implementations of lexers.Lexer, writers.Writer and stores.Store.

As independent modules, they may bring with them their own set of dependencies without altering the dependencies of the parent module.

The default implementations provided do not bring extra external dependencies.

Index

Constants

View Source 
const (
	// ErrPointer is an error raised by a JSON pointer.
	ErrPointer pointerError = "JSON pointer error"

	// ErrPointerNotFound is raised when a JSON pointer cannot be resolved against a document.
	ErrPointerNotFound pointerError = "JSON pointer not found"

	// ErrInvalidStart states that a JSON pointer must start with a separator ("/"), or be the empty JSON pointer.
	ErrInvalidStart pointerError = `JSON pointer must be empty or start with a /"`
)

Variables

View Source 
var EmptyDocument = Make()

EmptyDocument is the JSON document of the null type.

It has no stores.Store attached to it.

View Source 
var EmptyPointer = Pointer([]stringOrInt{})

EmptyPointer represents the empty JSON pointer, which matches a whole document.

Functions

func RedeemBuilder 

func RedeemBuilder(b *Builder)

func RedeemDocument 

func RedeemDocument(d *Document)

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder builds or transforms JSON Document s programmatically.

You may either use it directly, starting from the EmptyDocument or clone from an existing Document using Builder.From.

Since a Document is immutable, the Builder always produces a shallow clone of the original Document.

The Builder exposes fluent building methods which may be chained to construct a JSON document.

You should always check the final error state, since the building ceases to be effective as soon as an error is encountered.

func BorrowBuilder 

func BorrowBuilder() *Builder

func NewBuilder 

func NewBuilder(s stores.Store) *Builder

NewBuilder produces a Builder of JSON Document s.

func (*Builder) AppendElem 

func (b *Builder) AppendElem(value Document) *Builder

AppendElem appends a new element to an array.

func (*Builder) AppendElems 

func (b *Builder) AppendElems(values ...Document) *Builder

AppendElems appends new elements to an array.

func (*Builder) AppendKey 

func (b *Builder) AppendKey(key string, value Document) *Builder

AppendKey appends a new (key,value) to an object.

func (*Builder) Array 

func (b *Builder) Array() *Builder

Array builds a JSON array.

func (*Builder) AtPointer 

func (b *Builder) AtPointer(p Pointer, value Document) *Builder

AtPointer replaces a value in a Document at Pointer.

No replacement is made if the Pointer is not found.

func (*Builder) AtPointerMerge 

func (b *Builder) AtPointerMerge(p Pointer, value Document) *Builder

func (*Builder) BoolValue 

func (b *Builder) BoolValue(value bool) *Builder

BoolValue builds a scalar JSON boolean value.

func (Builder) Document 

func (b Builder) Document() Document

Document returns the Document produced by the Builder.

If a build error has occurred, it returns the EmptyDocument.

func (Builder) Err 

func (b Builder) Err() error

func (*Builder) From 

func (b *Builder) From(d Document) *Builder

From makes a builder that will clone a Document, possibly mutations.

func (*Builder) MakeBool 

func (b *Builder) MakeBool(value bool) Document

MakeBool is a shorthand for NewBuilder(b.Store()).BoolValue(value).Document().

func (*Builder) MakeNull 

func (b *Builder) MakeNull() Document

MakeNull is a shorthand for NewBuilder(b.Store()).Null().Document().

func (*Builder) MakeNumber 

func (b *Builder) MakeNumber(value any) Document

MakeNumber is a shorthand for NewBuilder(b.Store()).NumericalValue(value).Document().

func (*Builder) MakeString 

func (b *Builder) MakeString(value string) Document

MakeString is a shorthand for NewBuilder(b.Store()).StringValue(value).Document().

func (*Builder) Null 

func (b *Builder) Null() *Builder

NullValue builds a scalar JSON null value.

func (*Builder) NumberValue 

func (b *Builder) NumberValue(value types.Number) *Builder

NumberValue builds a scalar JSON number value.

func (*Builder) NumericalValue 

func (b *Builder) NumericalValue(value any) *Builder

NumericalValue builds a scalar JSON number value from any go numerical type, including types from math/big.

func (*Builder) Object 

func (b *Builder) Object() *Builder

Object builds a JSON object.

func (Builder) Ok 

func (b Builder) Ok() bool

func (*Builder) Reset 

func (b *Builder) Reset()

func (*Builder) SetErr 

func (b *Builder) SetErr(err error)

func (Builder) Store 

func (b Builder) Store() stores.Store

func (*Builder) StringValue 

func (b *Builder) StringValue(value string) *Builder

StringValue builds a scalar JSON string

func (*Builder) WithRoot 

func (b *Builder) WithRoot(root light.Node) *Builder

func (*Builder) WithStore 

func (b *Builder) WithStore(s stores.Store) *Builder

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

Collection is a collection of Document s, that share the same options.

It can serve as a factory to produce Document s using Collection.DecodeAppend.

It marshals as an array of [Document]s.

func NewCollection 

func NewCollection(opts ...Option) *Collection

NewCollection builds a new empty Collection of Document s.

func (*Collection) Append 

func (c *Collection) Append(docs ...Document)

Append a Document to the Collection.

If the Collection is empty, the options will be taken from the first Document appended.

Limitations

func (Collection) AppendText 

func (c Collection) AppendText(b []byte) ([]byte, error)

AppendText appends the JSON bytes to the provided buffer and returns the resulting slice.

func (*Collection) DecodeAppend 

func (c *Collection) DecodeAppend(r io.Reader) error

DecodeAppend decodes a Document from the provided reader and appends it to the Collection.

func (*Collection) Document 

func (c *Collection) Document(index int) Document

Document yields the Document at the index position in the Collection.

It panics if index >= Collection.Len.

func (*Collection) Documents 

func (c *Collection) Documents() iter.Seq[Document]

Documents iterates over the Document s in the Collection.

func (Collection) Encode 

func (c Collection) Encode(w io.Writer) error

Encode a collection of Document s as a stream of JSON bytes.

func (Collection) Len 

func (c Collection) Len() int

Len returns the number of Document s in the collection.

func (Collection) LexerFactory 

func (o Collection) LexerFactory() func([]byte) (lexers.Lexer, func())

func (Collection) LexerFromReaderFactory 

func (o Collection) LexerFromReaderFactory() func(io.Reader) (lexers.Lexer, func())

func (Collection) MarshalJSON 

func (c Collection) MarshalJSON() ([]byte, error)

MarshalJSON marshals the Collection as an array of JSON documents.

func (*Collection) Reset 

func (c *Collection) Reset()

Reset the collection, so it may be recycled.

func (Collection) Store 

func (c Collection) Store() stores.Store

Store returns the underlying stores.Store where JSON values are kept.

func (Collection) WriterToWriterFactory 

func (o Collection) WriterToWriterFactory() func(io.Writer) (writers.StoreWriter, func())

type Context 

type Context struct {
	light.Context
}

Context of a node, i.e. the offset in the originally parsed JSON input.

type DecodeError 

type DecodeError struct {
	ErrContext *codes.ErrContext
	Path       light.Path
}

DecodeError contains details about a JSON decode error.

func (DecodeError) AsError 

func (d DecodeError) AsError() error

func (DecodeError) Error 

func (d DecodeError) Error() string

type Document 

type Document struct {
	// contains filtered or unexported fields
}

Document represents a JSON document as a hierarchy of JSON data nodes.

A Document knows how to marshal or unmarshal bytes or decode/encode with streams.

A Document is immutable: it may be unmarshaled from JSON, from [Dynamic JSON] or built programmatically using the Builder.

Accessing nodes

Document.AtKey retrieves an individual keys in an object. Document.Elem does the same for an element in an array.

Iterators

The hierarchy of nodes defined by a Document may be explored using iterators like Document.Pairs and Document.Elems.

Iterators maintain the original order in which object keys and array elements have been provided.

Exploring a document

JSON pointers are supported within a Document using Document.GetPointer.

An implementation of JSONPath is provided in github.com/fredbi/core/json/documents.jsonpath to resolve JSONPath expressions as a Document iterator.

Working with values

TODO(fred): documentation

Dynamic JSON

We call "dynamic JSON" (sometimes referred to as "untyped") refers to the go structures made up of "map[string]any" and "[]any" that you typically get when the go standard library unmarshals JSON into an "any" type (aka "interface{}").

A Document may unmarshal such a data structure or may be converted into one.

In that case, due to the implementation of go maps, the order of keys in objects cannot be maintained.

This package only deals with pure JSON, not schemas.

Package github.com/fredbi/core/jsonschma brings the additional logic required to deal with JSON schemas.

Package github.com/fredbi/core/spec brings the additional logic required to deal with OpenAPI documents.

func BorrowDocument 

func BorrowDocument() *Document

func Make 

func Make(opts ...Option) Document

Make an empty Document.

The empty Document marshals as "null".

func (Document) AppendText 

func (d Document) AppendText(b []byte) ([]byte, error)

AppendText appends the JSON bytes to the provided buffer and returns the resulting slice.

func (Document) AtKey 

func (d Document) AtKey(k string) (Document, bool)

AtKey returns the value held under a key in an object, or false if not found.

Key lookup is constant-time (map index).

func (Document) Context 

func (d Document) Context() Context

Context returns the decode context of the document root, i.e a bytes count offset.

func (*Document) Decode 

func (d *Document) Decode(r io.Reader) error

Decode builds a Document from a stream of JSON bytes.

TODO: rename DecodeJSON

func (Document) Elem 

func (d Document) Elem(i int) (Document, bool)

Elem returns the i-th element of an array.

func (Document) Elems 

func (d Document) Elems() iter.Seq[Document]

Elems returns all elements in an array.

Iteration order is stable and honors the original ordering in which elements were declared.

func (Document) Encode 

func (d Document) Encode(w io.Writer) error

Encode the Document as a JSON stream to an io.Writer.

TODO: should renamed EncodeJSON

func (Document) GetPointer 

func (d Document) GetPointer(p Pointer) (Document, error)

GetPointer returns the JSON Document pointed by a JSON Pointer inside the current Document, or an error if it is not found.

func (Document) IsEmpty 

func (d Document) IsEmpty() bool

func (Document) JSONLookup 

func (d Document) JSONLookup(pointer string) (any, error)

JSONLookup implements the classical github.com/go-openapi/jsonpointer.JSONPointable interface, so users of this package can resolve JSON pointers against Document s.

The returned value is always a JSON Document.

func (Document) KeyIndex 

func (d Document) KeyIndex(k string) (int, bool)

KeyIndex returns the index of a key, of false if not found.

func (Document) Kind 

func (d Document) Kind() nodes.Kind

func (Document) Len 

func (d Document) Len() int

func (Document) LexerFactory 

func (o Document) LexerFactory() func([]byte) (lexers.Lexer, func())

func (Document) LexerFromReaderFactory 

func (o Document) LexerFromReaderFactory() func(io.Reader) (lexers.Lexer, func())

func (Document) MarshalJSON 

func (d Document) MarshalJSON() ([]byte, error)

MarshalJSON writes the Document as JSON bytes.

func (*Document) Node 

func (d *Document) Node() *light.Node

Node low-level access to the current node in the document hierarchy.

Most users won't need this "backdoor" to the internal node representation of the Document. It is however necessary when constructing constrained types on top of the Document type.

See github.com/fredbi/core/json/constrained.Object for an example.

func (Document) Pairs 

func (d Document) Pairs() iter.Seq2[string, Document]

Pairs return all (key,Node) pairs inside an object.

Iteration order is stable and honors the original ordering in which keys were declared.

func (Document) Store 

func (d Document) Store() stores.Store

func (Document) String 

func (d Document) String() string

func (*Document) UnmarshalJSON 

func (d *Document) UnmarshalJSON(data []byte) error

UnmarshalJSON builds a Document from JSON bytes.

func (Document) Value 

func (d Document) Value() (values.Value, bool)

Value of a scalar document (single node).

func (Document) WriterToWriterFactory 

func (o Document) WriterToWriterFactory() func(io.Writer) (writers.StoreWriter, func())

type DocumentFactory 

type DocumentFactory struct {
	// contains filtered or unexported fields
}

DocumentFactory is a factory that produces Document s with the same settings.

TODO: the idea with a factory is to be able to keep track of all things generated so we may recycle them

func NewDocumentFactory 

func NewDocumentFactory(opts ...Option) *DocumentFactory

NewDocumentFactory builds a factory for Document s

func (DocumentFactory) Clone 

func (f DocumentFactory) Clone(d Document) Document

func (DocumentFactory) Empty 

func (f DocumentFactory) Empty() Document

func (DocumentFactory) LexerFactory 

func (o DocumentFactory) LexerFactory() func([]byte) (lexers.Lexer, func())

func (DocumentFactory) LexerFromReaderFactory 

func (o DocumentFactory) LexerFromReaderFactory() func(io.Reader) (lexers.Lexer, func())

func (DocumentFactory) WriterToWriterFactory 

func (o DocumentFactory) WriterToWriterFactory() func(io.Writer) (writers.StoreWriter, func())

type Option 

type Option func(*options)

func WithLexer 

func WithLexer(l lexers.Lexer) Option

func WithStore 

func WithStore(s stores.Store) Option

func WithWriter 

func WithWriter(w writers.StoreWriter) Option

type Pointer 

type Pointer []stringOrInt

Pointer represents a JSON Pointer.

func MakePointer 

func MakePointer(s string) (Pointer, error)

MakePointer builds a JSON Pointer from its string representation.

RFC6901 definition of a JSON pointer:

  • may be empty
  • if not empty, must start by "/"
  • all tokens are separated by "/"
  • "/" is escaped by "~1"
  • "~" is escaped by "~0"
  • tokens representing a numerical array index are non-negative integers
  • an integer digit may be "0" or any integer without a leading "0"

Notice that this definition of a JSON pointer does not yield a unique match: token "123" would both match key "123" in an object or item 123 in an array.

func MakePointerFromElements 

func MakePointerFromElements(elems ...any) (Pointer, error)

MakePointerFromElements buils a JSON Pointer from a list of elements that constitute the search path.

Elements may be of type string, values.InternedKey or int.

String elements are not escaped.

Integer elements only apply to arrays. In this representation, we no longer have an ambiguous search that could match a key with the string representation of the integer element.

func (Pointer) String 

func (p Pointer) String() string

String representation of a JSON pointer, with escaping rules as per RFC 6901

type VerbatimDocument 

type VerbatimDocument struct {
}

VerbatimDocument holds a JSON document verbatim.

All original marks kepts, including non-significant white space and escaped unicode points.

TODO(fred)

Source Files

View all Source files

Directories

Path Synopsis
Package constrained exposes constrained json.Document types.
 Package constrained exposes constrained json.Document types.
Package dynamic converts a json.Document to dynamic JSON and vice versa.
 Package dynamic converts a json.Document to dynamic JSON and vice versa.
Package lexers exposes interfaces for lexing JSON.
 Package lexers exposes interfaces for lexing JSON.
default-lexer
Package lexer provides a JSON lexer.
 Package lexer provides a JSON lexer.
error-codes
Package codes exposes common lexer errors.
 Package codes exposes common lexer errors.
ld-lexer
Package lexer exposes lexer for ld-json (line-delimited JSON).
 Package lexer exposes lexer for ld-json (line-delimited JSON).
token
Package token defines the JSON token types with their kind.
 Package token defines the JSON token types with their kind.
Package nodes declares node types for JSON documents.
 Package nodes declares node types for JSON documents.
error-codes
light
Package light exposes an implementation for nodes.
 Package light exposes an implementation for nodes.
Package stores exposes the interface to work with a JSON Store.
 Package stores exposes the interface to work with a JSON Store.
default-store
Package store provides default implementations for stores.Store.
 Package store provides default implementations for stores.Store.
values
Package types defines common types to work with json.
 Package types defines common types to work with json.
Package writers exposes interfaces to write JSON data.
 Package writers exposes interfaces to write JSON data.
default-writer
Package writer exposes an implementation of the JSON writer interface writers.Writer.
 Package writer exposes an implementation of the JSON writer interface writers.Writer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.