web

package

v0.1.14 Latest Latest Go to latest Published: Jul 8, 2025 License: BSD-3-Clause Imports: 9 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

codeberg.org/derat/yambs

Links

Open Source Insights

Documentation ¶

Overview ¶

Package web interacts with web pages.

Index ¶

Variables
func GetAttr(n *html.Node, attr string) (string, bool)
func GetText(n *html.Node, addSpaces bool) string
func Open(ctx context.Context, method, url string, opts ...OpenOption) (*http.Response, error)
func SetUserAgent(ua string)
type OpenOption
- func OpenSetHeader(k, v string) OpenOption
type Page
- func FetchPage(ctx context.Context, url string) (*Page, error)
- func (p *Page) Query(query string) QueryResult
- func (p *Page) QueryAll(query string) QueryAllResult
type QueryAllResult
- func (res QueryAllResult) Text(addSpaces bool) ([]string, error)
type QueryResult
- func QueryNode(root *html.Node, query string) QueryResult
- func (res QueryResult) Attr(attr string) (string, error)
- func (res QueryResult) Text(addSpaces bool) (string, error)

Constants ¶

This section is empty.

Variables ¶

View Source

var OpenDisableCompression = func(_ *http.Request, cl *http.Client) {
	if cl.Transport == nil {
		cl.Transport = &http.Transport{}
	}

	cl.Transport.(*http.Transport).DisableCompression = true
}

OpenDisableCompression disables compression so that Content-Length will be usable.

Functions ¶

func GetAttr ¶ added in v0.1.14

func GetAttr(n *html.Node, attr string) (string, bool)

GetAttr returns the first occurrence of the named attribute in n. If the attribute is not present, it returns an empty string and false.

func GetText ¶

func GetText(n *html.Node, addSpaces bool) string

GetText concatenates all text content in and under n.

func Open ¶ added in v0.1.14

func Open(ctx context.Context, method, url string, opts ...OpenOption) (*http.Response, error)

Open returns an HTTP connection to the supplied URL, following redirects. An error and a nil response are returned for non-200 status codes. The caller is responsible for closing the response's Body field.

func SetUserAgent ¶

func SetUserAgent(ua string)

SetUserAgent sets a value for the "User-Agent" header to be sent in all future HTTP requests.

Types ¶

type OpenOption ¶ added in v0.1.14

type OpenOption func(req *http.Request, cl *http.Client)

OpenOption configures Open's behavior.

func OpenSetHeader ¶ added in v0.1.14

func OpenSetHeader(k, v string) OpenOption

OpenSetHeader sets the named header to the supplied value.

type Page ¶

type Page struct {
	Root *html.Node
}

Page represents a parsed HTML page.

func FetchPage ¶

func FetchPage(ctx context.Context, url string) (*Page, error)

FetchPage fetches and parses the HTML page at the supplied URL.

func (*Page) Query ¶

func (p *Page) Query(query string) QueryResult

Query calls QueryNode using p.Root.

func (*Page) QueryAll ¶

func (p *Page) QueryAll(query string) QueryAllResult

QueryAll returns all nodes matched by the supplied CSS selector. Unlike Query/QueryNode, an error is not returned if no nodes are matched.

type QueryAllResult ¶

type QueryAllResult struct {
	Nodes []*html.Node
	Err   error
}

QueryAllResult contains the result of a call to QueryAll.

func (QueryAllResult) Text ¶

func (res QueryAllResult) Text(addSpaces bool) ([]string, error)

Text returns the contents of all text nodes under res.Nodes.

type QueryResult ¶

type QueryResult struct {
	Node *html.Node
	Err  error
}

QueryResult contains the result of a call to Query or QueryNode.

func QueryNode ¶

func QueryNode(root *html.Node, query string) QueryResult

QueryNode returns the first node matched by the supplied CSS selector. The returned result has a non-nil Err field if no node was matched.

func (QueryResult) Attr ¶

func (res QueryResult) Attr(attr string) (string, error)

Attr returns the first occurrence of the named attribute. An error is returned if the attribute isn't present.

func (QueryResult) Text ¶

func (res QueryResult) Text(addSpaces bool) (string, error)

Text recursively concatenates the contents of all child text nodes.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL