scan

package
v0.0.0-...-dd6fa2d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 27, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package scan implements functionality for scanning Swarming datastore.

It is used by `monitor` binary.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ActiveTasks 

func ActiveTasks(ctx context.Context, visitors []TaskVisitor) error

ActiveTasks visits all pending and running tasks.

Returns a multi-error with the scan error (if any) and errors from all visitors' finalizers (if any) in arbitrary order.

func Bots 

func Bots(ctx context.Context, visitors []BotVisitor) error

Bots visits all bots via multiple parallel queries.

Takes a list of all registered visitors. Will run only ones that are due for execution now per their run frequency.

Returns a multi-error with the scan error (if any) and errors from all visitors' finalizers (if any) in arbitrary order.

Types

type ActiveJobsReporter 

type ActiveJobsReporter struct {
	// ServiceName is a service name to put into metrics' target.
	ServiceName string
	// JobName is a job name to put into metrics' target.
	JobName string
	// Monitor to use to flush metrics.
	Monitor monitor.Monitor
	// contains filtered or unexported fields
}

ActiveJobsReporter is TaskVisitor that reports the number of active jobs per combination of dimensions to monitoring.

func (*ActiveJobsReporter) Finalize 

func (r *ActiveJobsReporter) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of TaskVisitor interface.

func (*ActiveJobsReporter) Prepare 

func (r *ActiveJobsReporter) Prepare(ctx context.Context)

Prepare prepares the visitor state.

Part of TaskVisitor interface.

func (*ActiveJobsReporter) Visit 

func (r *ActiveJobsReporter) Visit(ctx context.Context, trs *model.TaskResultSummary)

Visit is called for every visited task.

Part of TaskVisitor interface.

type BotVisitor 

type BotVisitor interface {
	// ID returns an unique identifier of this visitor used for storing its state.
	ID() string

	// Frequency returns how frequently this visitor should run.
	//
	// This is approximate. Granularity would be ~= 1 min. Value of 0 means this
	// visitor will approximately run every minute.
	Frequency() time.Duration

	// Prepare prepares the visitor to use `shards` parallel queries.
	//
	// It should initialize per-shard state used by Visit. `lastRun` is the last
	// time this visitor ran or a zero time if this is the first time it is
	// running.
	Prepare(ctx context.Context, shards int, lastRun time.Time)

	// Visit is called for every bot.
	//
	// The overall scan is split into multiple shards. Within a shard the scan is
	// sequential, but shards themselves are processed in parallel.
	Visit(ctx context.Context, shard int, bot *model.BotInfo)

	// Finalize is called once the scan is done.
	//
	// It is passed an error if the scan was incomplete. If the scan was complete,
	// (i.e. Visit visited all bots), it receives nil.
	//
	// The returned error will be reported as an overall scan error.
	Finalize(ctx context.Context, scanErr error) error
}

BotVisitor examines bots in parallel.

Should keep track of errors internally during the scan, reporting them only in the end in Finalize.

type BotsDimensionsAggregator 

type BotsDimensionsAggregator struct {
	// contains filtered or unexported fields
}

BotsDimensionsAggregator is BotVisitor that collects the set of all possible bot dimensions and stores it in the datastore.

The result is used by GetBotDimensions RPCs.

func (*BotsDimensionsAggregator) Finalize 

func (a *BotsDimensionsAggregator) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Frequency 

func (*BotsDimensionsAggregator) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) ID 

func (*BotsDimensionsAggregator) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Prepare 

func (a *BotsDimensionsAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*BotsDimensionsAggregator) Visit 

func (a *BotsDimensionsAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type BotsMetricsReporter 

type BotsMetricsReporter struct {
	// ServiceName is a service name to put into metrics' target.
	ServiceName string
	// JobName is a job name to put into metrics' target.
	JobName string
	// Monitor to use to flush metrics.
	Monitor monitor.Monitor
	// contains filtered or unexported fields
}

BotsMetricsReporter is BotVisitor that reports stats about bots to the monitoring.

func (*BotsMetricsReporter) Finalize 

func (r *BotsMetricsReporter) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Frequency 

func (*BotsMetricsReporter) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*BotsMetricsReporter) ID 

func (*BotsMetricsReporter) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Prepare 

func (r *BotsMetricsReporter) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*BotsMetricsReporter) Visit 

func (r *BotsMetricsReporter) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type DeadBotDetector 

type DeadBotDetector struct {
	// BotDeathTimeout is how long a bot must be away before being declared dead.
	BotDeathTimeout time.Duration
	// TasksManager is used to change state of tasks.
	TasksManager tasks.Manager
	// contains filtered or unexported fields
}

DeadBotDetector is a BotVisitor that recognizes bots that haven't been seen for a while and moves them into "DEAD" state, terminating whatever tasks they were running with "BOT_DIED" status.

func (*DeadBotDetector) Finalize 

func (r *DeadBotDetector) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*DeadBotDetector) Frequency 

func (*DeadBotDetector) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*DeadBotDetector) ID 

func (*DeadBotDetector) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*DeadBotDetector) Prepare 

func (r *DeadBotDetector) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*DeadBotDetector) Visit 

func (r *DeadBotDetector) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type NamedCachesAggregator 

type NamedCachesAggregator struct {
	// contains filtered or unexported fields
}

NamedCachesAggregator is a BotVisitor that collects all named_caches present on the bot and their respective size.

func (*NamedCachesAggregator) Finalize 

func (a *NamedCachesAggregator) Finalize(ctx context.Context, scanErr error) error

Finalize is called once the scan is done.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Frequency 

func (*NamedCachesAggregator) Frequency() time.Duration

Frequency returns how frequently this visitor should run.

Part of BotVisitor interface.

func (*NamedCachesAggregator) ID 

func (*NamedCachesAggregator) ID() string

ID returns an unique identifier of this visitor used for storing its state.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Prepare 

func (a *NamedCachesAggregator) Prepare(ctx context.Context, shards int, lastRun time.Time)

Prepare prepares the visitor to use `shards` parallel queries.

Part of BotVisitor interface.

func (*NamedCachesAggregator) Visit 

func (a *NamedCachesAggregator) Visit(ctx context.Context, shard int, bot *model.BotInfo)

Visit is called for every bot.

Part of BotVisitor interface.

type SliceExpirationEnforcer 

type SliceExpirationEnforcer struct {
	// GracePeriod is how long to wait past actual expiration before doing
	// anything.
	//
	// This exists to make sure we do not race with the RBE's PubSub-based slice
	// expiration mechanism, to avoid unnecessary datastore contention.
	GracePeriod time.Duration

	// TasksManager is used to change state of tasks.
	TasksManager tasks.Manager

	// Config is a snapshot of the server configuration.
	Config *cfg.Config
	// contains filtered or unexported fields
}

SliceExpirationEnforcer is TaskVisitor that updates tasks whose current task slice (represented by a TaskToRun entity) has expired.

Normally a TaskToRun entity is associated with an RBE reservation and RBE itself keeps track of the expiration, sending us PubSub notifications when they happen (see expireSliceBasedOnReservation in ReservationServer).

But sometimes these notifications either get significantly delayed or, maybe, do not arrive at all (hard to tell). This is when this scanner kicks in.

func (*SliceExpirationEnforcer) Finalize 

func (s *SliceExpirationEnforcer) Finalize(ctx context.Context, _ error) error

Finalize implements TaskVisitor.

func (*SliceExpirationEnforcer) Prepare 

func (s *SliceExpirationEnforcer) Prepare(ctx context.Context)

Prepare implements TaskVisitor.

func (*SliceExpirationEnforcer) Visit 

func (s *SliceExpirationEnforcer) Visit(ctx context.Context, trs *model.TaskResultSummary)

Visit implements TaskVisitor.

type TaskVisitor 

type TaskVisitor interface {
	// Prepare prepares the visitor state.
	//
	// It should initialize the state used by Visit.
	Prepare(ctx context.Context)

	// Visit is called for every visited task.
	//
	// Called sequentially from a single goroutine.
	Visit(ctx context.Context, task *model.TaskResultSummary)

	// Finalize is called once the scan is done.
	//
	// It is passed an error if the scan was incomplete. If the scan was complete,
	// (i.e. Visit visited all tasks), it receives nil.
	//
	// The returned error will be reported as an overall scan error.
	Finalize(ctx context.Context, scanErr error) error
}

TaskVisitor examines tasks, **sequentially**.

Should keep track of errors internally during the scan, reporting them only in the end in Finalize.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.