rdb

package module
v1.0.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 9, 2026 License: MIT Imports: 13 Imported by: 0

README

rdb Build Status

rdb is a Go package that implements parsing and encoding of the Redis RDB file format.

This package was heavily inspired by redis-rdb-tools by Sripathi Krishnan.

Documentation

rdb是一个Go包,实现了对redis rdb文件的解析和编码。

  • 基于github.com/dongmx/rdb进行的二次开发,实现了redis7+支持。
  • 本包支持redis rdb 版本为 1 <= version <= 12

注:dongmx/rdb 源自github.com/cupcake/rdb,仅支持redis6,2019年10月停止维护。

Installation

go get github.com/cupcake/rdb

开发

rdb 文件解析,请阅读 Redis-RDB-Dump-File-FormatDocumentation For cupcake/rdb

  • redis7+支持,\core\下的源码参考了 RedisShake ,它是阿里云 Tair 团队 积极维护的一个用于处理和迁移 Redis 数据的工具。在此,表示感谢!

  • redis7+,rdb文件解析主要是解决listpack数据类型问题。鉴于 redis stream 用于消息队列,我们通常不用redis作为mq,因此stream增加的类型未处理。

//Decode parses a RDB file from r and calls the decode hooks on d.
//Decode 从 r 解析 RDB 文件并调用 d 上的解码挂钩.
func Decode(r io.Reader, d Decoder) error

//Decode a byte slice from the Redis DUMP command. The dump does not contain the database, key or expiry, so they must be included in the function call (but can be zero values).
//从 Redis DUMP 命令中解码字节片。转储不包含数据库、密钥或过期时间,因此它们必须包含在函数调用中.
func DecodeDump(dump []byte, db int, key []byte, expiry int64, d Decoder) error

//A Decoder must be implemented to parse a RDB file. 必须实现解码器来解析 RDB 文件。
type Decoder interface {
	// StartRDB is called when parsing of a valid RDB file starts.
	StartRDB()
	...
}

常见问题

Q:为什么使用命令(memory usage )获取的和rdr算的总是不一致 A:Key和value所对应的struct和指针大小。在jemalloc分配后,字节对齐部分所占用的大小也会计算在used_memory中 无论是用命令还是rdr都计算了这两块,为什么不一致?可读下 https://blog.csdn.net/f80407515/article/details/122387859

Q:如何处理报错decode rdbfile error: rdb: unknown object type 116 for key? A:该报错表示实例中存在非标准或新版本增加的数据结构,暂不支持分析,你可以在还原到测试实例删除后再进行分析。

Q:为什么Redis缓存分析中String类型Key的元素数量和元素长度是一样的? A:在Redis缓存分析中,针对String类型的Key,其元素数量就是其元素长度。

Q:Redis缓存分析的前缀分隔符是什么? A:目前Redis缓存分析的前缀分隔符是按照固定的前缀:;,_-+@=|# 区分的字符串。

Q:各key的内存占用为什么比HDT3213/rdb算的大28? A:HDT3213/rdb V.1.3.0没有计算lru_bits占用,lru_bits默认占用24比特位,而本工具将起计算在内了,请看源码d.m.TopLevelObjOverhead。

贡献

欢迎社区的贡献。对于重大变更,请先开一个 issue 来讨论你想要改变的内容。如果想共同维护此项目,可以加我微信(Sd-LiYanJing)。

特别感兴趣的是:

  1. 随着redis版本变化,增加新类型的解析支持
  2. 优化、改善代码,提升性能

Documentation

Overview

Package rdb implements parsing and encoding of the Redis RDB file format.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Decode

func Decode(r io.Reader, d Decoder) error

Decode parses a RDB file from r and calls the decode hooks on d.

func DecodeDump

func DecodeDump(dump []byte, db int, key []byte, expiry int64, d Decoder) error

DecodeDump a byte slice from the Redis DUMP command. The dump does not contain the database, key or expiry, so they must be included in the function call (but can be zero values).

Types

type Decoder

type Decoder interface {
	// StartRDB is called when parsing of a valid RDB file starts.
	StartRDB(ver int)
	// StartDatabase is called when database n starts.
	// Once a database starts, another database will not start until EndDatabase is called.
	StartDatabase(n int)
	// AUX field
	Aux(key, value []byte)
	// ResizeDB hint
	ResizeDatabase(dbSize, expiresSize uint32)
	// Set is called once for each string key.
	Set(key, value []byte, expiry int64, info *Info)
	// StartHash is called at the beginning of a hash.
	// Hset will be called exactly length times before EndHash.
	StartHash(key []byte, length, expiry int64, info *Info)
	// Hset is called once for each field=value pair in a hash.
	Hset(key, field, value []byte)
	// EndHash is called when there are no more fields in a hash.
	EndHash(key []byte)
	// StartSet is called at the beginning of a set.
	// Sadd will be called exactly cardinality times before EndSet.
	StartSet(key []byte, cardinality, expiry int64, info *Info)
	// Sadd is called once for each member of a set.
	Sadd(key, member []byte)
	// EndSet is called when there are no more fields in a set.
	EndSet(key []byte)
	// StartStream is called at the beginning of a stream.
	// Xadd will be called exactly length times before EndStream.
	StartStream(key []byte, cardinality, expiry int64, info *Info)
	// Xadd is called once for each id in a stream.
	Xadd(key, id, listpack []byte)
	// EndHash is called when there are no more fields in a hash.
	EndStream(key []byte, items uint64, lastEntryID string, cgroupsData StreamGroups)
	// StartList is called at the beginning of a list.
	// Rpush will be called exactly length times before EndList.
	// If length of the list is not known, then length is -1
	StartList(key []byte, length, expiry int64, info *Info)
	// Rpush is called once for each value in a list.
	// rdb v1.0.8增加NodeEncodings是为了支持redis7+的Quicklist2,qucklist2中节点有两种编码1和2,其他数据类型传0
	Rpush(key, value []byte, NodeEncodings uint64)
	// EndList is called when there are no more values in a list.
	EndList(key []byte)
	// StartZSet is called at the beginning of a sorted set.
	// Zadd will be called exactly cardinality times before EndZSet.
	StartZSet(key []byte, cardinality, expiry int64, info *Info)
	// Zadd is called once for each member of a sorted set.
	Zadd(key []byte, score float64, member []byte)
	// EndZSet is called when there are no more members in a sorted set.
	EndZSet(key []byte)
	// EndDatabase is called at the end of a database.
	EndDatabase(n int)
	// EndRDB is called when parsing of the RDB file is complete.
	EndRDB()
}

A Decoder must be implemented to parse a RDB file.

type Info added in v1.0.3

type Info struct {
	Encoding    string
	Idle        uint64
	Freq        int
	SizeOfValue int
	Zips        uint64
	ListPacks   uint64
}

type StreamConsumerData added in v1.0.3

type StreamConsumerData struct {
	Name       []byte
	SeenTime   uint64
	Pending    []*StreamConsumerPendingEntry
	ActiveTime uint64
}

type StreamConsumerPendingEntry added in v1.0.3

type StreamConsumerPendingEntry struct {
	ID []byte
}

type StreamGroup added in v1.0.3

type StreamGroup struct {
	Name        []byte
	LastEntryId string
	Pending     []*StreamPendingEntry
	Consumers   []*StreamConsumerData
}

type StreamGroups added in v1.0.3

type StreamGroups []*StreamGroup

type StreamId added in v1.0.8

type StreamId struct {
	Ms       uint64 `json:"ms"`
	Sequence uint64 `json:"sequence"`
}

rdd v1.0.8 2026-01-09 add stream 消息队列 支持redis7增加的两个存储类型TypeStreamListPacks2=19,rdbTypeStreamListpacks3=21 此部分参考的 github.com/linyue515/rdr 等有精力和时间了,再对比RedisShake进行梳理改善

type StreamPendingEntry added in v1.0.3

type StreamPendingEntry struct {
	ID            *StreamId
	DeliveryTime  uint64
	DeliveryCount uint64
}

type ValueType

type ValueType byte

ValueType of redis type

const (
	TypeString  ValueType = 0 // RDB_TYPE_STRING
	TypeList    ValueType = 1
	TypeSet     ValueType = 2
	TypeZSet    ValueType = 3
	TypeHash    ValueType = 4 // RDB_TYPE_HASH
	TypeZSet2   ValueType = 5 // ZSET version 2 with doubles stored in binary.
	TypeModule  ValueType = 6 // RDB_TYPE_MODULE
	TypeModule2 ValueType = 7 // RDB_TYPE_MODULE2 Module value with annotations for parsing without the generating module being loaded.

	// Object types for encoded objects.
	TypeHashZipMap      ValueType = 9
	TypeListZipList     ValueType = 10
	TypeSetIntSet       ValueType = 11
	TypeZSetZipList     ValueType = 12
	TypeHashZipList     ValueType = 13
	TypeListQuickList   ValueType = 14 // RDB_TYPE_LIST_QUICKLIST
	TypeStreamListPacks ValueType = 15 // RDB_TYPE_STREAM_LISTPACKS,

	//rdb v1.0.5 add,注:Redis Stream 主要用于消息队列,虽然我们一般不会用它,参考 github.com/linyue515/rdr 做了支持,后期有精力时间时再参考RedisShake进行梳理
	TypeHashListPack     ValueType = 16 // RDB_TYPE_HASH_ZIPLIST
	TypeZSetListPack     ValueType = 17 // RDB_TYPE_ZSET_LISTPACK
	TypeListQuickList2   ValueType = 18 // DB_TYPE_LIST_QUICKLIST_2 https://github.com/redis/redis/pull/9357
	TypeStreamListPacks2 ValueType = 19 // RDB_TYPE_STREAM_LISTPACKS2
	TypeSetListPack      ValueType = 20 // RDB_TYPE_SET_LISTPACK
	TypeStreamListPacks3 ValueType = 21 // RDB_TYPE_STREAM_LISTPACKS_3

	// https://github.com/redis/redis/pull/13391
	TypeHashMetadataPreGa ValueType = 22 // RDB_TYPE_HASH_METADATA_PRE_GA
	TypeHashListPackExPre ValueType = 23 // RDB_TYPE_HASH_LISTPACK_EX_PRE_GA
	TypeHashMetaData      ValueType = 24 // RDB_TYPE_HASH_METADATA
	TypeHashListPackEx    ValueType = 25 // RDB_TYPE_HASH_LISTPACK_EX

)

types value string: TypeString list: TypeList, TypeListZipList, TypeListQuickList, TypeListQuickList2 set: TypeSet, TypeSetIntSet, TypeSetListPack Sorted Set(zset): TypeZSet, TypeZSet2, TypeZSetZipList, TypeZSetListPack hash: TypeHash, TypeHashZipMap, TypeHashZipList, TypeHashListPack, TypeHashMetadataPreGa, TypeHashListPackExPre, TypeHashMetaData, TypeHashListPackEx 注:Redis7.0开始使用listpack替代了ziplist,小于阈值时使用listpack

Directories

Path Synopsis
core
Package crc64 implements the Jones coefficients with an init value of 0.
Package crc64 implements the Jones coefficients with an init value of 0.
This is a very basic example of a program that implements rdb.decoder and outputs a human readable diffable dump of the rdb file.
This is a very basic example of a program that implements rdb.decoder and outputs a human readable diffable dump of the rdb file.
internal
log

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL