Failure to symbolize binary #5291
Description
Parca can't symbolize addresses for ClickHouse binaries.
How to reproduce
First, download and decompress ClickHouse binary:
curl -LO https://s3.amazonaws.com/clickhouse-builds/24.10/2e054d555719f509b96e0cfea4d5f5819c69a61f/package_release/clickhouse
chmod +x clickhouse
./clickhouse --version
Normal flow reproducer
Then, execute ClickHouse server:mkdir ch-data && cd ch-data
../clickhouse server
Try to profile it with Parca (you will need to compile parca-agent with the latest version of the upstream OTel eBPF profiler to get the fix for this). Also use --debuginfo-upload-max-size=2244458216 to be able to upload the binary debuginfo. You will observe that stacks are missing symbols in the UI.
Smaller reproducer
Compile and run this as a binary in this repo (pardon my poor Go):
package main
import (
"context"
"debug/elf"
"os"
"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/parca-dev/parca/pkg/symbol/addr2line"
"github.com/parca-dev/parca/pkg/symbol/demangle"
)
func main() {
ctx := context.Background()
path := "clickhouse"
f, err := elf.Open(path)
if err != nil {
print(err)
}
demangler := demangle.NewDemangler("simple", false)
writer := log.NewSyncWriter(os.Stdout)
logger := log.NewLogfmtLogger(writer)
liner, err := addr2line.DWARF(logger, path, f, demangler)
if err != nil {
print(err)
}
lines, err := liner.PCToLines(ctx, 0x1192c860)
if err != nil {
level.Error(logger).Log("err", err)
}
}Error logs
These come ultimately from debug/dwarf/entry.go, method dwarf5Ranges.
Parca:
level=debug name=parca ts=2024-11-15T12:39:31.37757694Z caller=symbolizer.go:172 component=symbolizer msg="failed to get lines" err="liner pctolines: failed to extract dwarf tree: invalid rnglist offset 409180350 (max 636512)"
Reproducer bin:
level=error err="failed to extract dwarf tree: invalid rnglist offset 2858131463 (max 636512)"
Cross-checking with other tools
Using gimli's addr2line (0x1192c860 is one of the addresses it fails to symbolize):
$ ./addr2line-gimli -e clickhouse -fC 0x1192c860
DB::AggregateUtils::isAggregateFunction(DB::ASTFunction const&)
./build_docker/./src/AggregateFunctions/AggregateFunctionFactory.h:119
Also to note, I've checked some of the generated stacks with this addr2line implementation and they seem to make sense, so stack walking seems to be producing valid addresses.
Other comments
Any tips to debug or cross-check this with dwarfdump or other tools are welcome. For example, how could I check if the offsets that the entry offsets the code is parsing at each step are valid?