[Bug binutils/18616] New: oprofile fails with "BFD: Dwarf Error: found dwarf version '64617', this reader only handles version 2, 3 and 4 information"

[daniel.santos at pobox dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

Bug ID: 18616
Summary: oprofile fails with "BFD: Dwarf Error: found dwarf
version '64617', this reader only handles version 2, 3
and 4 information"
Product: binutils
Version: 2.25
Status: NEW
Severity: normal
Priority: P2
Component: binutils
Assignee: unassigned at sourceware dot org
Reporter: daniel.santos at pobox dot com
Target Milestone: ---

This is on gentoo where I've built most of my system with -flto and
importantly,the FEATURES="compressdebug splitdebug". My guess is that for some
reason bdf_openr is failing to realize that the debug file is compressed, as
the purported dwarf version is different for each type I get this error (which
appears to be for every object file involved). For the session in question,
there I get this error running "opannotate --source 350,225" times and the
version ranges from 0 to 65535, so I presume this is just a two byte field to
which it's picking from compressed data w/o decompressing it (so the struct is
just crap).

I had similar errors previously attempting to merge packages with -flto until I
learned that you can tell portage to use the gcc wrappers for ar, ranlib and nm

AR=gcc-ar
NM=gcc-nm
RANLIB=gcc-ranlib

I have this error on two different machines with gentoo built similarly, so I'm
not sure what isn't happening in binutils for it to fail, but I presume that
this isn't oprofile.



# emerge --info sys-devel/binutils dev-util/oprofile
!!! Repository 'sunrise' has sync-uri attribute, but is missing sync-type
attribute
Portage 2.2.18 (python 2.7.9-final-0, default/linux/amd64/13.0/desktop/kde,
gcc-4.9.2, glibc-2.20-r2, 3.17.8-gentoo-r1 x86_64)
=================================================================
System Settings
=================================================================
System uname:
Linux-3.17.8-gentoo-r1-x86_64-Intel-R-_Core-TM-***@_2.00GHz-with-gentoo-2.2
KiB Mem: 4043108 total, 1668760 free
KiB Swap: 4194300 total, 3845300 free
Timestamp of repository gentoo: Sat, 06 Jun 2015 19:00:01 +0000
sh bash 4.3_p33-r2
ld GNU ld (Gentoo 2.25 p1.0) 2.25
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
ccache version 3.2.1 [disabled]
app-shells/bash: 4.3_p33-r2::gentoo
dev-java/java-config: 2.2.0::gentoo
dev-lang/perl: 5.20.2::gentoo
dev-lang/python: 2.7.9-r2::gentoo, 3.3.5-r1::gentoo, 3.4.3::gentoo
dev-util/ccache: 3.2.1-r1::gentoo
dev-util/cmake: 3.2.1::gentoo
dev-util/pkgconfig: 0.28-r2::gentoo
sys-apps/baselayout: 2.2::gentoo
sys-apps/openrc: 0.13.11::gentoo
sys-apps/sandbox: 2.6-r1::gentoo
sys-devel/autoconf: 2.13::gentoo, 2.69-r1::gentoo
sys-devel/automake: 1.11.6-r1::gentoo, 1.12.6::gentoo, 1.14.1::gentoo,
1.15::gentoo
sys-devel/binutils: 2.25::gentoo
sys-devel/gcc: 4.8.4::gentoo, 4.9.2::gentoo
sys-devel/gcc-config: 1.8::gentoo
sys-devel/libtool: 2.4.6-r1::gentoo
sys-devel/make: 4.1-r1::gentoo
sys-kernel/linux-headers: 3.19::gentoo (virtual/os-headers)
sys-libs/glibc: 2.20-r2::gentoo
Repositories:

gentoo
location: /usr/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
priority: -1000

x-portage
location: /usr/local/portage
masters: gentoo
priority: 0

java
location: /var/lib/layman/java
masters: gentoo
priority: 50

rion
location: /var/lib/layman/rion
masters: gentoo
priority: 50

sunrise
location: /var/lib/layman/sunrise
sync-uri: git://anongit.gentoo.org/proj/sunrise-reviewed.git
masters: gentoo
priority: 50

zugaina
location: /var/lib/layman/zugaina
masters: gentoo
priority: 50

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA cadsoft"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -g -pipe -march=native -mtune=native -flto"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config
/usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d
/etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild
/etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -g -pipe -march=native -mtune=native -flto"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="-j1"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg candy cgroup compress-build-logs
compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles
installsources merge-sync news parallel-fetch preserve-libs protect-owned
sandbox sfperms split-elog split-log splitdebug strict unknown-features-warn
unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://loudmouth
http://gentoo.mirrors.easynews.com/linux/gentoo/
http://gentoo.cites.uiuc.edu/pub/gentoo/
http://www.gtlib.gatech.edu/pub/gentoo"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times
--omit-dir-times --compress --force --whole-file --delete --stats
--human-readable --timeout=180 --exclude=/distfiles --exclude=/local
--exclude=/packages"
PORTAGE_TMPDIR="/tmp"
USE="7zip X a52 aac aacplus aacs acl acpi aio alsa amd64 apache2 bazaar berkdb
bindist bluetooth bluray branding bs2b bzip2 cairo cdda cddb cdio cdr celt
clang cli consolekit cracklib crypt cups custom-cflags cvs cxx dbus declarative
designer development dirac directfb dot dri dts dvd dvdr emboss encode exif
extraengine faac fam ffmpeg fftw firefox flac fluidsynth fontconfig fortran
frei0r games gcj gdbm gif git glamor gme gnutls gphoto2 gpm graphviz gsm
gstreamer gtk gtk3 hardened highlight iconv ipv6 java jpeg jpeg2k kate kde
kdrive kipi ladspa lame lcms ldap libass libcaca libnotify libsamplerate libv4l
lto lua lzma lzo mad matroska md5sum mercurial midi mmx mmxext mng modules mp3
mp4 mpeg mtp mudflap multicall multilib multitarget musepack natspec ncurses
networkmanager nls nptl nsplugin objc objc++ objc-gc offensive ogg okteta omr
openal opencl opengl openmp opus pam pango pch pcre pdf perl phonon plasma
playlist png policykit postproc ppds projectm qt3support qt4 qthelp quvi rar
readline reviewboard rfc3779 sasl scanner schroedinger sdl seccomp
semantic-desktop session sid smp speex spell sqlite sqlite3 sse sse2 ssh ssl
startup-notification static-libs subversion svg syslog taglib tcpd theora
threads thumbnail tiff tk tools tremor truetype twolame udev udisks unicode
upnp upnp-av upower usb utils v4l vaapi vcdx vdpau vim-syntax vlc vorbis vpx
wavpack webkit webp wifi wxwidgets x264 x265 xattr xcb xcomposite xft xine
xinerama xml xrandr xscreensaver xv xvid xvmc zeroconf zlib zvbi" ABI_X86="64
32" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb
unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default
authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner
authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache
env expires ext_filter file_cache filter headers include info log_config logio
mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id
userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets
stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df
interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse
sse2 sse3 ssse3 sse4_1" CURL_SSL="gnutls" ELIBC="glibc"
FOO2ZJS_DEVICES="hpp1006" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18
garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle
oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx"
GRUB_PLATFORMS="multiboot pc qemu" INPUT_DEVICES="evdev keyboard mouse joystick
wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216
lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console
presenter-minimizer" LINGUAS="en_US en" OFFICE_IMPLEMENTATION="libreoffice"
PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7
python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="nvidia"
XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface
geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac
delude chaos account"
Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND,
PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
Package Settings
=================================================================

sys-devel/binutils-2.25::gentoo was built with the following:
USE="cxx multitarget nls (policykit) static-libs zlib (-multislot) -test
-vanilla" ABI_X86="64"
CFLAGS="-O2 -ggdb -pipe -march=core2 -mtune=core2"
CXXFLAGS="-O2 -ggdb -pipe -march=core2 -mtune=core2"


dev-util/oprofile-1.0.0-r1::gentoo was built with the following:
USE="java pch" ABI_X86="64"
CFLAGS="-O2 -ggdb -pipe -march=core2 -mtune=core2 -mcx16 -msahf -msse4.1 -mfxsr
--param l1-cache-size=32 --param l1-cache-line-size=64 --param
l2-cache-size=3072"
CXXFLAGS="-O2 -ggdb -pipe -march=core2 -mtune=core2 -mcx16 -msahf -msse4.1
-mfxsr --param l1-cache-size=32 --param l1-cache-line-size=64 --param
l2-cache-size=3072"

[nickc at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

Nick Clifton <nickc at redhat dot com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |WAITING
CC| |nickc at redhat dot com

--- Comment #1 from Nick Clifton <nickc at redhat dot com> ---
Hi Daniel,

Please could you upload a test binary that demonstrates this problem, so that
we can investigate further.

Cheers
Nick

[Marco.Leise at gmx dot de]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

Marco Leise <Marco.Leise at gmx dot de> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |Marco.Leise at gmx dot de

--- Comment #2 from Marco Leise <Marco.Leise at gmx dot de> ---
I have a similar Gentoo setup with binutils 2.29.1 and see the same errors.
Removing "compressdebug" from FEATURES during installation of glibc fixed the
problem for me.

From the Gentoo documentation:

compressdebug: Compress the debug sections in the split debug files with zlib
to save space. Make sure you have built both binutils and gdb with USE=zlib
support for this to work. See splitdebug for general split debug information
(upon which this feature depends).


The instructions seem outdated, as neither binutils nor gdb currently offer the
"zlib" USE-flag. A USE-flag controls configuration options and dependencies of
a package about to be compiled and installed on Gentoo. Maybe it is
auto-detected now. In any case zlib is installed.
This was the patch that introduced "compressdebug" in Gentoo:
https://archives.gentoo.org/gentoo-portage-dev/message/fabff63fccb0bce10b4f93fcf4acf3e1
You can see the compiler/linker flags used to make it work.

[Marco.Leise at gmx dot de]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #3 from Marco Leise <Marco.Leise at gmx dot de> ---
Created attachment 10938
--> https://sourceware.org/bugzilla/attachment.cgi?id=10938&action=edit
Compressed glibc amd64 debug information from a Gentoo system.

[nickc at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #4 from Nick Clifton <nickc at redhat dot com> ---
Hi Marco,

Post by Marco.Leise at gmx dot de
I have a similar Gentoo setup with binutils 2.29.1 and see the same errors.

This is surprising.

Post by Marco.Leise at gmx dot de
The instructions seem outdated, as neither binutils nor gdb currently offer
the "zlib" USE-flag. A USE-flag controls configuration options and
dependencies of a package about to be compiled and installed on Gentoo.
Maybe it is auto-detected now.

Yes, it is.

I ran the debug files in the tarball you uploaded through objdump (which
uses the BFD library) and readelf (which does not) from the 2.29 binutils
release, and they were able to display the debug information without any
problems. (Admittedly this was using the Fedora 27 binutils, not Gentoo).

I did not try using oprofile, but that is because I do not know how to use
it. Do you have an example command line that I could try ?

Detection of compressed sections relies upon a special bit in the ELF
section header flags (SHF_COMPRESSED). Is it possible that oprofile is
somehow stripping this bit out, before calling the BFD routines ?

Cheers
Nick

[Marco.Leise at gmx dot de]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #5 from Marco Leise <Marco.Leise at gmx dot de> ---
I use this for callgraphs:

operf --vmlinux /usr/src/linux/vmlinux --callgraph ./app_to_be_profiled
opreport -c%g -t=0.1% -p=/lib/modules/`uname -r`/kernel/,/lib/modules/`uname
-r`/video/

And this for detailed samples:

operf --vmlinux /usr/src/linux/vmlinux ./app_to_be_profiled
opreport -gd%rl -t=1% -p=/lib/modules/`uname -r`/kernel/,/lib/modules/`uname
-r`/video/

The app needs no instrumentation other than - of course - debug information and
should dynamically link with libc. (OProfile also displays time spent in the
kernel, so I added the path to the uncompressed vmlinux and modules.) Hope that
helps!

[nickc at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #6 from Nick Clifton <nickc at redhat dot com> ---
(In reply to Marco Leise from comment #5)
Hi Marco,

Hope that helps!

Sorry - I have not been able to get anywhere with this. :-(

Are you able to test the newer release of the binutils (2.30) to see if the
problem exists there too ?

My gut feeling is that this is not a binutils bug at all, but that something
is producing corrupt dwarf binaries instead. Of course I cannot prove this,
which is why I would like to investigate. But so far I cannot reproduce the
problem. :-(

Cheers
Nick

[Marco.Leise at gmx dot de]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #7 from Marco Leise <Marco.Leise at gmx dot de> ---
I am actually using a git checkout past 2.30 right now and the issue is still
present. What is the command line you use for objdump? I.e. is there a way to
make it print source lines for symbols as oprofile would?

[nickc at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18616

--- Comment #8 from Nick Clifton <nickc at redhat dot com> ---
(In reply to Marco Leise from comment #7)

[Sorry for the long delay in responding...]

Post by Marco.Leise at gmx dot de
I am actually using a git checkout past 2.30 right now and the issue is
still present.

Darn.

Post by Marco.Leise at gmx dot de
What is the command line you use for objdump?

objdump -W <file>

Post by Marco.Leise at gmx dot de
I.e. is there a
way to make it print source lines for symbols as oprofile would?

objdump -WL <file>

I doubt if this is the same as oprofile will produce, but both tools
will be displaying processed DWARF line number information so they
should be roughly equivalent. You can also use "objdump -Wl <file>"
(with a lowercase-l rather than an uppercase-L) to see the raw DWARF
debug line information.

Cheers
Nick