Discussion:
[Valgrind-developers] Why valgrind could not be used on cavium octeon3?
网络尖兵
2017-03-29 11:43:21 UTC
Permalink
Hello, everyone!
I have posted this question on the forum, but I got this message"This post has NOT been accepted by the mailing list yet." I am not sure if my post could be seen by somebody else. So I send this mail.


I am trying to use valgrind to do the memcheck on Cavium Octeon CN73XX, but unfortunately I got the following message.
root# valgrind
--3846:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (13 segments)
--3846:0: aspacem 1 segment names in 1 slots
--3846:0: aspacem freelist is empty
--3846:0: aspacem (0,4,2) /usr/lib/valgrind/memcheck-mips64-linux
--3846:0: aspacem 0: RSVN 0000000000-0003ffffff 64m ----- SmFixed
--3846:0: aspacem 1: 0004000000-0037ffffff 832m
--3846:0: aspacem 2: FILE 0038000000-003823ffff 2359296 r-x-- d=0xb301 i=12614 o=0 (0,4)
--3846:0: aspacem 3: FILE 0038240000-003826ffff 196608 rw--- d=0xb301 i=12614 o=2293760 (0,4)
--3846:0: aspacem 4: ANON 0038270000-003943ffff 17m rwx--
--3846:0: aspacem 5: 0039440000-0801ffffff 31883m
--3846:0: aspacem 6: RSVN 0802000000-0802000fff 4096 ----- SmFixed
--3846:0: aspacem 7: 0802001000-0fffffffff 32735m
--3846:0: aspacem 8: RSVN 1000000000-ffdfcbffff 982524m ----- SmFixed
--3846:0: aspacem 9: ANON ffdfcc0000-ffdfceffff 196608 rwx--
--3846:0: aspacem 10: RSVN ffdfcf0000-fffffeffff 515m ----- SmFixed
--3846:0: aspacem 11: ANON ffffff0000-ffffffffff 65536 r-x--
--3846:0: aspacem 12: RSVN 10000000000-ffffffffffffffff 16383e ----- SmFixed
--3846:0: aspacem >>>
--3846-- core : 0/ 0 max/curr mmap'd, 0/0 unsplit/split sb unmmap'd, 0/ 0 max/curr, 0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--3846-- dinfo : 0/ 0 max/curr mmap'd, 0/0 unsplit/split sb unmmap'd, 0/ 0 max/curr, 0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--3846-- (null) : 0/ 0 max/curr mmap'd, 0/0 unsplit/split sb unmmap'd, 0/ 0 max/curr, 0/ 0 totalloc-blocks/bytes, 0 searches 0 rzB
--3846-- demangle: 0/ 0 max/curr mmap'd, 0/0 unsplit/split sb unmmap'd, 0/ 0 max/curr, 0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--3846-- ttaux : 0/ 0 max/curr mmap'd, 0/0 unsplit/split sb unmmap'd, 0/ 0 max/curr, 0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--3846-- translate: no SP updates identified
--3846-- translate: PX: SPonly 0, UnwRegs 0, AllRegs 0, AllRegsAllInsns 0
--3846-- tt/tc: 0 tt lookups requiring 0 probes
--3846-- tt/tc: 0 fast-cache updates, 0 flushes
--3846-- transtab: new 0 (0 -> 0; ratio 0.0) [0 scs] avg tce size 0
--3846-- transtab: dumped 0 (0 -> ??) (sectors recycled 0)
--3846-- transtab: discarded 0 (0 -> ??)
--3846-- scheduler: 0 event checks.
--3846-- scheduler: 0 indir transfers, 0 misses (1 in 0)
--3846-- scheduler: 0/0 major/minor sched events.
--3846-- sanity: 0 cheap, 0 expensive checks.
==3846==
==3846== Valgrind's memory management: out of memory:
==3846== newSuperblock's request for 4194304 bytes failed.
==3846== 18,939,904 bytes have already been mmap-ed ANONYMOUS.
==3846== Valgrind cannot continue. Sorry.
==3846==
==3846== There are several possible reasons for this.
==3846== - You have some kind of memory limit in place. Look at the
==3846== output of 'ulimit -a'. Is there a limit on the size of
==3846== virtual memory or address space?
==3846== - You have run out of swap space.
==3846== - Valgrind has a bug. If you think this is the case or you are
==3846== not sure, please let us know and we'll try to fix it.
==3846== Please note that programs can take substantially more memory than
==3846== normal when running under Valgrind tools, eg. up to twice or
==3846== more, depending on the tool. On a 64-bit machine, Valgrind
==3846== should be able to make use of up 32GB memory. On a 32-bit
==3846== machine, Valgrind should be able to use all the memory available
==3846== to a single process, up to 4GB if that's how you have your
==3846== kernel configured. Most 32-bit Linux setups allow a maximum of
==3846== 3GB per process.
==3846==
==3846== Whatever the reason, Valgrind cannot continue. Sorry.

I have searched and found somebody have meet this kind of problem, but I still have no idea on how to deal with it.
http://valgrind.10908.n7.nabble.com/Valgrind-13854-Cross-compiling-for-Cavium-MIPS64-N32-ABI-td48980.html
http://valgrind.10908.n7.nabble.com/valgrind-out-of-memory-error-td54401.html

The valgrind was configured and build by yocto with "-meb -mabi=64 -mhard-float -march=octeon3" .
The valgrind version is 3.11.0.
I have tried to expand swap on my box, but it make no sense.
John Reiser
2017-03-29 12:29:41 UTC
Permalink
Post by 网络尖兵
I am trying to use valgrind to do the memcheck on Cavium Octeon CN73XX, but unfortunately I got the following message.
root# valgrind
Do not run as root (superuser). Create an ordinary userID, and run as that user.
On some systems a new userID can be created by using /usr/sbin/adduser.
Then login as that userID, or login as root and "su userID" before running valgrind.
Post by 网络尖兵
--3846:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (13 segments)
Please run
strace -e trace=memory valgrind ...
to show the actual system calls to mmap, mprotect, munmap, brk, etc.
John Reiser
2017-03-29 14:00:29 UTC
Permalink
Post by John Reiser
Please run
strace -e trace=memory valgrind ...
to show the actual system calls to mmap, mprotect, munmap, brk, etc.
On chips designed for embedded systems, there is a temptation to save
area, power, and/or a few picoseconds by reducing the generality of
memory management hardware. Such as: the PAGESIZE may be 4KiB (0x1000),
but any mapping must start on a 16KiB boundary (0x4000); this happens
on quite a few 32-bit ARM chips.

If valgrind does not know (or has a bug) then it is easy to request
an impossible mapping, and the kernel's denial is easy to interpret as
"out of memory". Look in the valgrind source:
grep -sr SHMLBA coregrind
particularly mips_PRE_sys_mmap() to see some attempts to deal with this.
Also see PRE(sys_shmat) in coregrind/m_syswrap/syswrap-linux.c .
zboom
2017-03-30 10:23:55 UTC
Permalink
Here is the system message about memory:
$ free -m
total used free shared buff/cache
available
Mem: 8070 527 6535 254 1007
6392
Swap: 414 0 414
$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 8070
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 8070
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

$ uname -a
Linux 3.10.87-rt80-yocto-standard #2 SMP Wed Mar 22 17:17:38 CST 2017 mips64
mips64 mips64 GNU/Linux

$ cat /proc/meminfo
MemTotal: 8264384 kB
MemFree: 6886656 kB
Buffers: 42560 kB
Cached: 935424 kB
SwapCached: 0 kB
Active: 492672 kB
Inactive: 687744 kB
Active(anon): 345088 kB
Inactive(anon): 118144 kB
Active(file): 147584 kB
Inactive(file): 569600 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 424704 kB
SwapFree: 424704 kB
Dirty: 768 kB
Writeback: 0 kB
AnonPages: 202496 kB
Mapped: 51968 kB
Shmem: 260864 kB
Slab: 59712 kB
SReclaimable: 25984 kB
SUnreclaim: 33728 kB
KernelStack: 8704 kB
PageTables: 9792 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8275840 kB
Committed_AS: 712512 kB
VmallocTotal: 4290772864 kB
VmallocUsed: 15232 kB
VmallocChunk: 4290754624 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB

As there is no getconf cmd on my board, I wrote a program called
'getpagesize'. And the pagesize it returned is 65536B.



--
View this message in context: http://valgrind.10908.n7.nabble.com/Why-valgrind-could-not-be-used-on-cavium-octeon3-tp57531p57547.html
Sent from the Valgrind - Dev mailing list archive at Nabble.com.
zboom
2017-03-30 09:01:55 UTC
Permalink
Here is the message printed by the non superuser:
***@xxx:~# adduser test
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
***@xxx:~# su test
***@xxx:~$ strace -e trace=memory valgrind
brk(NULL) = 0x12b160000
mmap(NULL, 9594, PROT_READ, MAP_PRIVATE, 3, 0) = 0xffe80d0000
mmap(NULL, 1790224, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0xffe7f10000
mmap(0xffe80a0000, 196608, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x180000) = 0xffe80a0000
mprotect(0xffe80a0000, 65536, PROT_READ) = 0
munmap(0xffe80d0000, 9594) = 0
brk(NULL) = 0x12b160000
brk(0x12b190000) = 0x12b190000
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
--4198:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (13 segments)
--4198:0: aspacem 1 segment names in 1 slots
--4198:0: aspacem freelist is empty
--4198:0: aspacem (0,4,2) /usr/lib/valgrind/memcheck-mips64-linux
--4198:0: aspacem 0: RSVN 0000000000-0003ffffff 64m ----- SmFixed
--4198:0: aspacem 1: 0004000000-0037ffffff 832m
--4198:0: aspacem 2: FILE 0038000000-003823ffff 2359296 r-x-- d=0xb301
i=12614 o=0 (0,4)
--4198:0: aspacem 3: FILE 0038240000-003826ffff 196608 rw--- d=0xb301
i=12614 o=2293760 (0,4)
--4198:0: aspacem 4: ANON 0038270000-003943ffff 17m rwx--
--4198:0: aspacem 5: 0039440000-0801ffffff 31883m
--4198:0: aspacem 6: RSVN 0802000000-0802000fff 4096 ----- SmFixed
--4198:0: aspacem 7: 0802001000-0fffffffff 32735m
--4198:0: aspacem 8: RSVN 1000000000-ffdf95ffff 982521m ----- SmFixed
--4198:0: aspacem 9: ANON ffdf960000-ffdf98ffff 196608 rwx--
--4198:0: aspacem 10: RSVN ffdf990000-fffffeffff 518m ----- SmFixed
--4198:0: aspacem 11: ANON ffffff0000-ffffffffff 65536 r-x--
--4198:0: aspacem 12: RSVN 10000000000-ffffffffffffffff 16383e -----
SmFixed
--4198:0: aspacem >>>
--4198-- core : 0/ 0 max/curr mmap'd, 0/0
unsplit/split sb unmmap'd, 0/ 0 max/curr,
0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--4198-- dinfo : 0/ 0 max/curr mmap'd, 0/0
unsplit/split sb unmmap'd, 0/ 0 max/curr,
0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--4198-- (null) : 0/ 0 max/curr mmap'd, 0/0
unsplit/split sb unmmap'd, 0/ 0 max/curr,
0/ 0 totalloc-blocks/bytes, 0 searches 0 rzB
--4198-- demangle: 0/ 0 max/curr mmap'd, 0/0
unsplit/split sb unmmap'd, 0/ 0 max/curr,
0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--4198-- ttaux : 0/ 0 max/curr mmap'd, 0/0
unsplit/split sb unmmap'd, 0/ 0 max/curr,
0/ 0 totalloc-blocks/bytes, 0 searches 8 rzB
--4198-- translate: no SP updates identified
--4198-- translate: PX: SPonly 0, UnwRegs 0, AllRegs 0, AllRegsAllInsns 0
--4198-- tt/tc: 0 tt lookups requiring 0 probes
--4198-- tt/tc: 0 fast-cache updates, 0 flushes
--4198-- transtab: new 0 (0 -> 0; ratio 0.0) [0 scs] avg tce size 0
--4198-- transtab: dumped 0 (0 -> ??) (sectors recycled 0)
--4198-- transtab: discarded 0 (0 -> ??)
--4198-- scheduler: 0 event checks.
--4198-- scheduler: 0 indir transfers, 0 misses (1 in 0)
--4198-- scheduler: 0/0 major/minor sched events.
--4198-- sanity: 0 cheap, 0 expensive checks.
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
==4198==
==4198== Valgrind's memory management: out of memory:
==4198== newSuperblock's request for 4194304 bytes failed.
==4198== 18,939,904 bytes have already been mmap-ed ANONYMOUS.
==4198== Valgrind cannot continue. Sorry.
==4198==
==4198== There are several possible reasons for this.
==4198== - You have some kind of memory limit in place. Look at the
==4198== output of 'ulimit -a'. Is there a limit on the size of
==4198== virtual memory or address space?
==4198== - You have run out of swap space.
==4198== - Valgrind has a bug. If you think this is the case or you are
==4198== not sure, please let us know and we'll try to fix it.
==4198== Please note that programs can take substantially more memory
than
==4198== normal when running under Valgrind tools, eg. up to twice or
==4198== more, depending on the tool. On a 64-bit machine, Valgrind
==4198== should be able to make use of up 32GB memory. On a 32-bit
==4198== machine, Valgrind should be able to use all the memory
available
==4198== to a single process, up to 4GB if that's how you have your
==4198== kernel configured. Most 32-bit Linux setups allow a maximum of
==4198== 3GB per process.
==4198==
==4198== Whatever the reason, Valgrind cannot continue. Sorry.
+++ exited with 1 +++




--
View this message in context: http://valgrind.10908.n7.nabble.com/Why-valgrind-could-not-be-used-on-cavium-octeon3-tp57531p57546.html
Sent from the Valgrind - Dev mailing list archive at Nabble.com.
John Reiser
2017-03-30 11:22:09 UTC
Permalink
Post by zboom
As there is no getconf cmd on my board, I wrote a program called
'getpagesize'. And the pagesize it returned is 65536B.
[[those lines above are from another message]]
[[snip]]
Post by zboom
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
[[snip]]
Post by zboom
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
So the valgrind bug is asking for MAP_FIXED with an address that is not
on a page boundary (PAGE_SIZE [also spelled PAGESIZE] being 65536==0x10000).
(Notice that all the successful calls to mmap(), which have been snipped above,
all specify an address that is a multiple of 64KiB.)

Please file a bug report at bugs.kde.org with component 'valgrind'.
In the bug report please include the hardware info (output from
/usr/bin/lscpu or "cat /proc/cpuinfo") and software info ("uname -a")
and the info that the pagesize is 64KiB.

Then run valgrind under gdb, use a breakpoint to find the call to
mmap(0x....1000,,,MAP_FIXED,,), look at the backtrace,
and try to figure out why valgrind thought it could specify
an address that was not a multiple of the pagesize.
Aleksandar Rikalo
2017-03-30 12:26:48 UTC
Permalink
I'm not able to reproduce this issue on Octeon3 (or any other board).
It looks like https://bugs.kde.org/show_bug.cgi?id=342356 issue which is
fixed in r15813.

Can you confirm that you are running the latest SVN version of Valgrind ?
Post by John Reiser
Post by zboom
As there is no getconf cmd on my board, I wrote a program called
'getpagesize'. And the pagesize it returned is 65536B.
[[those lines above are from another message]]
[[snip]]
Post by zboom
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
[[snip]]
Post by zboom
mmap(0x802001000, 4194304, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EINVAL (Invalid argument)
So the valgrind bug is asking for MAP_FIXED with an address that is not
on a page boundary (PAGE_SIZE [also spelled PAGESIZE] being 65536==0x10000).
(Notice that all the successful calls to mmap(), which have been snipped above,
all specify an address that is a multiple of 64KiB.)
Please file a bug report at bugs.kde.org with component 'valgrind'.
In the bug report please include the hardware info (output from
/usr/bin/lscpu or "cat /proc/cpuinfo") and software info ("uname -a")
and the info that the pagesize is 64KiB.
Then run valgrind under gdb, use a breakpoint to find the call to
mmap(0x....1000,,,MAP_FIXED,,), look at the backtrace,
and try to figure out why valgrind thought it could specify
an address that was not a multiple of the pagesize.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Valgrind-developers mailing list
https://lists.sourceforge.net/lists/listinfo/valgrind-developers
zboom
2017-03-31 11:46:23 UTC
Permalink
This post might be inappropriate. Click to display it.
Petar Jovanovic
2017-03-31 13:17:26 UTC
Permalink
Post by zboom
I was using the release version 3.11.0, and I compared the code of bug
342356. My local version was actually not including this fix. Then I updated
my local version to the release version 3.12.0 and also the latest svn
version. The valgrind can show its version and help message now. But it is
pending on the terminal when I am doing the memcheck. It will never return
until I kill it forcibly.
Can you confirm that you are using the latest SVN code?
What does 'svn info' say?

Petar
zboom
2017-04-01 07:03:00 UTC
Permalink
The valgrind can be running on my cavium octeon3 board now!
Actually, my yocto build environment was caching the 3.12.0 building file
yesterday.
Today I use the latest SVN code, and delete the cache. Finally it works!
Thanks a lot!



--
View this message in context: http://valgrind.10908.n7.nabble.com/Why-valgrind-could-not-be-used-on-cavium-octeon3-tp57531p57555.html
Sent from the Valgrind - Dev mailing list archive at Nabble.com.

Aleksandar Rikalo
2017-03-29 15:16:03 UTC
Permalink
Hello,

Have you tried running the latest SVN version ?
There are recently fixed issues regarding Cavium Octeon:

https://bugs.kde.org/show_bug.cgi?id=376142
https://bugs.kde.org/show_bug.cgi?id=344524


Best Regards,
Aleksandar
Post by 网络尖兵
I am trying to use valgrind to do the memcheck on Cavium Octeon
CN73XX, but unfortunately I got the following message.
root# valgrind
--3846:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (13 segments)
--3846:0: aspacem 1 segment names in 1 slots
...
The valgrind was configured and build by yocto with "-meb -mabi=64
-mhard-float -march=octeon3" .
The valgrind version is 3.11.0.
zboom
2017-03-30 10:49:26 UTC
Permalink
I have got the svn trunk code, but I could not find the Macro defined in the
patch, Why ?

$ svn info
Path: .
Working Copy Root Path: /home/zboom/Downloads/valgrind
URL: svn://svn.valgrind.org/valgrind/trunk
Relative URL: ^/trunk
Repository Root: svn://svn.valgrind.org/valgrind
Repository UUID: a5019735-40e9-0310-863c-91ae7b9d1cf9
Revision: 16289
Node Kind: directory
Schedule: normal
Last Changed Author: iraisr
Last Changed Rev: 16287
Last Changed Date: 2017-03-27 13:06:32 +0800 (一, 27 3月 2017)

***@zboom-Lenovo:~/Downloads/valgrind$ grep -rn "_MIPS_ARCH_OCTEON3" .




--
View this message in context: http://valgrind.10908.n7.nabble.com/Why-valgrind-could-not-be-used-on-cavium-octeon3-tp57531p57548.html
Sent from the Valgrind - Dev mailing list archive at Nabble.com.
Loading...