[gem5-dev] x86 kvm status?

Discussion:

Steve Reinhardt

2013-08-29 21:49:31 UTC

What is the status of using kvm for x86 simulation? I know Andreas S. has
committed a bunch of patches, but I see that x86 is still not listed
in the is_isa_kvm_compatible()
function in the SConstruct. Does that mean there are still known issues?

I think this is a question primarily for Andreas, but I figured I'd ask in
public in case others are interested.

Thanks,

Steve

Andreas Sandberg

2013-09-03 17:32:34 UTC

Permalink

Post by Steve Reinhardt
What is the status of using kvm for x86 simulation? I know Andreas S. has
committed a bunch of patches, but I see that x86 is still not listed
in the is_isa_kvm_compatible()
function in the SConstruct. Does that mean there are still known issues?

Currently, the main issue that has kept me from submitting the patches
is the x87 interface. gem5 uses 64-bit doubles internally to represent
floating point numbers, while the hardware uses 80-bits. The problem
boils down to converting between the two. My current version is using a
gcc-specific 80-bit float type, which is something I need to fix before
submitting. I have the code[1] to do the conversion in software, but I
haven't had time to integrate it into gem5 yet.

Apart from the FP issues, things should work fine. I've seen less than
10% slowdown compared to native when executing SPEC and features like
CPU switching and checkpointing work.

Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.

//Andreas

[1] https://github.com/andysan/libfputils

nathan binkert

2013-09-03 17:47:43 UTC

Permalink

I'm curious, does this conversion mean that you lose precision? I'm
not sure I care, but it would be good to be sure that people know that
we don't do x87 correctly if so.

Did anyone ever consider using softfloat?
http://www.jhauser.us/arithmetic/SoftFloat.html

Nate

On Tue, Sep 3, 2013 at 10:32 AM, Andreas Sandberg

Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent floating
point numbers, while the hardware uses 80-bits. The problem boils down to
converting between the two. My current version is using a gcc-specific
80-bit float type, which is something I need to fix before submitting. I
have the code[1] to do the conversion in software, but I haven't had time to
integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than 10%
slowdown compared to native when executing SPEC and features like CPU
switching and checkpointing work.
Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.
//Andreas
[1] https://github.com/andysan/libfputils
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

Steve Reinhardt

2013-09-03 20:18:12 UTC

Permalink

Thanks for the reply, Andreas. I'm glad to hear things are mostly working.

Like Nate, I'm a little confused about the x87/FP situation. What are the
issues there? It seems like there are several distinct but related things
that (could be) involved:

1. How much do we care about 80-bit precision for x87? I can see where
being able to get the same precise answer whether we're running in KVM or
not would be very desirable, but unless things actually break without it,
I'd stop short of saying it's an absolute requirement (as long as we
advertise that limitation, of course).

2. How much do we use x87 anyway? 64-bit apps can assume SSE, so x87 isn't
necessary for FP, and my understanding was that we didn't support much of
x87 in our x86 ISA implementation in the first place.

3. How much do we care about portability, both across compilers and across
architectures? Obviously KVM support depends on running on an x86 machine.
If we have that same restriction for full 80-bit x87 support (i.e., only
on x86 hosts), I don't see a big problem with that. I doubt x86 gem5 ever
gets run on any other host. Assuming that we're running on an x86 machine,
it makes sense to use the built-in x87 hardware to get 80-bit support and
not emulate it in software. If gcc extensions are the only way to do that,
then I'm willing to make that compromise as well. If (for example) clang
doesn't provide access to 80-bit FP on x86 hosts, I'm willing to write that
up as a limitation of clang and not something we need to solve. Of course,
that's easy for me, since I don't use clang...

Thoughts?

Steve

Post by nathan binkert
I'm curious, does this conversion mean that you lose precision? I'm
not sure I care, but it would be good to be sure that people know that
we don't do x87 correctly if so.
Did anyone ever consider using softfloat?
http://www.jhauser.us/arithmetic/SoftFloat.html
Nate
On Tue, Sep 3, 2013 at 10:32 AM, Andreas Sandberg

Post by Steve Reinhardt
What is the status of using kvm for x86 simulation? I know Andreas S.

has

Post by Steve Reinhardt
committed a bunch of patches, but I see that x86 is still not listed
in the is_isa_kvm_compatible()
function in the SConstruct. Does that mean there are still known

issues?

Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent

floating

point numbers, while the hardware uses 80-bits. The problem boils down to
converting between the two. My current version is using a gcc-specific
80-bit float type, which is something I need to fix before submitting. I
have the code[1] to do the conversion in software, but I haven't had

time to

integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than

10%

slowdown compared to native when executing SPEC and features like CPU
switching and checkpointing work.
Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.
//Andreas
[1] https://github.com/andysan/libfputils
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

Andreas Sandberg

2013-09-03 21:35:06 UTC

Permalink

1. We probably don't need 80-bit precision when doing x87-arithmetics,
but we need it for other reasons: 1) to transfer state to and from KVM.
2) the x87 is expected to support loading and storing of 80-bit floats.
I've seen programs using this feature to convert between 80-bit floats
and 64-bit floats.

2. The problem here is that the standard libraries assume that the x87
is present, libc contains fair amount of hand-coded x87 assembly. I had
a hard time getting things to work because of this. I'm pretty sure that
I've seen GCC to generating x87 code for operations like modulo even
though it is configured to generate SSE code, but I can't remember the
details at the moment.

3. I have actually been running gem5 on ARM, but I must admit that it
seems unlikely that someone is would simulate x86 on ARM. I don't really
like the idea of losing portability, besides we should continue to
support clang since a lot of people are using it. I would be OK using
gcc extensions in the KVM module, but we need to support 80-bit loads &
stores in the ISA as well. We might want to do FP conversion in software
anyway to correctly handle overflows etc. Anyway, it wouldn't be hard
supporting FP conversion in software, there is both the library that
Nathan pointed to and my code, so it's just the small matter of doing it.

//Andreas

Post by Steve Reinhardt
Thanks for the reply, Andreas. I'm glad to hear things are mostly working.
Like Nate, I'm a little confused about the x87/FP situation. What are the
issues there? It seems like there are several distinct but related things
1. How much do we care about 80-bit precision for x87? I can see where
being able to get the same precise answer whether we're running in KVM or
not would be very desirable, but unless things actually break without it,
I'd stop short of saying it's an absolute requirement (as long as we
advertise that limitation, of course).
2. How much do we use x87 anyway? 64-bit apps can assume SSE, so x87 isn't
necessary for FP, and my understanding was that we didn't support much of
x87 in our x86 ISA implementation in the first place.
3. How much do we care about portability, both across compilers and across
architectures? Obviously KVM support depends on running on an x86 machine.
If we have that same restriction for full 80-bit x87 support (i.e., only
on x86 hosts), I don't see a big problem with that. I doubt x86 gem5 ever
gets run on any other host. Assuming that we're running on an x86 machine,
it makes sense to use the built-in x87 hardware to get 80-bit support and
not emulate it in software. If gcc extensions are the only way to do that,
then I'm willing to make that compromise as well. If (for example) clang
doesn't provide access to 80-bit FP on x86 hosts, I'm willing to write that
up as a limitation of clang and not something we need to solve. Of course,
that's easy for me, since I don't use clang...
Thoughts?
Steve

Post by Steve Reinhardt
What is the status of using kvm for x86 simulation? I know Andreas S.

has

Post by Steve Reinhardt
committed a bunch of patches, but I see that x86 is still not listed
in the is_isa_kvm_compatible()
function in the SConstruct. Does that mean there are still known

issues?

Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent

floating

time to

integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than

10%

_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

Steve Reinhardt

2013-09-03 22:32:45 UTC

Permalink

Thanks for the responses. It makes sense that, at the very least, we'll
need some portable conversion facilities between the 64-bit and 80-bit
representations.

I'm not in favor of losing portability either, but I'm relatively
pragmatic, so if a loss of portability saves orders of magnitude of effort
or performance, or worse yet is the difference between something happening
and not happening, then I'm willing to live with it.

Steve

1. We probably don't need 80-bit precision when doing x87-arithmetics, but
we need it for other reasons: 1) to transfer state to and from KVM. 2) the
x87 is expected to support loading and storing of 80-bit floats. I've seen
programs using this feature to convert between 80-bit floats and 64-bit
floats.
2. The problem here is that the standard libraries assume that the x87 is
present, libc contains fair amount of hand-coded x87 assembly. I had a hard
time getting things to work because of this. I'm pretty sure that I've seen
GCC to generating x87 code for operations like modulo even though it is
configured to generate SSE code, but I can't remember the details at the
moment.
3. I have actually been running gem5 on ARM, but I must admit that it
seems unlikely that someone is would simulate x86 on ARM. I don't really
like the idea of losing portability, besides we should continue to support
clang since a lot of people are using it. I would be OK using gcc
extensions in the KVM module, but we need to support 80-bit loads & stores
in the ISA as well. We might want to do FP conversion in software anyway to
correctly handle overflows etc. Anyway, it wouldn't be hard supporting FP
conversion in software, there is both the library that Nathan pointed to
and my code, so it's just the small matter of doing it.
//Andreas

Post by nathan binkert
not sure I care, but it would be good to be sure that people know that
we don't do x87 correctly if so.
Did anyone ever consider using softfloat?
http://www.jhauser.us/**arithmetic/SoftFloat.html<http://www.jhauser.us/arithmetic/SoftFloat.html>
Nate
On Tue, Sep 3, 2013 at 10:32 AM, Andreas Sandberg

Post by Steve Reinhardt

Post by Steve Reinhardt
What is the status of using kvm for x86 simulation? I know Andreas S.

has
committed a bunch of patches, but I see that x86 is still not listed

Post by Steve Reinhardt
in the is_isa_kvm_compatible()
function in the SConstruct. Does that mean there are still known

issues?
Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent

floating

Post by Steve Reinhardt
point numbers, while the hardware uses 80-bits. The problem boils down to
converting between the two. My current version is using a gcc-specific
80-bit float type, which is something I need to fix before submitting. I
have the code[1] to do the conversion in software, but I haven't had

time to

Post by Steve Reinhardt
integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than

10%

Post by Steve Reinhardt
slowdown compared to native when executing SPEC and features like CPU
switching and checkpointing work.
Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.
//Andreas
[1] https://github.com/andysan/**libfputils<https://github.com/andysan/libfputils>
______________________________**_________________
gem5-dev mailing list
http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>

______________________________**_________________
gem5-dev mailing list
http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>
______________________________**_________________

gem5-dev mailing list
http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>

______________________________**_________________
gem5-dev mailing list
http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>

Andreas Sandberg

2013-09-03 21:19:40 UTC

Permalink

Yes, we loose precision since gem5 only supports 64-bit floats. However,
I don't think that's a huge issue. It's a small issue compared to the
way x87 loads and stores are implemented (IIRC, the instructions use the
wrong float type when accessing memory). I have a bug-fix on my
fixes-x87 branch on github, but it requires 80-bit float support.

The state of the x87 is currently shaky to say the least. There is some
support for instructions that appear in the standard libraries. I
recently needed to a handful of new instructions (some are probably
still in my fixes branch) to get a few more SPEC benchmarks to run. One
major class of missing features is exceptions. Flag handling is probably
a bit shaky as well. At least in my branch, stack management (the tag
word is updated correctly), loads and stores (32-bit, 64-bit, and
80-bit), and most add and subtract instructions are working. However,
there is still a large number of unimplemented instructions.

I was planning to add more features to it, but I got distracted thinking
about efficient ways to implement test cases (it's hard to get FP stuff
right!) instead. I had some ideas about using the KVM CPU to verify test
cases running on the simulated CPU, but it's all a bit half-baked at the
moment. I did that manually for the instructions I added, but I'd like
to streamline the process a bit.

Thanks for the tip about SoftFloat, I'll have a look once I get out of
the deadline-panic mode.

//Andreas

Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent floating
point numbers, while the hardware uses 80-bits. The problem boils down to
converting between the two. My current version is using a gcc-specific
80-bit float type, which is something I need to fix before submitting. I
have the code[1] to do the conversion in software, but I haven't had time to
integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than 10%
slowdown compared to native when executing SPEC and features like CPU
switching and checkpointing work.
Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.
//Andreas
[1] https://github.com/andysan/libfputils
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

Steve Reinhardt

2013-09-03 22:39:22 UTC

Permalink

Post by Andreas Sandberg
I was planning to add more features to it, but I got distracted thinking
about efficient ways to implement test cases (it's hard to get FP stuff
right!) instead. I had some ideas about using the KVM CPU to verify test
cases running on the simulated CPU, but it's all a bit half-baked at the
moment. I did that manually for the instructions I added, but I'd like to
streamline the process a bit.

On this front... didn't Gabe have some kind of tracing facility put
together where he could diff instruction effects between gem5 and a real
machine on an instruction-by-instruction basis? I don't recall the details
though. It seems like that could be really handy here.

Steve

nathan binkert

2013-09-03 22:51:35 UTC

Permalink

Yes. I believe that it is the statetrace stuff. It uses ptrace to
single step through a real process on a host machine. Not sure how
ptrace interacts with KVM. Should mostly work I'd imagine.

Nate

Post by Steve Reinhardt

Andreas Sandberg

2013-09-04 07:47:44 UTC

Permalink

Post by nathan binkert

Post by Steve Reinhardt

Yes. I believe that it is the statetrace stuff. It uses ptrace to
single step through a real process on a host machine. Not sure how
ptrace interacts with KVM. Should mostly work I'd imagine.

That looks like a very useful tool. I'll give it a spin an see if it can
be used to automate my test cases.

//Andreas

Radivoje Vasiljevic

2013-09-04 14:13:05 UTC

Permalink

What about multi word arithmetic with additional exponent field? Three
doubles are more then enough to represent 80b and to have enough precision
to enable exact rounding to 64b mantissa in case of division and square
root, and to take care of flags which are probably trickiest part to get
absolutely right or at least almost always right. There are three cases for
emulation:
1) native x87, no emulation needed
2) doubles, with true fused multiply add instruction - a bit slower but not
that much, code is somewhat subtle, some would say dirty
3) doubles, no fused madd (or it has two roundings) - few times slower then
2nd case and somewhat messier
2nd and 3rd case can be done with double word or triple word, 2nd case with
double word is especially fast with all benefits of inlining. Additional
exponent would not be used that much, except in rare cases so total impact
on speed should not be large.
libm instructions should not be problem for two reasons: there are not same
on Intel and AMD CPUs (at least, this was case about 10 year ago, but
results were consistent on several generations) and both have error of few
ulps so call to libm is actually more accurate, and they are not used much
in these days as call to libm is both faster and more accurate.

----- Original Message -----
From: "nathan binkert" <***@binkert.org>
To: "gem5 Developer List" <gem5-***@gem5.org>
Sent: 3. септембар 2013 19:47
Subject: Re: [gem5-dev] x86 kvm status?

Currently, the main issue that has kept me from submitting the patches is
the x87 interface. gem5 uses 64-bit doubles internally to represent floating
point numbers, while the hardware uses 80-bits. The problem boils down to
converting between the two. My current version is using a gcc-specific
80-bit float type, which is something I need to fix before submitting. I
have the code[1] to do the conversion in software, but I haven't had time to
integrate it into gem5 yet.
Apart from the FP issues, things should work fine. I've seen less than 10%
slowdown compared to native when executing SPEC and features like CPU
switching and checkpointing work.
Unfortunately, I don't expect to get time to prepare the code for review
until mid/late September.
//Andreas
[1] https://github.com/andysan/libfputils
_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev

_______________________________________________
gem5-dev mailing list
http://m5sim.org/mailman/listinfo/gem5-dev