Modify MOESI_hammer protocol

Discussion:

gem5 gem5

2012-07-25 21:29:34 UTC

Hi all,

I want to modify MOESI_hammer protocol to broadcast all the messages. Since
multicast is supported, I think it's possible to do this. I got some
problem with my first step. When I
modify out_msg.Destination.broadcast(MachineType:L1Cache); to
be out_msg.Destination.broadcast() which I assume means to broadcast to all
nodes, inside of action(fn_forwardRequestIfNecessary, "fn", desc="Forward
requests if necessary"), I got some error like this when run the
rubynetwork tester with Pt2Pt topology:

panic: Ordering property of has not been set
@ cycle 40
[enqueue:build/X86_SE/mem/ruby/buffers/MessageBuffer.cc, line 165]
Memory Usage: 241400 KBytes
Program aborted at cycle 40
Abort

I tried to set the corresponding virtual network forwardFromDir's
ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Jinzhu

Tushar Krishna

2012-07-26 16:53:01 UTC

Permalink

The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The ordering itself might be true or false depending on the vnet, but this function has to be called for the message buffer to be valid].

If all you have changed in the protocol is to broadcast to all L1's and directories, instead only only to L1's, you also need to account for what happens when these directories receive these messages. My guess is that these messages are being inserted into some message buffer that is not part of the protocol specifications and hence this error shows up.

- Tushar

Post by gem5 gem5
Hi all,
panic: Ordering property of has not been set
@ cycle 40
[enqueue:build/X86_SE/mem/ruby/buffers/MessageBuffer.cc, line 165]
Memory Usage: 241400 KBytes
Program aborted at cycle 40
Abort
I tried to set the corresponding virtual network forwardFromDir's ordered="true", but still I cannot get rid of this error. Any help will be appreciated. Thanks a lot!
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

gem5 gem5

2012-07-26 22:21:26 UTC

Permalink

Hi Tushar,

Thanks for your reply! I think you are right. I need to deal with those
extra messages. Any suggestions how to do that? Should I define some new
transitions and get rid of those messages?

I have read your 1 to many paper before. If I want to implement some
crossbar topology which supports broadcast like mXbar, do you think
modifying the protocol is the right way to do or there is some other better
method? Thank you very much for your help.

Best,

Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs to
be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's and
directories, instead only only to L1's, you also need to account for what
happens when these directories receive these messages. My guess is that
these messages are being inserted into some message buffer that is not part
of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the messages.

Since multicast is supported, I think it's possible to do this. I got some
problem with my first step. When I modify
out_msg.Destination.broadcast(MachineType:L1Cache); to be
out_msg.Destination.broadcast() which I assume means to broadcast to all
nodes, inside of action(fn_forwardRequestIfNecessary, "fn", desc="Forward
requests if necessary"), I got some error like this when run the

Post by gem5 gem5
panic: Ordering property of has not been set
@ cycle 40
[enqueue:build/X86_SE/mem/ruby/buffers/MessageBuffer.cc, line 165]
Memory Usage: 241400 KBytes
Program aborted at cycle 40
Abort
I tried to set the corresponding virtual network forwardFromDir's

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Tushar Krishna

2012-07-26 23:30:05 UTC

Permalink

Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add more messages in the network? MOESI_hammer inherently broadcasts to all L1s which gives you a broadcast to all nodes. I don't think you should mess with the protocol, unless you have a strong justification for why those extra messages are needed: example you come up with some new protocol that sends more broadcasts instead of tracking some state etc In which case, yes you would have to define extra transitions to handle the extra messages.

The two broadcast based protocols provided with gem5 are MOESI_hammer (where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's broadcast to all L2s

cheers,
Tushar

Post by gem5 gem5
Hi Tushar,
Thanks for your reply! I think you are right. I need to deal with those extra messages. Any suggestions how to do that? Should I define some new transitions and get rid of those messages?
I have read your 1 to many paper before. If I want to implement some crossbar topology which supports broadcast like mXbar, do you think modifying the protocol is the right way to do or there is some other better method? Thank you very much for your help.
Best,
Jinzhu
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The ordering itself might be true or false depending on the vnet, but this function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's and directories, instead only only to L1's, you also need to account for what happens when these directories receive these messages. My guess is that these messages are being inserted into some message buffer that is not part of the protocol specifications and hence this error shows up.
- Tushar

gem5 gem5

2012-07-27 00:33:09 UTC

Permalink

Tushar,

The reason I am modifying the protocol is that I want to implement a
shared-bus based crossbar where each node has its own dedicated write
channel and other nodes listen to it. However, I see that ruby does not
support hardware broadcast, so modifying the protocol seems to me the only
way to implement this. I think by modifying all the messages to be
broadcast messages can solve this. It would be really helpful if you could
provide me more information how to implement this kind of topology. I think
mXbar is very very close to what I wanna do. Thank you very much!

Best,

Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add more
messages in the network? MOESI_hammer inherently broadcasts to all L1s
which gives you a broadcast to all nodes. I don't think you should mess
with the protocol, unless you have a strong justification for why those
extra messages are needed: example you come up with some new protocol that
sends more broadcasts instead of tracking some state etc In which case,
yes you would have to define extra transitions to handle the extra messages.
The two broadcast based protocols provided with gem5 are MOESI_hammer
(where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's
broadcast to all L2s
cheers,
Tushar
Hi Tushar,
Thanks for your reply! I think you are right. I need to deal with those
extra messages. Any suggestions how to do that? Should I define some new
transitions and get rid of those messages?
I have read your 1 to many paper before. If I want to implement some
crossbar topology which supports broadcast like mXbar, do you think
modifying the protocol is the right way to do or there is some other better
method? Thank you very much for your help.
Best,
Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs
to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's and
directories, instead only only to L1's, you also need to account for what
happens when these directories receive these messages. My guess is that
these messages are being inserted into some message buffer that is not part
of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the messages.

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Tushar Krishna

2012-07-27 04:44:44 UTC

Permalink

Hi Jinzhu,
Ok so I see two components in your design:
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar

For (1) you want every message to be a broadcast. I am not sure if MOESI_hammer is the correct protocol to use as your starting point. The reason is that MOESI_hammer is essentially a directory protocol, with the directory having no state. So every cache miss first goes to its home node, and the home node then performs a broadcast to all other caches, and then the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol for a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to model the kind of protocol you want.

For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I had optimized this pipeline, and added an mXbar in the switch traversal stage. Also note that the released version of garnet does not have broadcast support, i.e. a broadcast is converted into individual messages at the source, one for each destination and sent out.
If you use the simple network, the switch models there can handle broadcasts. You could perhaps use that as a starting point and then edit the code to model the crossbar you want.

cheers,
Tushar

Post by gem5 gem5
Tushar,
The reason I am modifying the protocol is that I want to implement a shared-bus based crossbar where each node has its own dedicated write channel and other nodes listen to it. However, I see that ruby does not support hardware broadcast, so modifying the protocol seems to me the only way to implement this. I think by modifying all the messages to be broadcast messages can solve this. It would be really helpful if you could provide me more information how to implement this kind of topology. I think mXbar is very very close to what I wanna do. Thank you very much!
Best,
Jinzhu
Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add more messages in the network? MOESI_hammer inherently broadcasts to all L1s which gives you a broadcast to all nodes. I don't think you should mess with the protocol, unless you have a strong justification for why those extra messages are needed: example you come up with some new protocol that sends more broadcasts instead of tracking some state etc In which case, yes you would have to define extra transitions to handle the extra messages.
The two broadcast based protocols provided with gem5 are MOESI_hammer (where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's broadcast to all L2s
cheers,
Tushar

gem5 gem5

2012-07-27 19:20:18 UTC

Permalink

Hi Tushar,

Your suggestions are really helpful. Thank you very much! For now, I think
I will take Pt2Pt as the basic topology to work with since it has all the
connections. I think I can view it the same way as the single write
multiple read shared-bus crossbar. I just need to create a broadcast
protocol for it.

You are right, MOESI_hammer is not a good choice since it's essentially a
directory protocol. MOSI_SMP_bcast is a much better reference, although I
am not sure how to modify that since it is for GEMS' hierarchical switch
and it also has some directories inside. Hope I will figure it out.

Do you think this is the right direction to go? Thanks a lot again!

Best,

Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar
For (1) you want every message to be a broadcast. I am not sure if
MOESI_hammer is the correct protocol to use as your starting point. The
reason is that MOESI_hammer is essentially a directory protocol, with the
directory having no state. So every cache miss first goes to its home node,
and the home node then performs a broadcast to all other caches, and then
the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be
broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol for
a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up
something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to model
the kind of protocol you want.
For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I had
optimized this pipeline, and added an mXbar in the switch traversal stage.
Also note that the released version of garnet does not have broadcast
support, i.e. a broadcast is converted into individual messages at the
source, one for each destination and sent out.
If you use the simple network, the switch models there can handle
broadcasts. You could perhaps use that as a starting point and then edit
the code to model the crossbar you want.
cheers,
Tushar
Tushar,
The reason I am modifying the protocol is that I want to implement a
shared-bus based crossbar where each node has its own dedicated write
channel and other nodes listen to it. However, I see that ruby does not
support hardware broadcast, so modifying the protocol seems to me the only
way to implement this. I think by modifying all the messages to be
broadcast messages can solve this. It would be really helpful if you could
provide me more information how to implement this kind of topology. I think
mXbar is very very close to what I wanna do. Thank you very much!
Best,
Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs
to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's and
directories, instead only only to L1's, you also need to account for what
happens when these directories receive these messages. My guess is that
these messages are being inserted into some message buffer that is not part
of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the messages.

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

gem5 gem5

2012-07-27 19:49:05 UTC

Permalink

Post by gem5 gem5
Hi Tushar,
Your suggestions are really helpful. Thank you very much! For now, I think
I will take Pt2Pt as the basic topology to work with since it has all the
connections. I think I can view it the same way as the single write
multiple read shared-bus crossbar. I just need to create a broadcast
protocol for it.
You are right, MOESI_hammer is not a good choice since it's essentially a
directory protocol. MOSI_SMP_bcast is a much better reference, although I
am not sure how to modify that since it is for GEMS' hierarchical switch
and it also has some directories inside. Hope I will figure it out.
Do you think this is the right direction to go? Thanks a lot again!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar
For (1) you want every message to be a broadcast. I am not sure if
MOESI_hammer is the correct protocol to use as your starting point. The
reason is that MOESI_hammer is essentially a directory protocol, with the
directory having no state. So every cache miss first goes to its home node,
and the home node then performs a broadcast to all other caches, and then
the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be
broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol
for a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up
something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to model
the kind of protocol you want.
For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I had
optimized this pipeline, and added an mXbar in the switch traversal stage.
Also note that the released version of garnet does not have broadcast
support, i.e. a broadcast is converted into individual messages at the
source, one for each destination and sent out.
If you use the simple network, the switch models there can handle
broadcasts. You could perhaps use that as a starting point and then edit
the code to model the crossbar you want.
cheers,
Tushar
Tushar,
The reason I am modifying the protocol is that I want to implement a
shared-bus based crossbar where each node has its own dedicated write
channel and other nodes listen to it. However, I see that ruby does not
support hardware broadcast, so modifying the protocol seems to me the only
way to implement this. I think by modifying all the messages to be
broadcast messages can solve this. It would be really helpful if you could
provide me more information how to implement this kind of topology. I think
mXbar is very very close to what I wanna do. Thank you very much!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add
more messages in the network? MOESI_hammer inherently broadcasts to all L1s
which gives you a broadcast to all nodes. I don't think you should mess
with the protocol, unless you have a strong justification for why those
extra messages are needed: example you come up with some new protocol that
sends more broadcasts instead of tracking some state etc In which case,
yes you would have to define extra transitions to handle the extra messages.
The two broadcast based protocols provided with gem5 are MOESI_hammer
(where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's
broadcast to all L2s
cheers,
Tushar
Hi Tushar,
Thanks for your reply! I think you are right. I need to deal with those
extra messages. Any suggestions how to do that? Should I define some new
transitions and get rid of those messages?
I have read your 1 to many paper before. If I want to implement some
crossbar topology which supports broadcast like mXbar, do you think
modifying the protocol is the right way to do or there is some other better
method? Thank you very much for your help.
Best,
Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function needs
to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's and
directories, instead only only to L1's, you also need to account for what
happens when these directories receive these messages. My guess is that
these messages are being inserted into some message buffer that is not part
of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the messages.

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Tushar Krishna

2012-07-27 22:10:34 UTC

Permalink

Yes good point, if you use MOSI_SMP_bcast, you do need some ordering point in the network.
gem5 does not have a hierarchical switch topology.

In terms of direction, I think you need to decide what your exact protocol should be, and then decide whether it is easier to code up MOSI_SMP_bcast in gem5, or modify an existing protocol in gem5 (such as MOESI_hammer) to always perform broadcasts.

cheers,
Tushar

Oh, I forgot to ask that if I also need to build hierarchical switch topology for the MOSI_SMP_bcast protocol to guarantee the order required or the current GEM5 supports that already....Thanks.
Best,
Jinzhu
Hi Tushar,
Your suggestions are really helpful. Thank you very much! For now, I think I will take Pt2Pt as the basic topology to work with since it has all the connections. I think I can view it the same way as the single write multiple read shared-bus crossbar. I just need to create a broadcast protocol for it.
You are right, MOESI_hammer is not a good choice since it's essentially a directory protocol. MOSI_SMP_bcast is a much better reference, although I am not sure how to modify that since it is for GEMS' hierarchical switch and it also has some directories inside. Hope I will figure it out.
Do you think this is the right direction to go? Thanks a lot again!
Best,
Jinzhu
Hi Jinzhu,
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar
For (1) you want every message to be a broadcast. I am not sure if MOESI_hammer is the correct protocol to use as your starting point. The reason is that MOESI_hammer is essentially a directory protocol, with the directory having no state. So every cache miss first goes to its home node, and the home node then performs a broadcast to all other caches, and then the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol for a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to model the kind of protocol you want.
For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I had optimized this pipeline, and added an mXbar in the switch traversal stage. Also note that the released version of garnet does not have broadcast support, i.e. a broadcast is converted into individual messages at the source, one for each destination and sent out.
If you use the simple network, the switch models there can handle broadcasts. You could perhaps use that as a starting point and then edit the code to model the crossbar you want.
cheers,
Tushar

Tushar Krishna

2012-07-27 22:24:50 UTC

Permalink

For instance, you could think of the "directory" in the MOESI_hammer protocol as something analogous to the bus arbiter, which receives unicast requests from all caches, and then sends them out on the bus (broadcast) in some order

- Tushar

Post by Tushar Krishna
Yes good point, if you use MOSI_SMP_bcast, you do need some ordering point in the network.
gem5 does not have a hierarchical switch topology.
In terms of direction, I think you need to decide what your exact protocol should be, and then decide whether it is easier to code up MOSI_SMP_bcast in gem5, or modify an existing protocol in gem5 (such as MOESI_hammer) to always perform broadcasts.
cheers,
Tushar

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

gem5 gem5

2012-07-27 22:36:21 UTC

Permalink

Post by Tushar Krishna
For instance, you could think of the "directory" in the MOESI_hammer
protocol as something analogous to the bus arbiter, which receives unicast
requests from all caches, and then sends them out on the bus (broadcast) in
some order
- Tushar
Yes good point, if you use MOSI_SMP_bcast, you do need some ordering point in the network.
gem5 does not have a hierarchical switch topology.
In terms of direction, I think you need to decide what your exact protocol
should be, and then decide whether it is easier to code up MOSI_SMP_bcast
in gem5, or modify an existing protocol in gem5 (such as MOESI_hammer) to
always perform broadcasts.
cheers,
Tushar
Oh, I forgot to ask that if I also need to build hierarchical switch
topology for the MOSI_SMP_bcast protocol to guarantee the order required or
the current GEM5 supports that already....Thanks.
Best,
Jinzhu

Post by gem5 gem5
Hi Tushar,
Your suggestions are really helpful. Thank you very much! For now, I
think I will take Pt2Pt as the basic topology to work with since it has all
the connections. I think I can view it the same way as the single write
multiple read shared-bus crossbar. I just need to create a broadcast
protocol for it.
You are right, MOESI_hammer is not a good choice since it's essentially a
directory protocol. MOSI_SMP_bcast is a much better reference, although I
am not sure how to modify that since it is for GEMS' hierarchical switch
and it also has some directories inside. Hope I will figure it out.
Do you think this is the right direction to go? Thanks a lot again!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar
For (1) you want every message to be a broadcast. I am not sure if
MOESI_hammer is the correct protocol to use as your starting point. The
reason is that MOESI_hammer is essentially a directory protocol, with the
directory having no state. So every cache miss first goes to its home node,
and the home node then performs a broadcast to all other caches, and then
the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be
broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol
for a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up
something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to
model the kind of protocol you want.
For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I had
optimized this pipeline, and added an mXbar in the switch traversal stage.
Also note that the released version of garnet does not have broadcast
support, i.e. a broadcast is converted into individual messages at the
source, one for each destination and sent out.
If you use the simple network, the switch models there can handle
broadcasts. You could perhaps use that as a starting point and then edit
the code to model the crossbar you want.
cheers,
Tushar
Tushar,
The reason I am modifying the protocol is that I want to implement a
shared-bus based crossbar where each node has its own dedicated write
channel and other nodes listen to it. However, I see that ruby does not
support hardware broadcast, so modifying the protocol seems to me the only
way to implement this. I think by modifying all the messages to be
broadcast messages can solve this. It would be really helpful if you could
provide me more information how to implement this kind of topology. I think
mXbar is very very close to what I wanna do. Thank you very much!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add
more messages in the network? MOESI_hammer inherently broadcasts to all L1s
which gives you a broadcast to all nodes. I don't think you should mess
with the protocol, unless you have a strong justification for why those
extra messages are needed: example you come up with some new protocol that
sends more broadcasts instead of tracking some state etc In which case,
yes you would have to define extra transitions to handle the extra messages.
The two broadcast based protocols provided with gem5 are MOESI_hammer
(where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's
broadcast to all L2s
cheers,
Tushar
Hi Tushar,
Thanks for your reply! I think you are right. I need to deal with those
extra messages. Any suggestions how to do that? Should I define some new
transitions and get rid of those messages?
I have read your 1 to many paper before. If I want to implement some
crossbar topology which supports broadcast like mXbar, do you think
modifying the protocol is the right way to do or there is some other better
method? Thank you very much for your help.
Best,
Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function
needs to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's
and directories, instead only only to L1's, you also need to account for
what happens when these directories receive these messages. My guess is
that these messages are being inserted into some message buffer that is not
part of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the

messages. Since multicast is supported, I think it's possible to do this.
I got some problem with my first step. When I modify
out_msg.Destination.broadcast(MachineType:L1Cache); to be
out_msg.Destination.broadcast() which I assume means to broadcast to all
nodes, inside of action(fn_forwardRequestIfNecessary, "fn", desc="Forward
requests if necessary"), I got some error like this when run the

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

Tushar Krishna

2012-07-27 23:29:11 UTC

Permalink

If you instantiate just one directory in MOESI_hammer, all requests will go to it, and it will automatically send out messages in some order You need to then make sure your interconnect delivers them to all nodes in the same order which is inherently true with a bus.

I think you should go through both protocols and see which is easier for you to understand and closer to what you wish to do, and then go along that direction.

It should not be hard to code any topology, including hierarchical switch in gem5.

All the best!
Tushar

sounds cool. In this way I need to modify the directory code and somehow create a correct order for all the messages? any idea on how to implement that?
Do you think it might be a little bit easier if I can create a hierarchical switch similar to the one in gems and then modify the MOSI_SMP_bcast? I should be able to create a hierarchical switch with gem5 right?
Thank you! It's so nice of you!
Best,
Jinzhu
For instance, you could think of the "directory" in the MOESI_hammer protocol as something analogous to the bus arbiter, which receives unicast requests from all caches, and then sends them out on the bus (broadcast) in some order
- Tushar

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

gem5 gem5

2012-07-28 02:54:13 UTC

Permalink

Ha, I see. thank you very much for explaining everything to me:) I will
start to learn those protocols now and might post more questions if I have
any later.

Best,

Jinzhu

Post by Tushar Krishna
If you instantiate just one directory in MOESI_hammer, all requests will
go to it, and it will automatically send out messages in some order You
need to then make sure your interconnect delivers them to all nodes in the
same order which is inherently true with a bus.
I think you should go through both protocols and see which is easier for
you to understand and closer to what you wish to do, and then go along that
direction.
It should not be hard to code any topology, including hierarchical switch in gem5.
All the best!
Tushar
sounds cool. In this way I need to modify the directory code and somehow
create a correct order for all the messages? any idea on how to implement
that?
Do you think it might be a little bit easier if I can create a
hierarchical switch similar to the one in gems and then modify
the MOSI_SMP_bcast? I should be able to create a hierarchical switch with
gem5 right?
Thank you! It's so nice of you!
Best,
Jinzhu

Post by Tushar Krishna
For instance, you could think of the "directory" in the MOESI_hammer
protocol as something analogous to the bus arbiter, which receives unicast
requests from all caches, and then sends them out on the bus (broadcast) in
some order
- Tushar
Yes good point, if you use MOSI_SMP_bcast, you do need some ordering
point in the network.
gem5 does not have a hierarchical switch topology.
In terms of direction, I think you need to decide what your exact
protocol should be, and then decide whether it is easier to code up
MOSI_SMP_bcast in gem5, or modify an existing protocol in gem5 (such as
MOESI_hammer) to always perform broadcasts.
cheers,
Tushar
Oh, I forgot to ask that if I also need to build hierarchical switch
topology for the MOSI_SMP_bcast protocol to guarantee the order required or
the current GEM5 supports that already....Thanks.
Best,
Jinzhu

Post by gem5 gem5
Hi Tushar,
Your suggestions are really helpful. Thank you very much! For now, I
think I will take Pt2Pt as the basic topology to work with since it has all
the connections. I think I can view it the same way as the single write
multiple read shared-bus crossbar. I just need to create a broadcast
protocol for it.
You are right, MOESI_hammer is not a good choice since it's essentially
a directory protocol. MOSI_SMP_bcast is a much better reference, although I
am not sure how to modify that since it is for GEMS' hierarchical switch
and it also has some directories inside. Hope I will figure it out.
Do you think this is the right direction to go? Thanks a lot again!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
(1) broadcast-based coherence protocol
(2) shared-bus based crossbar
For (1) you want every message to be a broadcast. I am not sure if
MOESI_hammer is the correct protocol to use as your starting point. The
reason is that MOESI_hammer is essentially a directory protocol, with the
directory having no state. So every cache miss first goes to its home node,
and the home node then performs a broadcast to all other caches, and then
the owner responds to the requestor.
What you need from what I understand is the cache miss to directly be
broadcast, like on a bus, and the owner then responding.
The original GEMS distribution actually came with a broadcast protocol
for a bus, called MOSI_SMP_bcast.
You could try to look at that from the GEMS repo, and perhaps code up
something similar in gem5.
Or stick to MOESI_hammer for now, but you will need to modify it to
model the kind of protocol you want.
For (2), you first need to create a topology.
You can look at configs/topologies to see how topologies are created.
The routers in the topology get modeled either as garnet-routers or simple-routers.
You can choose to work with either garnet or the default simple network.
Garnet will however model a 4-5 stage router pipeline. In my paper I
had optimized this pipeline, and added an mXbar in the switch traversal
stage. Also note that the released version of garnet does not have
broadcast support, i.e. a broadcast is converted into individual messages
at the source, one for each destination and sent out.
If you use the simple network, the switch models there can handle
broadcasts. You could perhaps use that as a starting point and then edit
the code to model the crossbar you want.
cheers,
Tushar
Tushar,
The reason I am modifying the protocol is that I want to implement a
shared-bus based crossbar where each node has its own dedicated write
channel and other nodes listen to it. However, I see that ruby does not
support hardware broadcast, so modifying the protocol seems to me the only
way to implement this. I think by modifying all the messages to be
broadcast messages can solve this. It would be really helpful if you could
provide me more information how to implement this kind of topology. I think
mXbar is very very close to what I wanna do. Thank you very much!
Best,
Jinzhu

Post by Tushar Krishna
Hi Jinzhu,
Why do you want to broadcast to L1s and the directories? Just to add
more messages in the network? MOESI_hammer inherently broadcasts to all L1s
which gives you a broadcast to all nodes. I don't think you should mess
with the protocol, unless you have a strong justification for why those
extra messages are needed: example you come up with some new protocol that
sends more broadcasts instead of tracking some state etc In which case,
yes you would have to define extra transitions to handle the extra messages.
The two broadcast based protocols provided with gem5 are MOESI_hammer
(where a directory broadcasts to all L1s) and MOESI_CMP_token, where L1's
broadcast to all L2s
cheers,
Tushar
Hi Tushar,
Thanks for your reply! I think you are right. I need to deal with
those extra messages. Any suggestions how to do that? Should I define some
new transitions and get rid of those messages?
I have read your 1 to many paper before. If I want to implement some
crossbar topology which supports broadcast like mXbar, do you think
modifying the protocol is the right way to do or there is some other better
method? Thank you very much for your help.
Best,
Jinzhu

Post by Tushar Krishna
The error has nothing to do with the virtual network being ordered or not.
If you look at MessageBuffer.hh, line 131, a setOrdering function
needs to be called for each MessageBuffer.
This is called from the coherence protocol's generated files. [The
ordering itself might be true or false depending on the vnet, but this
function has to be called for the message buffer to be valid].
If all you have changed in the protocol is to broadcast to all L1's
and directories, instead only only to L1's, you also need to account for
what happens when these directories receive these messages. My guess is
that these messages are being inserted into some message buffer that is not
part of the protocol specifications and hence this error shows up.
- Tushar

Post by gem5 gem5
Hi all,
I want to modify MOESI_hammer protocol to broadcast all the

ordered="true", but still I cannot get rid of this error. Any help will be
appreciated. Thanks a lot!

Post by gem5 gem5
Jinzhu
_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users

_______________________________________________
gem5-users mailing list
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users