Discussion:
Why are highly busy zIIPs worse than highly busy CPs?
(too old to reply)
Peter Hunkeler
2018-06-07 07:27:01 UTC
Permalink
There are some statements around zIIP utilization which I read here and there. Statements like:

- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".



For this discussion, lets assume equal speed CPs and zIIPs, and a reasonable CP to zIIP ratio, and more than on processor of each kind.


It has been a long time strength of IBM Z (and all the predecessors) that the CPs in an LPAR can be utilized way above 90% without major problems arising. I seem to understand that this has changed lately, but still some 85% (?) should be fine.


Now, all work running on zIIPs was once work running on CPs (and still is if there are no zIIPs). So the work is no different (apart from much being run under an SRB instead of a TCB), and the response time requirement is no different. Right?


If so, how comes that busy zIIPs are said to be more of a problem than busy CPs? If the work can accept some queueing when run on CPs, why not when run on zIIPs. Queueing theory should apply equally to both.



When a processor is busy 50%, then 50% of the time there is at least one ready task, the one executing. Maybe there are some more waiting on the work queue. But these 50% say nothing about the delay of the tasks on the work queue.


In a simplified case, assume 5 tasks with equal priority, each one quickly, say after 0.5 ms, coming to the point where it has to give up the processor for a very short period of time before being requeued on the work queue. They all constantly work that way for 30 seconds in row, then become undispatchable for the remaining 30 seconds of that 50% busy minute. During the first 30 seconds, the zIIP is 100% busy, and after 3.2ms (ZIIPAWMT), the zIIP will ask a CP for help.


None of the tasks has been delayed by 3.2ms, although the ZIIP recognized its work queue has not become empty for 3.2ms and asked for help. To the contrary, the work has gotten better service because two processors are now serving the single work queue. (Again for simplicity, not currently taking priorities into account).


Same case but the task are working 1ms each time. Now it always takes more than 3.2ms for the last task on the work queue before it is being redispatched as long as the zIIP has not asked for help. But the zIIP will ask for help after 3.2ms, and the delay for the tasks will shrink.


Isn't this a better situation for zIIP work than for non-zIIP work? Same scenario on CPs. There is no-one to help.


Any thoughts?
--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Elardus Engelbrecht
2018-06-07 09:36:20 UTC
Permalink
Post by Peter Hunkeler
- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
Who said it? And why 30%? Just curious.

My one zIIP (for about 8 LPARs on one machine) is usually anything from 5% to 40-50% in workdays, while the zIIP management overhead is usually from 0.5% to 1%.

No complaints from my users or colleagues.

Groete / Greetings
Elardus Engelbrecht

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Mike Schwab
2018-06-07 13:59:02 UTC
Permalink
Once the delay is long enough, the CP does the work. They cost about
10X the price of zIIPs.
On Thu, Jun 7, 2018 at 4:36 AM Elardus Engelbrecht
Post by Elardus Engelbrecht
Post by Peter Hunkeler
- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
Who said it? And why 30%? Just curious.
My one zIIP (for about 8 LPARs on one machine) is usually anything from 5% to 40-50% in workdays, while the zIIP management overhead is usually from 0.5% to 1%.
No complaints from my users or colleagues.
Groete / Greetings
Elardus Engelbrecht
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-07 16:13:56 UTC
Permalink
Once the delay is long enough, the CP does the work. They cost about 10X the price of zIIPs.
I understand the potential impact on the software bill zIIP-on-CP might have. That is not the point I want to get a better understanding. I'm interested in the technical aspects, only.


IMHO, from a technical point of view the invention of zIIPs and zAAPs is pure bullshit. From a financial point of view they are helping lowering costs -- and that is their only reason to be.


I predict (and no I don't have any insight into IBM's plans) zIIPs will be gone in the not so distant future. Container Pricing makes the zIIP/zAAP concept superfluous.


--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Martin Packer
2018-06-07 19:03:58 UTC
Permalink
As you'll see from my slides you really wouldn't want to delay the
Prefetch and Deferred Write engines in any DB2 you care about.

That's one example, perhaps the main one.

Martin Packer

zChampion, Systems Investigator & Performance Troubleshooter, IBM

+44-7802-245-584

email: ***@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker

Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/ or

https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2


Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From: Peter Hunkeler <***@GMX.CH>
To: IBM-***@LISTSERV.UA.EDU
Date: 07/06/2018 17:14
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy
CPs?
Once the delay is long enough, the CP does the work. They cost about 10X the price of zIIPs.
I understand the potential impact on the software bill zIIP-on-CP might
have. That is not the point I want to get a better understanding. I'm
interested in the technical aspects, only.


IMHO, from a technical point of view the invention of zIIPs and zAAPs is
pure bullshit. From a financial point of view they are helping lowering
costs -- and that is their only reason to be.


I predict (and no I don't have any insight into IBM's plans) zIIPs will be
gone in the not so distant future. Container Pricing makes the zIIP/zAAP
concept superfluous.


--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Phil Smith
2018-06-07 15:39:01 UTC
Permalink
Post by Peter Hunkeler
- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".
There are some knobs that can control this. And it's surely YMMV. If you had, for example, one heavyweight thread that was zIIP-eligible, one might assume that it could saturate that zIIP without issues. But that's not going to be a common scenario.

Scheduling an SRB isn't cheap. I don't know whether that's why, but for whatever reason, IBM has built this "fall back to the CP" mechanism. Kathy Walsh of IBM has a pretty good session that talks about this; I saw it at the IBM Systems Technical University in Orlando last month. Session is z101268.pdf; I can't find it on the web, either under that number or by searching some key strings in it. Sorry...
--
...phsiii

Phil Smith III
Senior Architect & Product Manager, Mainframe & Enterprise
Distinguished Technologist
Micro Focus (Voltage)

***@microfocus.com
T 703-476-4511
M 703-568-6662
Herndon, VA


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Charles Mills
2018-06-07 16:00:27 UTC
Permalink
Isn't "fall back to the CP" because one would typically want one's work to
run *somewhere* even if a zIIP were not available but perhaps a CP was?

zIIP's are all about software costs, right? So scheduling work is like
buying a seat on Southwest. I need to get there, so if none of the "wanna
get away" seats are available, I will buy a full-fare seat.

Charles


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On
Behalf Of Phil Smith
Sent: Thursday, June 7, 2018 8:38 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Peter Hunkeler
There are some statements around zIIP utilization which I read here and
- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before
the busy zIIP asks for help from a CP".

There are some knobs that can control this. And it's surely YMMV. If you
had, for example, one heavyweight thread that was zIIP-eligible, one might
assume that it could saturate that zIIP without issues. But that's not going
to be a common scenario.

Scheduling an SRB isn't cheap. I don't know whether that's why, but for
whatever reason, IBM has built this "fall back to the CP" mechanism. Kathy
Walsh of IBM has a pretty good session that talks about this; I saw it at
the IBM Systems Technical University in Orlando last month. Session is
z101268.pdf; I can't find it on the web, either under that number or by
searching some key strings in it. Sorry...
--
...phsiii

Phil Smith III
Senior Architect & Product Manager, Mainframe & Enterprise
Distinguished Technologist
Micro Focus (Voltage)

***@microfocus.com
T 703-476-4511
M 703-568-6662
Herndon, VA


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-07 16:21:52 UTC
Permalink
Post by Charles Mills
Isn't "fall back to the CP" because one would typically want one's work to
run *somewhere* even if a zIIP were not available but perhaps a CP was?




If you meant no zIIPs are available to the LPAR (not configured or the CEC does not have some), then there is no fall-back. Work units get queue on CP work queues initially; the zIIP work queue is not being used. At least this is how I understand it.


If you meant no free zIIP capacity is available, then part of the decision might have been that initially you could have only half as many zIIPs as you had CPs.


--
Peter Hunkeler





----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-07 16:16:25 UTC
Permalink
Post by Phil Smith
Scheduling an SRB isn't cheap. I don't know whether that's why, but for whatever reason, IBM has built this "fall back to the CP" mechanism.
What falls back is still SRBs. The scheduling overhead has already been done before.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Jesse 1 Robinson
2018-06-07 16:54:27 UTC
Permalink
We've had zIIPs for years. We monitored usage but were never terribly concerned as long as numbers didn't skyrocket. When we moved to DB2 V10, however, we were warned that 'some customers' were experiencing serious performance problems when zIIP eligible work spilled over to general CPs. The reason, we were told, was that while general CP management had evolved over the decades with a huge boatload of OS code to handle contention, zIIPs were newcomers that were more or less on their own in playground competition. The wiry little guys with little bully protection.

The message we got was that an overloaded zIIP could lead to performance problems--and rolling average spikes--that could be worse (!) than having no zIIP at all. We were sufficiently alarmed that we were moved to buy yet another zIIP to guard against calamity in production. Because we acquired the extra zIIP early on, I can't say what the consequence would have been if we had done nothing, but this is certainly the tale I'd rather be telling.

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
***@sce.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler
Sent: Thursday, June 07, 2018 9:21 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: (External):AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Charles Mills
Isn't "fall back to the CP" because one would typically want one's work
to
run *somewhere* even if a zIIP were not available but perhaps a CP was?




If you meant no zIIPs are available to the LPAR (not configured or the CEC does not have some), then there is no fall-back. Work units get queue on CP work queues initially; the zIIP work queue is not being used. At least this is how I understand it.


If you meant no free zIIP capacity is available, then part of the decision might have been that initially you could have only half as many zIIPs as you had CPs.


--
Peter Hunkeler


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Martin Packer
2018-06-07 19:04:58 UTC
Permalink
Right. Though you've never been in my audience :-( you would've got the
same message about DB2 V10, a fortiori 11, 12 from me.

Cheers, Martin

Martin Packer

zChampion, Systems Investigator & Performance Troubleshooter, IBM

+44-7802-245-584

email: ***@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker

Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/ or

https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2


Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA



From: Jesse 1 Robinson <***@SCE.COM>
To: IBM-***@LISTSERV.UA.EDU
Date: 07/06/2018 17:54
Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?
Sent by: IBM Mainframe Discussion List <IBM-***@LISTSERV.UA.EDU>



We've had zIIPs for years. We monitored usage but were never terribly
concerned as long as numbers didn't skyrocket. When we moved to DB2 V10,
however, we were warned that 'some customers' were experiencing serious
performance problems when zIIP eligible work spilled over to general CPs.
The reason, we were told, was that while general CP management had evolved
over the decades with a huge boatload of OS code to handle contention,
zIIPs were newcomers that were more or less on their own in playground
competition. The wiry little guys with little bully protection.

The message we got was that an overloaded zIIP could lead to performance
problems--and rolling average spikes--that could be worse (!) than having
no zIIP at all. We were sufficiently alarmed that we were moved to buy yet
another zIIP to guard against calamity in production. Because we acquired
the extra zIIP early on, I can't say what the consequence would have been
if we had done nothing, but this is certainly the tale I'd rather be
telling.

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
***@sce.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On
Behalf Of Peter Hunkeler
Sent: Thursday, June 07, 2018 9:21 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: (External):AW: Re: Why are highly busy zIIPs worse than highly
busy CPs?
Post by Charles Mills
Isn't "fall back to the CP" because one would typically want one's work
to
run *somewhere* even if a zIIP were not available but perhaps a CP was?




If you meant no zIIPs are available to the LPAR (not configured or the CEC
does not have some), then there is no fall-back. Work units get queue on
CP work queues initially; the zIIP work queue is not being used. At least
this is how I understand it.


If you meant no free zIIP capacity is available, then part of the decision
might have been that initially you could have only half as many zIIPs as
you had CPs.
--
Peter Hunkeler


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-07 19:08:41 UTC
Permalink
... however, we were warned that 'some customers' were experiencing serious performance problems when zIIP eligible work spilled over to general CPs.
Yeah, that is what I read and hear also, and I have no reason not to believe it. In fact, I suspect we've just been bitten by zIIP overload. However, this is not at the level of technical detail I'm seeking to understand.
The reason, we were told, was that while general CP management had evolved over the decades with a huge boatload of OS code to handle contention, zIIPs were newcomers that were more or less on their own in playground competition. The wiry little guys with little bully protection.
Huh? Do you want to say you've been told that zIIP management (dispatcher and WLM/SRM) is new code that is distinct from the CP management (dispatcher and WLM/SRM) code? I must be misunderstanding you.

zIIPs are general purpose processors exactly the same as CPs are, aren't they. They will run whatever work unit happens to be found on their dispatcher queue, don't they? The decision which work queue to chain a newly ready work unit on is made at queueing time. It is just an (not really) arbitrary decision that IBM chose only enclave SRBs are eligible to be queued on the zIIP WUQ.

The only new thing is that the dispatcher, when running on a zIIP, may ask, i.e. flag a CP to also look for work on the zIIP WUQ. And it is the dispatcher, when running on that CP, which looks at both WUQ to select the next work unit to dispatch. This is how I think it works. Tell me, anyone, if this is not correct.

--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Joel C. Ewing
2018-06-07 22:24:23 UTC
Permalink
Post by Peter Hunkeler
- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".
For this discussion, lets assume equal speed CPs and zIIPs, and a reasonable CP to zIIP ratio, and more than on processor of each kind.
It has been a long time strength of IBM Z (and all the predecessors) that the CPs in an LPAR can be utilized way above 90% without major problems arising. I seem to understand that this has changed lately, but still some 85% (?) should be fine.
Now, all work running on zIIPs was once work running on CPs (and still is if there are no zIIPs). So the work is no different (apart from much being run under an SRB instead of a TCB), and the response time requirement is no different. Right?
If so, how comes that busy zIIPs are said to be more of a problem than busy CPs? If the work can accept some queueing when run on CPs, why not when run on zIIPs. Queueing theory should apply equally to both.
When a processor is busy 50%, then 50% of the time there is at least one ready task, the one executing. Maybe there are some more waiting on the work queue. But these 50% say nothing about the delay of the tasks on the work queue.
In a simplified case, assume 5 tasks with equal priority, each one quickly, say after 0.5 ms, coming to the point where it has to give up the processor for a very short period of time before being requeued on the work queue. They all constantly work that way for 30 seconds in row, then become undispatchable for the remaining 30 seconds of that 50% busy minute. During the first 30 seconds, the zIIP is 100% busy, and after 3.2ms (ZIIPAWMT), the zIIP will ask a CP for help.
None of the tasks has been delayed by 3.2ms, although the ZIIP recognized its work queue has not become empty for 3.2ms and asked for help. To the contrary, the work has gotten better service because two processors are now serving the single work queue. (Again for simplicity, not currently taking priorities into account).
Same case but the task are working 1ms each time. Now it always takes more than 3.2ms for the last task on the work queue before it is being redispatched as long as the zIIP has not asked for help. But the zIIP will ask for help after 3.2ms, and the delay for the tasks will shrink.
Isn't this a better situation for zIIP work than for non-zIIP work? Same scenario on CPs. There is no-one to help.
Any thoughts?
I suspect the point of the guidelines was primarily to insure that work
intended for zIIPs didn't get redirected to general CPs and raise z/OS
software charges.  But, if those guidelines have been observed, zIIP
workloads would have been experiencing minimal CPU queueing delays and
any increase in delay from higher zIIP utilization and offloading part
of the workload to CPs may be perceived by users as a significant
performance hit.

How prevalent are installations today where the CPs run at top speed, in
other words at the same speed as zIIP engines? In other words, Is it
that valid to assume equal speed processors?  Clearly guidelines for
lower zIIP utilization matter more when there is a difference, as
offloading any zIIP work to a slower CP would elongate processing and
response time, even if there is no delay waiting for a processor.

Another possible issue is that application work that is zIIP-eligible at
least historically tended to be things like java that were more
CPU-intensive.  These applications quite possibly are in a service class
of lesser importance.  As long as they can run on a zIIP engine, they
only compete with similar workloads.  But, if the zIIP utilization
reaches the point that work is offloaded to a CP, it would seem logical
to me to expect other "more important" workloads competing for the CP to
get served first, and if that is indeed the case the queueing time to
actually get service for lower importance zIIP workload from an equally
busy CP could be much greater.  It doesn't seem valid to me to assume
that just because another CP is allowed to to zIIP work that a CP will
immediately be free to actually do that work, or that a CP will have
queueing delays similar to the zIIP engines -- the workload on the CP is
totally different.

    Joel C. Ewing
--
Joel C. Ewing, Bentonville, AR ***@acm.org

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-08 19:36:52 UTC
Permalink
.... the workload on the CP is totally different.
Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-08 19:41:50 UTC
Permalink
Post by Parwez Hamid
How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines?
I haven't got the faintest idea. We do, but that doesn't matter for this discussion. I thought this is complex enough, so I take one part of complexity out first: Different speed engines.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Christopher Y. Blaicher
2018-06-08 21:48:45 UTC
Permalink
I wish Peter Relson would comment on this so we all can get the straight answer, but he may not be able to.

From what little I know and most that I summarize and guess at, it seems the following to be what is happening.

First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.

I think the queuing process is basically FIFO and once unit of work is dispatched, it runs to the end or a program break point happens. A program break point could be some form of PAUSE or a IEAVXTSW wait.

Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.

There is the ZIIPAWMT parameter in IEAOPTxx which affects how often idle zIIPs get woken up to see if there is work. This has two effects. A unit of work put on the zIIP queue has to wait up to that long to get dispatched on a zIIP, or if at the end of that interval all zIIPs are busy, only then will the unit of work be dispatched on a GP. To be fair, when a zIIP completes a unit of work, it checks for more work to process, so not everything waits a dispatch cycle.

So what does this all mean? In a low activity environment, your mean time to dispatch is around half the ZIIPAWMT time. In a high activity environment it means the mean time to dispatch on a GP because all he zIIPs are busy is ZIIPAWMT.

This starts to get into queuing theory, but if a single zIIP processor is over 30% busy, about 30% or more of the work to run on a zIIP will wind up waiting the ZIIPAWMT time and then get dispatched on a GP, thus adding the insult of delaying the processing of the work to the injury of running on a GP that can cost real money.

If you add more zIIP engines, there is a greater chance of a unit of work ending on one of the engines and thus the work getting dispatched on a zIIP without waiting the full ZIIPAWMT.

A reasonable set of indicators can be gotten from the SMF type 30 record. The SMF30_TIME_ON_ZIIP and SMF30_TIME_ZIIP_ON_CP are valuable indicators.

If SMF30_TIME_ZIIP_ON_CP is 30% or more of SMF30_TIME_ON_ZIIP, then you probably need another zIIP engine. Because type 30 records are job centric, I would suggest using the SMF 30 SUBTYPE 2 and 3 records for all the jobs in a time interval and totaling the data. Using data from a single job or group of jobs may not provide a complete picture. Also, the 30% value mentioned maybe more or less than your environment can tolerate. YMMV, so only use it as a starting point.

As I said, most of this is just a guess at what is happening with zIIP dispatching. None of this information has been validated and neither Syncsort or I are implying or stating any course of action a user should pursue. Each user must evaluate their own needs and requirements and make decisions based solely on their own research and evaluation of their circumstances.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: ***@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler
Sent: Friday, June 8, 2018 3:42 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Parwez Hamid
How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines?
I haven't got the faintest idea. We do, but that doesn't matter for this discussion. I thought this is complex enough, so I take one part of complexity out first: Different speed engines.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Mike Schwab
2018-06-08 22:26:51 UTC
Permalink
I have seen press stories where Full speed CPs see a lot of wait time
(waiting for data, interrupt, etc.), where slower CPs credit the
delays to the reduced capacity so they will see little to no wait
time.
On Fri, Jun 8, 2018 at 4:48 PM Christopher Y. Blaicher
Post by Christopher Y. Blaicher
I wish Peter Relson would comment on this so we all can get the straight answer, but he may not be able to.
From what little I know and most that I summarize and guess at, it seems the following to be what is happening.
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
I think the queuing process is basically FIFO and once unit of work is dispatched, it runs to the end or a program break point happens. A program break point could be some form of PAUSE or a IEAVXTSW wait.
Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.
There is the ZIIPAWMT parameter in IEAOPTxx which affects how often idle zIIPs get woken up to see if there is work. This has two effects. A unit of work put on the zIIP queue has to wait up to that long to get dispatched on a zIIP, or if at the end of that interval all zIIPs are busy, only then will the unit of work be dispatched on a GP. To be fair, when a zIIP completes a unit of work, it checks for more work to process, so not everything waits a dispatch cycle.
So what does this all mean? In a low activity environment, your mean time to dispatch is around half the ZIIPAWMT time. In a high activity environment it means the mean time to dispatch on a GP because all he zIIPs are busy is ZIIPAWMT.
This starts to get into queuing theory, but if a single zIIP processor is over 30% busy, about 30% or more of the work to run on a zIIP will wind up waiting the ZIIPAWMT time and then get dispatched on a GP, thus adding the insult of delaying the processing of the work to the injury of running on a GP that can cost real money.
If you add more zIIP engines, there is a greater chance of a unit of work ending on one of the engines and thus the work getting dispatched on a zIIP without waiting the full ZIIPAWMT.
A reasonable set of indicators can be gotten from the SMF type 30 record. The SMF30_TIME_ON_ZIIP and SMF30_TIME_ZIIP_ON_CP are valuable indicators.
If SMF30_TIME_ZIIP_ON_CP is 30% or more of SMF30_TIME_ON_ZIIP, then you probably need another zIIP engine. Because type 30 records are job centric, I would suggest using the SMF 30 SUBTYPE 2 and 3 records for all the jobs in a time interval and totaling the data. Using data from a single job or group of jobs may not provide a complete picture. Also, the 30% value mentioned maybe more or less than your environment can tolerate. YMMV, so only use it as a starting point.
As I said, most of this is just a guess at what is happening with zIIP dispatching. None of this information has been validated and neither Syncsort or I are implying or stating any course of action a user should pursue. Each user must evaluate their own needs and requirements and make decisions based solely on their own research and evaluation of their circumstances.
Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com
Data quality leader Trillium Software is now a part of Syncsort.
-----Original Message-----
Sent: Friday, June 8, 2018 3:42 PM
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Parwez Hamid
How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines?
I haven't got the faintest idea. We do, but that doesn't matter for this discussion. I thought this is complex enough, so I take one part of complexity out first: Different speed engines.
--
Peter Hunkeler
----------------------------------------------------------------------
________________________________
ATTENTION: -----
The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Martin Packer
2018-06-09 06:59:12 UTC
Permalink
To correct one thing: Not all the work is high priority. Three examples:

1) DDF threads selected to be zIIP-eligible.

2) Java - via zAAP on zIIP.

3) System XML

Cheers, Martin

Sent from my iPad
Post by Christopher Y. Blaicher
I wish Peter Relson would comment on this so we all can get the straight
answer, but he may not be able to.
Post by Christopher Y. Blaicher
From what little I know and most that I summarize and guess at, it seems
the following to be what is happening.
Post by Christopher Y. Blaicher
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
I think the queuing process is basically FIFO and once unit of work is
dispatched, it runs to the end or a program break point happens. A program
break point could be some form of PAUSE or a IEAVXTSW wait.
Post by Christopher Y. Blaicher
Some of this dispatcher design by IBM could be based on the assumption
that all the work is SRB and will be high priority work and of short
duration.
Post by Christopher Y. Blaicher
There is the ZIIPAWMT parameter in IEAOPTxx which affects how often idle
zIIPs get woken up to see if there is work. This has two effects. A unit
of work put on the zIIP queue has to wait up to that long to get dispatched
on a zIIP, or if at the end of that interval all zIIPs are busy, only then
will the unit of work be dispatched on a GP. To be fair, when a zIIP
completes a unit of work, it checks for more work to process, so not
everything waits a dispatch cycle.
Post by Christopher Y. Blaicher
So what does this all mean? In a low activity environment, your mean
time to dispatch is around half the ZIIPAWMT time. In a high activity
environment it means the mean time to dispatch on a GP because all he zIIPs
are busy is ZIIPAWMT.
Post by Christopher Y. Blaicher
This starts to get into queuing theory, but if a single zIIP processor is
over 30% busy, about 30% or more of the work to run on a zIIP will wind up
waiting the ZIIPAWMT time and then get dispatched on a GP, thus adding the
insult of delaying the processing of the work to the injury of running on a
GP that can cost real money.
Post by Christopher Y. Blaicher
If you add more zIIP engines, there is a greater chance of a unit of work
ending on one of the engines and thus the work getting dispatched on a zIIP
without waiting the full ZIIPAWMT.
Post by Christopher Y. Blaicher
A reasonable set of indicators can be gotten from the SMF type 30 record.
The SMF30_TIME_ON_ZIIP and SMF30_TIME_ZIIP_ON_CP are valuable indicators.
Post by Christopher Y. Blaicher
If SMF30_TIME_ZIIP_ON_CP is 30% or more of SMF30_TIME_ON_ZIIP, then you
probably need another zIIP engine. Because type 30 records are job
centric, I would suggest using the SMF 30 SUBTYPE 2 and 3 records for all
the jobs in a time interval and totaling the data. Using data from a
single job or group of jobs may not provide a complete picture. Also, the
30% value mentioned maybe more or less than your environment can tolerate.
YMMV, so only use it as a starting point.
Post by Christopher Y. Blaicher
As I said, most of this is just a guess at what is happening with zIIP
dispatching. None of this information has been validated and neither
Syncsort or I are implying or stating any course of action a user should
pursue. Each user must evaluate their own needs and requirements and make
decisions based solely on their own research and evaluation of their
circumstances.
Post by Christopher Y. Blaicher
Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com
Data quality leader Trillium Software is now a part of Syncsort.
-----Original Message-----
Behalf Of Peter Hunkeler
Post by Christopher Y. Blaicher
Sent: Friday, June 8, 2018 3:42 PM
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Joel C. Ewing
How prevalent are installations today where the CPs run at top speed, in
other words at the same speed as zIIP engines?
Post by Christopher Y. Blaicher
I haven't got the faintest idea. We do, but that doesn't matter for this
discussion. I thought this is complex enough, so I take one part of
complexity out first: Different speed engines.
Post by Christopher Y. Blaicher
--
Peter Hunkeler
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send
________________________________
ATTENTION: -----
The information contained in this message (including any files
transmitted with this message) may contain proprietary, trade secret or
other confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with this
message is always confidential and cannot be shared with any third parties
without prior written approval from Syncsort. This message is intended to
be read only by the individual or entity to whom it is addressed or by
their designee. If the reader of this message is not the intended
recipient, you are on notice that any use, disclosure, copying or
distribution of this message, in any form, is strictly prohibited. If you
have received this message in error, please immediately notify the sender
and/or Syncsort and destroy all copies of this message in your possession,
custody or control.
Post by Christopher Y. Blaicher
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-09 07:33:02 UTC
Permalink
Post by Christopher Y. Blaicher
Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.
This doesn't sound correct to me. Client SRBs (preemptive SRBs) were invented to have some work done in another address space at client priority. Enclave SRBs were invented to let those client SRBs be run on zIIPs. So, there is a mix of SRBs with different priorities (WLM goals, WLM service classes) competing for zIIP capacity.


And it's all preemptive kind of work.
--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-09 10:49:03 UTC
Permalink
Post by Christopher Y. Blaicher
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
Do you know this, or is it just an assumption on your side? After all I read, it still would't make sense to me.


If you think of the "need help" process for the zIIP to be special, isn't there a similar process for CPs? With hiperdispatch, the system tries to redispatch work to the same (L)CP as much as possible. In support of this, the CPs are grouped into nodes (I believe that is the term). Each node mainly serving its own, separate work init queue. But CPs from other nodes will help if one node becomes overloaded.
The above is how I remember it from a discussion I had with Robert Vaupel (IBM).


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Ed Jaffe
2018-06-09 13:46:09 UTC
Permalink
Post by Peter Hunkeler
Post by Christopher Y. Blaicher
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
Do you know this, or is it just an assumption on your side? After all I read, it still would't make sense to me.
The dispatcher is the dispatcher -- and it works with different WUQs. No
need to make things overly complex...
--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
http://www.phoenixsoftware.com/
--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Christopher Y. Blaicher
2018-06-09 14:58:37 UTC
Permalink
Peter,
I started off as saying, a lot of the descriptions are based on assumptions, as IBM has let out little on how the zIIP dispatcher works.

Also, I was only talking about SRBs on zIIPs, so non-enclave SRBs were not part of the discussion.

I believe hyper dispatch is very different from zIIP dispatch. I stand by my assumption that GP dispatch is very different from zIIP dispatch, or why would there be the ZIIPAWMT parameter and have the comment about waking up after that interval to see if there is work. When non-zIIP work comes ready and a processor is free, the work is dispatched. No waiting. The comments in the parameter description for ZIIPAWMT is describing a polling environment, at least to me.

I have looked through the internet, OK not the be-all and end-all but a reasonable place to start, and there is a lot on what can get dispatched on a zIIP, but no detail on how.

As I said, I wish someone from IBM would at least chime in with some level of description as to how zIIP dispatch works and why high zIIP utilization rates are, shall we say, not good. Then maybe we can stop guessing.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: ***@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler
Sent: Saturday, June 9, 2018 6:49 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Christopher Y. Blaicher
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
Do you know this, or is it just an assumption on your side? After all I read, it still would't make sense to me.


If you think of the "need help" process for the zIIP to be special, isn't there a similar process for CPs? With hiperdispatch, the system tries to redispatch work to the same (L)CP as much as possible. In support of this, the CPs are grouped into nodes (I believe that is the term). Each node mainly serving its own, separate work init queue. But CPs from other nodes will help if one node becomes overloaded.
The above is how I remember it from a discussion I had with Robert Vaupel (IBM).


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Jim Mulder
2018-06-09 16:53:40 UTC
Permalink
zIIP dispatching is the same as GP dispatching. ZIIPAWMT
has analogous parameters for GP (CCCAWMT) and zAAP (ZAAPAWMT).
Alternate wait management was created long before there were
specialty engines.

Jim Mulder z/OS Diagnosis, Design, Development, Test IBM Corp.
Poughkeepsie NY
Date: 06/09/2018 12:46 PM
Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?
Peter,
I started off as saying, a lot of the descriptions are based on
assumptions, as IBM has let out little on how the zIIP dispatcher works.
Also, I was only talking about SRBs on zIIPs, so non-enclave SRBs
were not part of the discussion.
I believe hyper dispatch is very different from zIIP dispatch. I
stand by my assumption that GP dispatch is very different from zIIP
dispatch, or why would there be the ZIIPAWMT parameter and have the
comment about waking up after that interval to see if there is work.
When non-zIIP work comes ready and a processor is free, the work is
dispatched. No waiting. The comments in the parameter description
for ZIIPAWMT is describing a polling environment, at least to me.
I have looked through the internet, OK not the be-all and end-all
but a reasonable place to start, and there is a lot on what can get
dispatched on a zIIP, but no detail on how.
As I said, I wish someone from IBM would at least chime in with some
level of description as to how zIIP dispatch works and why high zIIP
utilization rates are, shall we say, not good. Then maybe we can
stop guessing.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Christopher Y. Blaicher
2018-06-09 23:03:10 UTC
Permalink
Jim,
Thank you for the clarification, however, that still leaves open the question of WHY more than 30% on a single zIIP or 60% on two is not a good idea.

That is what triggered this whole chain of emails. Why can GPs run reasonably well at over 95% and zIIPs struggle, or so people have reported. We are a development shop and so rarely, if ever, push them that hard. We have, but that was doing things you would not do in the real world, like having a loop in your code you didn't know about. :)

If you can't tell us the why, can you tell us what reasonable utilization numbers are for a range of numbers of zIIPs? I think the user community as a whole is interested.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: ***@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Jim Mulder
Sent: Saturday, June 9, 2018 12:53 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?

zIIP dispatching is the same as GP dispatching. ZIIPAWMT has analogous parameters for GP (CCCAWMT) and zAAP (ZAAPAWMT).
Alternate wait management was created long before there were specialty engines.

Jim Mulder z/OS Diagnosis, Design, Development, Test IBM Corp.
Poughkeepsie NY
Date: 06/09/2018 12:46 PM
Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?
Peter,
I started off as saying, a lot of the descriptions are based on
assumptions, as IBM has let out little on how the zIIP dispatcher works.
Also, I was only talking about SRBs on zIIPs, so non-enclave SRBs were
not part of the discussion.
I believe hyper dispatch is very different from zIIP dispatch. I
stand by my assumption that GP dispatch is very different from zIIP
dispatch, or why would there be the ZIIPAWMT parameter and have the
comment about waking up after that interval to see if there is work.
When non-zIIP work comes ready and a processor is free, the work is
dispatched. No waiting. The comments in the parameter description
for ZIIPAWMT is describing a polling environment, at least to me.
I have looked through the internet, OK not the be-all and end-all but
a reasonable place to start, and there is a lot on what can get
dispatched on a zIIP, but no detail on how.
As I said, I wish someone from IBM would at least chime in with some
level of description as to how zIIP dispatch works and why high zIIP
utilization rates are, shall we say, not good. Then maybe we can stop
guessing.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-10 07:33:41 UTC
Permalink
Post by Jim Mulder
zIIP dispatching is the same as GP dispatching. ZIIPAWMT
has analogous parameters for GP (CCCAWMT) and zAAP (ZAAPAWMT).
Alternate wait management was created long before there were
specialty engines.



Thank, Jim, much appreciated.
Sorry, guys, for not reading the latest posts before writing mine talking about CCCAWMT....


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Joel C. Ewing
2018-06-09 13:26:06 UTC
Permalink
That is consistent with Ed Jaffe's explanation of there being a
short-period (4µsec), periodic millicode CPU loop to eat up the excess,
un-bought CP capacity.  That would have the effect of making a program
running on a subcapacity CP run slower and longer with a lower average
instructions/sec rate than on a full speed CP, even though individual
instructions when doing productive work would run at full clock speed.  

With a reduced effective execution rate on a subcapacity CP, the balance
between CPU and I/O speeds  is shifted, and one would expect reduced I/O
wait time when running on the effectively-slower CP.
    Joel C. Ewing
Post by Mike Schwab
I have seen press stories where Full speed CPs see a lot of wait time
(waiting for data, interrupt, etc.), where slower CPs credit the
delays to the reduced capacity so they will see little to no wait
time.
On Fri, Jun 8, 2018 at 4:48 PM Christopher Y. Blaicher
Post by Christopher Y. Blaicher
I wish Peter Relson would comment on this so we all can get the straight answer, but he may not be able to.
From what little I know and most that I summarize and guess at, it seems the following to be what is happening.
First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
I think the queuing process is basically FIFO and once unit of work is dispatched, it runs to the end or a program break point happens. A program break point could be some form of PAUSE or a IEAVXTSW wait.
Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.
There is the ZIIPAWMT parameter in IEAOPTxx which affects how often idle zIIPs get woken up to see if there is work. This has two effects. A unit of work put on the zIIP queue has to wait up to that long to get dispatched on a zIIP, or if at the end of that interval all zIIPs are busy, only then will the unit of work be dispatched on a GP. To be fair, when a zIIP completes a unit of work, it checks for more work to process, so not everything waits a dispatch cycle.
So what does this all mean? In a low activity environment, your mean time to dispatch is around half the ZIIPAWMT time. In a high activity environment it means the mean time to dispatch on a GP because all he zIIPs are busy is ZIIPAWMT.
This starts to get into queuing theory, but if a single zIIP processor is over 30% busy, about 30% or more of the work to run on a zIIP will wind up waiting the ZIIPAWMT time and then get dispatched on a GP, thus adding the insult of delaying the processing of the work to the injury of running on a GP that can cost real money.
If you add more zIIP engines, there is a greater chance of a unit of work ending on one of the engines and thus the work getting dispatched on a zIIP without waiting the full ZIIPAWMT.
A reasonable set of indicators can be gotten from the SMF type 30 record. The SMF30_TIME_ON_ZIIP and SMF30_TIME_ZIIP_ON_CP are valuable indicators.
If SMF30_TIME_ZIIP_ON_CP is 30% or more of SMF30_TIME_ON_ZIIP, then you probably need another zIIP engine. Because type 30 records are job centric, I would suggest using the SMF 30 SUBTYPE 2 and 3 records for all the jobs in a time interval and totaling the data. Using data from a single job or group of jobs may not provide a complete picture. Also, the 30% value mentioned maybe more or less than your environment can tolerate. YMMV, so only use it as a starting point.
As I said, most of this is just a guess at what is happening with zIIP dispatching. None of this information has been validated and neither Syncsort or I are implying or stating any course of action a user should pursue. Each user must evaluate their own needs and requirements and make decisions based solely on their own research and evaluation of their circumstances.
Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com
Data quality leader Trillium Software is now a part of Syncsort.
-----Original Message-----
Sent: Friday, June 8, 2018 3:42 PM
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?
Post by Parwez Hamid
How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines?
I haven't got the faintest idea. We do, but that doesn't matter for this discussion. I thought this is complex enough, so I take one part of complexity out first: Different speed engines.
--
Peter Hunkeler
--
Joel C. Ewing, Bentonville, AR ***@acm.org

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Parwez Hamid
2018-06-08 07:50:28 UTC
Permalink
Re the comment:

How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines? In other words, Is it that valid to assume equal speed processors? Clearly guidelines for lower zIIP utilization matter more when there is a difference, as offloading any zIIP work to a slower CP would elongate processing and response time, even if there is no delay waiting for a processor.

For any given Z system, the 'speed' (GHz) is always the same for all types to (PUs) processing units (CPs, IFLs, zIIPs, ICFs, IFP and SAP). In case of CPs, these can be different 'sub capacity settings' All the rest are always full capacity setting. Speed is always the same.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Scott Chapman
2018-06-08 12:55:55 UTC
Permalink
This seems to come up a lot.

I'm going to start by taking the opposite tack: you probably shouldn't run your GCPs at 90-100% busy either. Busier CPUs are generally going to have more cache contention which means the work is generally going to run "somewhat" less efficiently (i.e. more CPU time / unit of work) than if the CPUs were running less busy. Apparently some IBMer at some point measured that as a 4% increase in CPU time / unit of work per 10% increase in overall utilization. Alas I have no further details about the source of that number. I believe that it's at least directionally correct though.

Now if you have a mix of different importances / priorities for the work that is driving the machine to 100% busy, then likely the most (but not only) impacted work is the lower importance work. So maybe that's ok. But, in all likelihood, if the machine had more capacity and was running at only say 70% busy, then likely the same work would consume fewer MSUs. Which may be a good thing.

From purely a performance perspective, running less busy is always better as there's less chance for queueing for a processor. But rarely is "as fast as possible" the required and most cost effective answer.

But this question is about zIIPs. But zIIPs are the same processors as GCPs and the aforementioned discussion is mostly the same: you can run the zIIPs busy, but things may not be as efficient. Of course "less efficient" matters a little less on the zIIPs given that the hardware is cheaper and they don't increase software costs.

The primary issue as I see it comes in where the zIIPs are running busier than the GCPs and so work is more delayed trying to get through the zIIP queue than if they had been just dispatched directly to the GCP. If that work is very important work (such as perhaps DB2 system tasks) then that could have relatively widespread negative impacts. Are those impacts greater than if there were no zIIPs in the configuration at all and the GCPs were running a similarly busy level? Maybe a bit due to the extra overhead of (unsuccessfully) scheduling work onto the zIIP.

But I believe the potential for harm is very situation dependent. In particular with the way the rules are today, and with the mix of the different workloads that are zIIP eligible, if you have zIIP capacity (both in terms of MIPS and engines) greater or equal to your GCP capacity, I'm hard pressed to believe that there's a significant risk to running the zIIPs as busy as you're comfortable running your GCPs. But my belief is also that many are comfortable running their GCPs hotter than is really ideal.

In Kathy Walsh's presentation from Orlando one of her slides has the statement: "Can run zIIPs very busy IF there are multiple classes of work with different response time objectives, but watch IIPCP time". I think that's a very reasonable statement. If you're starting to see significant crossover from the zIIPs to the GCPs, you're probably running the zIIPs too busy for that particular workload. Note that the crossover amounts are not necessarily well-correlated with zIIP busy, although generally when zIIPs are busy they are more likely to see significant crossover.

A potential issue is that some systems only have a single zIIP. When you have a single CP, your threshold for where you'll start feeling pain is significantly lower vs. having evern 2 CPs. The situation of having a 710 with a single zIIP is a significantly different situation than having a say a 505 with 4 zIIPs. I'm going to be a lot more concerned about zIIP utilization in the former vs. the latter. In the former, that zIIP very well might become a bottleneck that could be more problematic than not having a zIIP at all. That would seem to be much less likely in the second case.

My personal belief is that given the amount of work that's zIIP eligible today, most systems should have at least 2 zIIPs, and ideally the system should have zIIP capacity at least similar to the GCP capacity. Yes, there's a hardware cost to that, but in the grand scheme of things, the costs are not nearly as significant as GCP capacity, so err on the side of having too much zIIP capacity. That would be an interesting study: what's common ratio of zIIP capacity to GCP capacity? I suspect that that ratio has been creeping up over the years.

Scott Chapman

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Joel C. Ewing
2018-06-08 21:06:41 UTC
Permalink
Post by Parwez Hamid
How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines? In other words, Is it that valid to assume equal speed processors? Clearly guidelines for lower zIIP utilization matter more when there is a difference, as offloading any zIIP work to a slower CP would elongate processing and response time, even if there is no delay waiting for a processor.
For any given Z system, the 'speed' (GHz) is always the same for all types to (PUs) processing units (CPs, IFLs, zIIPs, ICFs, IFP and SAP). In case of CPs, these can be different 'sub capacity settings' All the rest are always full capacity setting. Speed is always the same.
I can't find a decent explanation of how the sub capacity enforcement on
a CP is managed beyond "the clock frequency... remains unchanged ...
adjustment is achieved through other means".  That statement doesn't
actually say that instruction execution speed in unaffected, only that
clock frequency is constant:  one way to enforce 50% capacity at the
same clock frequency would be the old IBM 407 approach of running at
full speed but doing  no productive work on half the clock cycles;  but,
considering all the parallelism in today's processors, I suspect it
would be much simpler to just force the entire CP to be idle or
non-dispatchable for an interval of time when the utilization exceeds
allowable levels. 

Would be curious if anyone has seen a more complete description about
how the sub-capacity CP enforcement works. 
Post by Parwez Hamid
.... the workload on the CP is totally different.
Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?
--
Peter Hunkeler
The zIIP-eligible criteria for choosing a subset of  tasks to run on
zIIP engines, as I understand it, has nothing to do with installation
defined service classes but is totally  based on IBM marketing
strategy.  There is no reason to expect the mix of tasks eligible for
zIIP resources to have the same service-class mix and CPU/IO usage
patterns as those restricted at that same time to CP resources --the
zIIP utilization may even peak at a totally different time of day.  The
total system workload may be the same as before things were made zIIP 
eligible, but with artificial separation into those that prefer a zIIP
and those that must run on a CP, that workload is now artificially
subdivided into two distinct and different workload subsets when
competing for CPU resources.  If zIIP utilization forces something that
would normally run on zIIP onto a CP, it is now competing for CPU
resources with a different subset of that total workload, and it would
be surprising if that shift didn't affect response time.
    Joel C. Ewing
--
Joel C. Ewing, Bentonville, AR ***@acm.org

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Ed Jaffe
2018-06-08 21:19:29 UTC
Permalink
Post by Joel C. Ewing
Would be curious if anyone has seen a more complete description about
how the sub-capacity CP enforcement works.
Every so often (I believe 4 usec on my z13s), there is something akin to
a timer interrupt in the chip that runs a millicode housekeeping
routine. At the end of that routine, before returning to running normal
instructions, a tight CPU loop is executed to burn up any excess
capacity you didn't pay for.
--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
http://www.phoenixsoftware.com/
--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-09 06:36:56 UTC
Permalink
Post by Peter Hunkeler
Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?
--
Peter Hunkeler
The zIIP-eligible criteria for choosing a subset of tasks to run on
zIIP engines, as I understand it, has nothing to do with installation
defined service classes but is totally based on IBM marketing
strategy. There is no reason to expect the mix of tasks eligible for
zIIP resources to have the same service-class mix and CPU/IO usage
patterns as those restricted at that same time to CP resources --the
zIIP utilization may even peak at a totally different time of day. The
total system workload may be the same as before things were made zIIP
eligible, but with artificial separation into those that prefer a zIIP
and those that must run on a CP, that workload is now artificially
subdivided into two distinct and different workload subsets when
competing for CPU resources. If zIIP utilization forces something that
would normally run on zIIP onto a CP, it is now competing for CPU
resources with a different subset of that total workload, and it would
be surprising if that shift didn't affect response time.




Didn't think that way when I read your previous statement. I now see what you meant and I agree. Very good point. Thanks.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Relson
2018-06-10 01:31:03 UTC
Permalink
Post by Parwez Hamid
the 'speed' (GHz) is always the same for all types
The 'speed' is not generally thought to be the cycle time. It is true that
the cycle time is the same across the models, but the effective speed is
not the same (such as MIPS or the MSU rating).
Post by Parwez Hamid
the dispatcher code for ZIIP processing is not the same as the GP
dispatcher.

Well, it is "the same" but there are swizzles (such as the fact that a
zIIP processor will not run CP work, but a CP might run zIIP work).
Post by Parwez Hamid
the queuing process is basically FIFO
The queueing process is the same. For all processors it can be thought of
as FIFO for a specific priority (where the priority has many sub-factors)
Post by Parwez Hamid
It runs to the end or a program break point happens. A program break
point could be some form of PAUSE or a IEAVXTSW wait.

zIIP work is timed just like CP work. There are dispatch durations
established and when the duration is exceeded, the timeslice ends and the
work will be undispatched if there is equal or higher priority work to
run. That applies regardless of the type of processor. It might be, but is
not necessarily, the case that the dispatch durations are the same for the
different processor types.

You really want Gary King or Dan Rosa to chime in, not me. I can rarely
keep the details straight. If they provide me with information to relay on
their behalf, I will be glad to.

Much of the discussion has been on target. If zIIPs are "overloaded" then
the work queues for zIIP will get "too long" and the zIIPs will ask for
"help". The percentages mentioned to some extent translate into how likely
it is that something will not find the resources available and then must
"wait" (or get someone else to do it). "Help" is a very complex process
which can include helping from other zIIPs (in part related to
hiperdispatch trying to run work as close as possible to the original work
to limit negative cache effects) and possibly CPs.

If you're running CPs at 100% and zIIPs need "help", then something is
going to suffer. You just don't have enough resources for the work you're
trying to accomplish.
Then there are also the questions of whether or not you allow CPs to help
or insist that zIIP-eligible work run only on zIIPs.

Peter Relson
z/OS Core Technology Design


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Peter Hunkeler
2018-06-10 07:41:35 UTC
Permalink
Post by Peter Relson
You really want Gary King or Dan Rosa to chime in, not me. I can rarely
keep the details straight. If they provide me with information to relay on
their behalf, I will be glad to.



Thanks, for the offer. How about forwarding my initial post. I had written the points of interest to me, and I think to many here.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN
Loading...