GDG +1 dynamic allocation collision between two concurrent jobs

[Kirk Wolf]

Is there a way around this?

Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.

For example, from two different jobs:

(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)

(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"

BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
IEFA110I DATA SET CONTENTION
...

Kirk Wolf
Dovetailed Technologies
http://dovetail.com

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.
(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)
(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
IEFA110I DATA SET CONTENTION

It feels as if Allocation fails to perform some needed serialization.

Might this be APARable?

How large is the hazard window? Does it extend beyond the activity of SVC 99?

Is it holding an ENQ on the GDG base even after allocating the new generation?

Do batch job behave similarly? I suppose one might wait for initiation until
the earlier one DEQs.

Why are you doing this? It looks like a race where it doesn't matter who wins.

Does a similar problem exist for generated DDNAMEs ("SYSnnnn") among
multiple concurrent tasks in a single job?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Kirk Wolf]

On Tue, Sep 12, 2017 at 12:24 PM, Paul Gilmartin <

Post by Paul Gilmartin
Is it holding an ENQ on the GDG base even after allocating the new generation?

Yes (and that's what the IKJ56225I indicates). Note that the first job
allocated a +1, but hasn't freed it yet.

Post by Paul Gilmartin
Why are you doing this? It looks like a race where it doesn't matter who wins.

Two concurrent non-batch jobs want to create new generations.
In particular, this happens between two file transfer address spaces (one
for each concurrent session), so the data set (generation) is allocated
during transfer.

In my example above, the first job's allocation succeeded and cataloged a
new goovoo (at allocation).

Wouldn't it be nice if the second job's allocation just got the next
goovoo?
Of course, you could end of with a hole in the goovoo sequence if the first
job failed w/ CONDISP=DELETE. Maybe that is what the current GDG design
is trying to protect against against. Gee, thanks. Maybe that will be a
feature in "Extended GDGs(+1)" ;-)

Post by Paul Gilmartin
Does a similar problem exist for generated DDNAMEs ("SYSnnnn") among
multiple concurrent tasks in a single job?
-- gil
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Steve Thompson]

The way I read this, z/UNIX is enqueing on the GDG base while it
does its work. And I can't tell from what is provided how long it
holds that before it frees it.

Regards,
Steve Thompson

Post by Paul Gilmartin

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.
(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)
(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
IEFA110I DATA SET CONTENTION

It feels as if Allocation fails to perform some needed serialization.
Might this be APARable?
How large is the hazard window? Does it extend beyond the activity of SVC 99?
Is it holding an ENQ on the GDG base even after allocating the new generation?
Do batch job behave similarly? I suppose one might wait for initiation until
the earlier one DEQs.
Why are you doing this? It looks like a race where it doesn't matter who wins.
Does a similar problem exist for generated DDNAMEs ("SYSnnnn") among
multiple concurrent tasks in a single job?
-- gil
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Kirk Wolf]

bpxwdyn() is just a text-interface to SVC99. I don't believe that z/OS
Unix System Services is related at all.

Kirk Wolf
Dovetailed Technologies
http://dovetail.com

The way I read this, z/UNIX is enqueing on the GDG base while it does its
work. And I can't tell from what is provided how long it holds that before
it frees it.
Regards,
Steve Thompson

Post by Kirk Wolf
Is there a way around this?

Post by Kirk Wolf
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.
(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)
(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
IEFA110I DATA SET CONTENTION
It feels as if Allocation fails to perform some needed serialization.

Might this be APARable?
How large is the hazard window? Does it extend beyond the activity of SVC 99?
Is it holding an ENQ on the GDG base even after allocating the new generation?
Do batch job behave similarly? I suppose one might wait for initiation until
the earlier one DEQs.
Why are you doing this? It looks like a race where it doesn't matter who wins.
Does a similar problem exist for generated DDNAMEs ("SYSnnnn") among
multiple concurrent tasks in a single job?
-- gil
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Andrew Rowley]

Post by Paul Gilmartin
It feels as if Allocation fails to perform some needed serialization.

I think this is the serialization.

Post by Paul Gilmartin
Do batch job behave similarly? I suppose one might wait for initiation until
the earlier one DEQs.

Careful, you might find a reason to like JCL!

Yes, JCL ensures the resources are available before the job starts (but
not before initiation). IMHO JCL and dataset serialization is one of the
key but under-appreciated features of z/OS. It's always frustrating on
other platforms where you have to manage serialization yourself, and
hope that everyone else does it the same way.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Peter Hunkeler]

Post by Andrew Rowley
Yes, JCL ensures the resources are available before the job starts (but

not before initiation).



I don't understand what you mean by "before the job starts but not before initiation".


The initiator enqueues on all data sets referred to in the job (all steps),before it starts the allocation phase for the first step (is this phase what you call "initiation"?). The initiator does not verify, if the data sets exist until allocation, so (in aJES2 environment), there is no guarantee. This is why you can get "JCL Error" after the job was started when a data set is not found or a new allocation fails (typically for duplicate name error).


JES3 will (may?) verify before it hands over the job to the initiator.

[Andrew Rowley]

Post by Peter Hunkeler
I don't understand what you mean by "before the job starts but not before initiation".

I was thinking of the assignment to an initiator as initiation, but you
are right, that is backwards and probably the wrong terminology. I guess
initiation is what is done by the initiator, so what I should have said
is something like "not before being assigned to an initiator".

[Allan Staller]

Nope. Not with "basic" gdg's.
I haven't read upi on GDG V2 ("extended GDG's") so I can't say.

The issue is using relative GDG numbers (+1) as opposed to absolute ....GnnnnV00

HTH,

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Kirk Wolf
Sent: Tuesday, September 12, 2017 11:54 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: GDG +1 dynamic allocation collision between two concurrent jobs

Is there a way around this?

Even with SMS-managed GDGs, where the generation is cataloged at allocation, two jobs that dynamically allocate +1 generations (with GDGNT) will collide.

For example, from two different jobs:

(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)

(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new catalog gdgnt lrecl(1028)"

BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+ IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER IEFA110I DATA SET CONTENTION ...

Kirk Wolf
Dovetailed Technologies
http://dovetail.com

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Ed Jaffe]

Post by Kirk Wolf
Is there a way around this?

Serialize against some existing data set with DISP=OLD
--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
http://www.phoenixsoftware.com/

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Tom Marchant]

On Tue, 12 Sep 2017 12:24:1

Post by Paul Gilmartin
Does a similar problem exist for generated DDNAMEs ("SYSnnnn") among
multiple concurrent tasks in a single job?

Not likely. Allocation within an address space is serialized with SYSZTIOT.

[Paul Gilmartin]

Post by Ed Jaffe

Post by Kirk Wolf
Is there a way around this?

Serialize against some existing data set with DISP=OLD

For how long? From ALLOCATE through FREE? Longer? Shorter?

But doesn't this vitiate the value of concurrency that Kirk mentioned
in his original statement of his problem?

Does BPXWDYN( 'alloc rtdsn(name) ...' ) return the ...G00V00 value in the
returned DSN? If so, FREE the DDNAME and reallocate with the G00V00 string.
There'd still be a window for collision which a spin loop could handle.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Binyamin Dissen]

WAD.

What if job #1 now allocates +2?

+1 remains that thru the job.

On Tue, 12 Sep 2017 11:54:04 -0500 Kirk Wolf <***@DOVETAIL.COM> wrote:

:>Is there a way around this?
:>
:>Even with SMS-managed GDGs, where the generation is cataloged at
:>allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
:>will collide.
:>
:>For example, from two different jobs:
:>
:>(job 1)
:>./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
:>catalog gdgnt lrecl(1028)"
:>BPXWDYN RC= 0 (0x0)
:>
:>(job 2)
:>./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
:>catalog gdgnt lrecl(1028)"
:>
:>BPXWDYN RC= 34603008 (0x2100000)
:>IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
:>IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
:>IEFA110I DATA SET CONTENTION
:>...
:>
:>Kirk Wolf
:>Dovetailed Technologies
:>http://dovetail.com
:>
:>----------------------------------------------------------------------
:>For IBM-MAIN subscribe / signoff / archive access instructions,
:>send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

--
Binyamin Dissen <***@dissensoftware.com>
http://www.dissensoftware.com

Director, Dissen Software, Bar & Grill - Israel


Should you use the mailblocks package and expect a response from me,
you should preauthorize the dissensoftware.com domain.

I very rarely bother responding to challenge/response systems,
especially those from irresponsible companies.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Binyamin Dissen
What if job #1 now allocates +2?

What does a relative generation number greater than +1 mean? Must
be something because the JCL Ref. says:

v The numeric portion (not + or -) of a relative generation number must be expressed
in 1 to 3 numeric characters. For example, +100, -002, +4, -09, 000.

Does allocating MANAGED.TEST.GDG(+100) leave a gap of 99 values in the
numbering? Or does it fill in the 99 not mentioned generations with new data sets?

Post by Binyamin Dissen
+1 remains that thru the job.
:>
:>BPXWDYN RC= 34603008 (0x2100000)
:>IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
:>IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
:>IEFA110I DATA SET CONTENTION
:>

I agree with Kirk's surmise that the ENQ conflict is on the base, not on
the generation, so +2 is probably not a circumvention.

What does a Display GRS from the console or from DDLIST show?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Feller, Paul]

I think someone mentioned WAD. If I remember correctly when using the +1 the system does an enqueue on the BASE as part of the process. The enqueue is not release until the file is closed. I believe this is done to insure proper "roll in" of the new generation.

I believe this same behavior happens when you are reading a GDG by the +1 or +0 or whatever. Any new allocation of the GDG will fail or hang until the other job closes the file it is reading. For reads I have gotten around this by doing the allocation using the full dataset name.

Thanks..

Paul Feller
AGT Mainframe Technical Support

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin
Sent: Tuesday, September 12, 2017 15:08
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: GDG +1 dynamic allocation collision between two concurrent jobs

Post by Binyamin Dissen
What if job #1 now allocates +2?

What does a relative generation number greater than +1 mean? Must
be something because the JCL Ref. says:

v The numeric portion (not + or -) of a relative generation number must be expressed
in 1 to 3 numeric characters. For example, +100, -002, +4, -09, 000.

Does allocating MANAGED.TEST.GDG(+100) leave a gap of 99 values in the
numbering? Or does it fill in the 99 not mentioned generations with new data sets?

Post by Binyamin Dissen
+1 remains that thru the job.
:>
:>BPXWDYN RC= 34603008 (0x2100000)
:>IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
:>IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
:>IEFA110I DATA SET CONTENTION
:>

I agree with Kirk's surmise that the ENQ conflict is on the base, not on
the generation, so +2 is probably not a circumvention.

What does a Display GRS from the console or from DDLIST show?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Binyamin Dissen]

On Tue, 12 Sep 2017 15:08:15 -0500 Paul Gilmartin
<0000000433f07816-dmarc-***@LISTSERV.UA.EDU> wrote:

:>On Tue, 12 Sep 2017 22:42:47 +0300, Binyamin Dissen wrote:

:>>What if job #1 now allocates +2?

:>What does a relative generation number greater than +1 mean? Must
:>be something because the JCL Ref. says:

+2 means 2 more than what it was at the time the GDG was allocated.

+1 remains +1 (even if freed and reallocated) for the life of the job.

:> v The numeric portion (not + or -) of a relative generation number must be expressed
:> in 1 to 3 numeric characters. For example, +100, -002, +4, -09, 000.

:>Does allocating MANAGED.TEST.GDG(+100) leave a gap of 99 values in the
:>numbering? Or does it fill in the 99 not mentioned generations with new data sets?

:>>+1 remains that thru the job.

:>>On Tue, 12 Sep 2017 11:54:04 -0500 Kirk Wolf wrote:

:>>:>BPXWDYN RC= 34603008 (0x2100000)
:>>:>IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
:>>:>IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
:>>:>IEFA110I DATA SET CONTENTION

:>I agree with Kirk's surmise that the ENQ conflict is on the base, not on
:>the generation, so +2 is probably not a circumvention.

:>What does a Display GRS from the console or from DDLIST show?

:>-- gil

--
Binyamin Dissen <***@dissensoftware.com>
http://www.dissensoftware.com

Director, Dissen Software, Bar & Grill - Israel


Should you use the mailblocks package and expect a response from me,
you should preauthorize the dissensoftware.com domain.

I very rarely bother responding to challenge/response systems,
especially those from irresponsible companies.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Tom Marchant]

Post by Paul Gilmartin

Post by Binyamin Dissen
What if job #1 now allocates +2?

What does a relative generation number greater than +1 mean?

If you create a Generation Data Set with JCL using +1 in one step,
and want to refer to it in a subsequent step in the job, you
reference it as +1. If you want to create another GDS for the same
GDG, you would specify +2.

In a subsequent job, assuming no other GDS created for that GDG,
these same data sets would be -2 and -1.

[Paul Gilmartin]

Post by Tom Marchant

Post by Paul Gilmartin
What does a relative generation number greater than +1 mean?

If you create a Generation Data Set with JCL using +1 in one step,
and want to refer to it in a subsequent step in the job, you
reference it as +1. If you want to create another GDS for the same
GDG, you would specify +2.

OK.

If a job uses +2 but never mentions +1, is an error reported?
What if the step that creates +1 is skipped by COND or IF?

Post by Tom Marchant
In a subsequent job, assuming no other GDS created for that GDG,
these same data sets would be -2 and -1.

Then I'd have expected -1 and 0. But I'm often wrong. Fencepost errors and all.

Post by Tom Marchant
I think someone mentioned WAD. If I remember correctly when using the +1 the system does an enqueue on the BASE as part of the process. The enqueue is not release until the file is closed. I believe this is done to insure proper "roll in" of the new generation.

"CLOSE"d or "FREE"d?

Post by Tom Marchant
I believe this same behavior happens when you are reading a GDG by the +1 or +0 or whatever. Any new allocation of the GDG will fail or hang until the other job closes the file it is reading. For reads I have gotten around this by doing the allocation using the full dataset name.

Would that not also work for writes? Or can a generation be written only once?

The OP mentioned BPXWDYN. Can the "full data[ ]set name" be found by
BPXWDYN(INFO ...) or BPXWDYN( ALLOC RTDSN(...) ...)?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Feller, Paul]

Paul.. I've never paid that close of attention to say if it is "CLOSE"D or "FREE"D. All I remember is when the job that was creating a new generation completed other jobs could "access" the GDG by the relative generation.

As for writes/creation I never tried to use the full name. I only did it for reads. I'm not sure how the system would handle the roll in process.

Thanks..

Paul Feller
AGT Mainframe Technical Support

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin
Sent: Tuesday, September 12, 2017 16:07
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: GDG +1 dynamic allocation collision between two concurrent jobs

Post by Tom Marchant

Post by Paul Gilmartin
What does a relative generation number greater than +1 mean?

If you create a Generation Data Set with JCL using +1 in one step,
and want to refer to it in a subsequent step in the job, you
reference it as +1. If you want to create another GDS for the same
GDG, you would specify +2.

OK.

If a job uses +2 but never mentions +1, is an error reported?
What if the step that creates +1 is skipped by COND or IF?

Post by Tom Marchant
In a subsequent job, assuming no other GDS created for that GDG,
these same data sets would be -2 and -1.

Then I'd have expected -1 and 0. But I'm often wrong. Fencepost errors and all.

Post by Tom Marchant
I think someone mentioned WAD. If I remember correctly when using the +1 the system does an enqueue on the BASE as part of the process. The enqueue is not release until the file is closed. I believe this is done to insure proper "roll in" of the new generation.

"CLOSE"d or "FREE"d?

Post by Tom Marchant
I believe this same behavior happens when you are reading a GDG by the +1 or +0 or whatever. Any new allocation of the GDG will fail or hang until the other job closes the file it is reading. For reads I have gotten around this by doing the allocation using the full dataset name.

Would that not also work for writes? Or can a generation be written only once?

The OP mentioned BPXWDYN. Can the "full data[ ]set name" be found by
BPXWDYN(INFO ...) or BPXWDYN( ALLOC RTDSN(...) ...)?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Peter Hunkeler]

Post by Feller, Paul
I've never paid that close of attention to say if it is "CLOSE"D or "FREE"D. All I remember is when the job that was creating a new generation completed other jobs could "access" the GDG by the relative generation.

Remember, it is the Initiator which does the ENQ/DEQ at job step allocation and unallocation, resp. So I guess it is the Initiator enqueueing on the GDG base(s) when GDSs are being used in the job. So, I further guess that the Initiator will dequeue the GDG base at the end of the last step referring to a GDS of that GDG. So it's not a question of neither CLOSE nor FREE,

Post by Feller, Paul
As for writes/creation I never tried to use the full name. I only did it for reads. I'm not sure how the system would handle the roll in process.

I would have to verify, but when you create a data set using the full name, it is just a data set that happens to have name that looks like a GDS name. The system will not relate it to the GDG.

[Peter Hunkeler]

Post by Paul Gilmartin

Post by Tom Marchant
If you create a Generation Data Set with JCL using +1 in one step,
and want to refer to it in a subsequent step in the job, you
reference it as +1. If you want to create another GDS for the same
GDG, you would specify +2.

OK.
If a job uses +2 but never mentions +1, is an error reported?
What if the step that creates +1 is skipped by COND or IF?

The FM says: "Though any positive relative generation number can be used, a number greater than 1 can cause absolute generation numbers to be skipped. For example, if you have a single step job, and the generation being cataloged is a +2, one generation number is skipped. However, in a multiple-step job, one step might have a +1 and a second step a +2, in which case no numbers are skipped."


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Binyamin Dissen
+2 means 2 more than what it was at the time the GDG was allocated.
+1 remains +1 (even if freed and reallocated) for the life of the job.

Do successive Rexx EXECs run under interactive TSO count separate jobs
or as a single job? Is there any way the second EXEC can say that it
really, really wants to create a new generation, not to reuse the generation
created by a prior EXEC?

What about under PGM=IKJEFT01?

What about under PGM=IRXJCL?

What about under PGM=BPXBATCH?

What about under UNIX with BPX_SHAREAS=MUST?

What about under UNIX with BPX_SHAREAS=YES?
(That may be unpredictable.)

What about under UNIX with BPX_SHAREAS=NO?

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Andrew Rowley
Careful, you might find a reason to like JCL!

Yeah, right.

Post by Andrew Rowley
Yes, JCL ensures the resources are available before the job starts (but
not before initiation). IMHO JCL and dataset serialization is one of the
key but under-appreciated features of z/OS. It's always frustrating on
other platforms where you have to manage serialization yourself, and
hope that everyone else does it the same way.

It comes at a price. Consider the gyrations to create a new res pack,
ALLOCATE SYS1.LINKLIB vol(newres) NEW ...
I know IBM has lately provided a circumvention, but it's so dicey that
it requires confirmation from a console.

Here's my current *untested* suggestion for Kirk's original problem:
/* Rexx */ signal on novalue; /* ***************************
*/
do Success = 1 for 1
do Failure = 1 /* Try several times if ENQ EXC. */
RC = BPXWDYN( "alloc rtdsn(GDG1) fi(mydd) da('managed.test.gdg(+1)') recfm(v,b)" ,
"new catalog gdgnt lrecl(1028) msg(S99MSG.)" )
if RC==0 then leave Success
if Failure>=10 then leave Failure /* Give up on too many failures. */
address SYSCALL "sleep 1" /* Try again later. */
end Failure /* Fall through after too many tries. */
do I = 1 to S99MSG.0 /* Allocation failed. */
say S99MSG.I; end
exit 99 /* Recovery action? */
end Success /* Free GDG base and allocate generation by full DSN. */
RC = BPXWDYN( "free fi(mydd)" )
RC = BPXWDYN( "alloc fi(mydd) da('"GDG1"') old msg(S99MSG.)" )

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Peter Hunkeler]

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.

Not according to the FM "DFSMS Using Data Sets", which says under "Submitting Multiple Jobs to Update a Generation Data Group":


This topic provides guidelines that you can use when you submit multiple jobs that update a particular GDG:
- No two jobs running concurrently can refer to the same GDG.
- ...


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Peter Hunkeler]

Sorry, sending out too early. I repost with the full list of guidelines

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.

Not according to the FM "DFSMS Using Data Sets", which says under "Submitting Multiple Jobs to Update a Generation Data Group": Submitting Multiple Jobs to Update a Generation Data Group
This topic provides guidelines that you can use when you submit multiple jobs that update a particular GDG:
- No two jobs running concurrently can refer to the same GDG.
- For batch or dynamic allocation jobs that specify relative generation numbers, the system enqueues the GDG base name as shared or exclusive, depending on the highest disposition that is used in the job. The GDG base name is exclusive if the highest job disposition is NEW or MOD. The GDG base name is shared if the highest job disposition is SHR. This safeguard prevents concurrent users from updating the GDG by adding or deleting generation data sets while other users are using the GDG.
- For batch or dynamic allocation jobs that use absolute generation data setnames, the system does not enqueue the GDG base. Multiple users are able to update the GDG by deleting or adding generation data sets at the same time. This situation does not affect the integrity of the GDG or generation data sets. However, jobs that use relative generation numbers might obtain the wrong generation, because the numbers can change. Even if you use absolute generation numbers, a job might accidentally replace a generation data set that another job is using.
The only time that you can use absolute generation numbers is when you need to run concurrent jobs that use the same GDG and at least one of the jobs uses a disposition of NEW or MOD. Ensure that the jobs do not accidentally overlay a generation data set that another job is using.
Restriction: Be careful when you update GDGs because two or more jobs can compete for the same resource and accidentally replace the generation data set with the wrong version in the GDG. To prevent two users from allocating the same absolute generation data set, take one of the following actions:
o Specify DISP=OLD.
o Specify DISP=SHR and open the data set for output.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Peter Hunkeler]

I think the main problem here is dynamic allocation (SVC99) is not allowing the unauthorized caller to specify it should enqueue uncondtionally. For that reason BPXWDYN, TSO ALLOCATE, etc return with "data set in use" instead of offering the option to wait until it becomes available.

--Peter Hunkeler


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Allan Staller]

I wonder how this plays with the new (z/OS 2.2?) enqueue demotion?

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler
Sent: Wednesday, September 13, 2017 12:18 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: AW: GDG +1 dynamic allocation collision between two concurrent jobs

Sorry, sending out too early. I repost with the full list of guidelines

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with
GDGNT) will collide.

Not according to the FM "DFSMS Using Data Sets", which says under "Submitting Multiple Jobs to Update a Generation Data Group": Submitting Multiple Jobs to Update a Generation Data Group This topic provides guidelines that you can use when you submit multiple jobs that update a particular GDG:
- No two jobs running concurrently can refer to the same GDG.
- For batch or dynamic allocation jobs that specify relative generation numbers, the system enqueues the GDG base name as shared or exclusive, depending on the highest disposition that is used in the job. The GDG base name is exclusive if the highest job disposition is NEW or MOD. The GDG base name is shared if the highest job disposition is SHR. This safeguard prevents concurrent users from updating the GDG by adding or deleting generation data sets while other users are using the GDG.
- For batch or dynamic allocation jobs that use absolute generation data setnames, the system does not enqueue the GDG base. Multiple users are able to update the GDG by deleting or adding generation data sets at the same time. This situation does not affect the integrity of the GDG or generation data sets. However, jobs that use relative generation numbers might obtain the wrong generation, because the numbers can change. Even if you use absolute generation numbers, a job might accidentally replace a generation data set that another job is using.
The only time that you can use absolute generation numbers is when you need to run concurrent jobs that use the same GDG and at least one of the jobs uses a disposition of NEW or MOD. Ensure that the jobs do not accidentally overlay a generation data set that another job is using.
Restriction: Be careful when you update GDGs because two or more jobs can compete for the same resource and accidentally replace the generation data set with the wrong version in the GDG. To prevent two users from allocating the same absolute generation data set, take one of the following actions:
o Specify DISP=OLD.
o Specify DISP=SHR and open the data set for output.


--
Peter Hunkeler



----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Mike Schwab]

If you work with Relative GDG numbers, the whole GDG is enqued until
the job ends. Bypass by using a specific G####V00 in your JCL. Or a
different GDG base name.

Post by Kirk Wolf
Is there a way around this?
Even with SMS-managed GDGs, where the generation is cataloged at
allocation, two jobs that dynamically allocate +1 generations (with GDGNT)
will collide.
(job 1)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 0 (0x0)
(job 2)
./bpxwdyn.rexx "alloc fi(mydd) da('managed.test.gdg(+1)') recfm(v,b) new
catalog gdgnt lrecl(1028)"
BPXWDYN RC= 34603008 (0x2100000)
IKJ56225I DATA SET MANAGED.TEST.GDG ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
IEFA110I DATA SET CONTENTION
...
Kirk Wolf
Dovetailed Technologies
http://dovetail.com
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

[Tom Marchant]

Post by Paul Gilmartin

Post by Tom Marchant
In a subsequent job, assuming no other GDS created for that GDG,
these same data sets would be -2 and -1.

Then I'd have expected -1 and 0. But I'm often wrong. Fencepost errors and all.

Yes, you are correct.

[Giliad Wilf]

The behaviour I recall seeing in batch environment is that when a new
GDG dataset is being created, GDG base is being ENQ'ed exclusively.

For any other job attempting creation of another dataset per the same
GDG base - this base must be *IMMEDIATELY* available, or the second
request will be failed on some JCL error, not being waited upon for base
availability.

I think I've seen something like "DATA SET RESERVATION UNSUCCESSFUL",
or alike.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

Post by Peter Hunkeler
I think the main problem here is dynamic allocation (SVC99) is not allowing the unauthorized caller to specify it should enqueue uncondtionally. For that reason BPXWDYN, TSO ALLOCATE, etc return with "data set in use" instead of offering the option to wait until it becomes available.

That's allowed if the request specifies S99WTDSN. That, in turn, requires
APF authorization, assuming an authorized program can be trusted to
circumvent deadlocks.

-- gil

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Scott Ballentine]

I'll confirm what several others have suggested. In job 1, Allocation gets the ENQ on the GDG base name (MANAGED.TEST.GDG in this case) before allocating the generation data set. Since NEW was specified, the ENQ is obtained exclusive. We'll also get the ENQ for the GDS (MANAGED.TEST.GDG.GxxxxV00) exclusive, and we hold both until you unallocate it. Job 2 also tries to get the ENQ on the GDG base name, and since Job 1 holds it exclusive, you get S99ERROR 0210 (data set unavailable.)

Holding both ENQs keeps things somewhat "sane" - if multiple jobs are trying create or delete generations at the same time, then generations might not get rolled in properly or duplicate data sets can occur or "holes" can be created where some generations are missing.

Batch JCL will wait for ENQs, so if these DDs were coded in JCL, job 2 would end up waiting for job 1 to unallocate the GDG. Batch allocation has more information to work with when it comes to deadlock prevention than dynamic allocation, so dynamic allocation is a little different - you can wait if you ask for it (and are an authorized program), but waiting isn't the default. I'm not sure if BPXWDYN supports waiting for data sets, but I suspect it doesn't.

There's a few tricks you can try. Job 2 can have its own wait and retry logic (like others have already suggested.) Allocating the GDS using the ...GxxxxV00 name does not get the ENQ for the base name, so you could do something like allocate the (+1), get the ...GxxxxV00 name, unallocate it, and then reallocate with the GxxxxV00 name and write your data. That doesn't eliminate the window where the base name is held, but it could significantly shrink it. (There is a chance that somebody could play with your data set in the window where it is unallocated, so this might not be a viable option - that's for you to decide based on what your application is doing.)

-Scott Ballentine (***@us.ibm.com)
z/OS Device Allocation Development

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Kirk Wolf]

Post by Scott Ballentine
Holding both ENQs keeps things somewhat "sane" - if multiple jobs are
trying create or delete generations at the same time, then generations
might not get rolled in properly or duplicate data sets can occur or
"holes" can be created where some generations are missing.

I appreciate your response. I don't question that this is how it works,
but I don't really agree that this is the only sane implementation.

- "generations might not get rolled in properly"
this can happen in other ways, right?

- "duplicate data sets can occur"
With SMS? In this case the goovoo data set is cataloged at allocation
time and also ENQed

- "holes can be created where some generations are missing"
this can also happen in other ways.

"Extended GDG" support added new options, but IMO it is a pity that
concurrency was not considered as an option when creating new generations.
All of the suggested "tricks" to support this use case might lead one to
consider using something besides a GDG.


Kirk Wolf
Dovetailed Technologies
http://dovetail.com

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Allan Staller]

How do you propose to handle reference by relative generation if multiple entities can change relativity?



-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Kirk Wolf
Sent: Wednesday, September 13, 2017 12:25 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: GDG +1 dynamic allocation collision between two concurrent jobs

Post by Scott Ballentine
Holding both ENQs keeps things somewhat "sane" - if multiple jobs are
trying create or delete generations at the same time, then generations
might not get rolled in properly or duplicate data sets can occur or
"holes" can be created where some generations are missing.

I appreciate your response. I don't question that this is how it works, but I don't really agree that this is the only sane implementation.

- "generations might not get rolled in properly"
this can happen in other ways, right?

- "duplicate data sets can occur"
With SMS? In this case the goovoo data set is cataloged at allocation time and also ENQed

- "holes can be created where some generations are missing"
this can also happen in other ways.

"Extended GDG" support added new options, but IMO it is a pity that concurrency was not considered as an option when creating new generations.
All of the suggested "tricks" to support this use case might lead one to consider using something besides a GDG.


Kirk Wolf
Dovetailed Technologies
http://dovetail.com

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions, send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Paul Gilmartin]

This post might be inappropriate. Click to display it.

[Jesse 1 Robinson]

I learned early on as a programmer trainee the effects of DISP=NEW for GDG(+1). My first assignment was to analyze existing data in this GDG. +1 was always 'in creation' by an online system. In order to read a previous version, it was not sufficient to specify DISP=SHR for an older generation; the exclusive ENQ blocked any allocation by GDG base name. I had to specify an older data set by fully qualified name. Same restriction for batch and TSO.

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
***@sce.com


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Scott Ballentine
Sent: Wednesday, September 13, 2017 6:45 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: (External):Re: GDG +1 dynamic allocation collision between two concurrent jobs

I'll confirm what several others have suggested. In job 1, Allocation gets the ENQ on the GDG base name (MANAGED.TEST.GDG in this case) before allocating the generation data set. Since NEW was specified, the ENQ is obtained exclusive. We'll also get the ENQ for the GDS (MANAGED.TEST.GDG.GxxxxV00) exclusive, and we hold both until you unallocate it. Job 2 also tries to get the ENQ on the GDG base name, and since Job 1 holds it exclusive, you get S99ERROR 0210 (data set unavailable.)

Holding both ENQs keeps things somewhat "sane" - if multiple jobs are trying create or delete generations at the same time, then generations might not get rolled in properly or duplicate data sets can occur or "holes" can be created where some generations are missing.

Batch JCL will wait for ENQs, so if these DDs were coded in JCL, job 2 would end up waiting for job 1 to unallocate the GDG. Batch allocation has more information to work with when it comes to deadlock prevention than dynamic allocation, so dynamic allocation is a little different - you can wait if you ask for it (and are an authorized program), but waiting isn't the default. I'm not sure if BPXWDYN supports waiting for data sets, but I suspect it doesn't.

There's a few tricks you can try. Job 2 can have its own wait and retry logic (like others have already suggested.) Allocating the GDS using the ...GxxxxV00 name does not get the ENQ for the base name, so you could do something like allocate the (+1), get the ...GxxxxV00 name, unallocate it, and then reallocate with the GxxxxV00 name and write your data. That doesn't eliminate the window where the base name is held, but it could significantly shrink it. (There is a chance that somebody could play with your data set in the window where it is unallocated, so this might not be a viable option - that's for you to decide based on what your application is doing.)

-Scott Ballentine (***@us.ibm.com)
z/OS Device Allocation Development


----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN