Sending z/OS Events to Windows Server

[John Roberts]

I have a need to recognize certain events (creation of a file, completion of a JOB, etc) and send basic information about each event to a Windows process near real-time.

The Windows process would be a RESTful web server deployed on the Intranet alongside the z/OS box.

I have been out of the SYSPROG loop for quite a while, but I seem to recall that the SMF exits are a good place to detect these events. But I know that the SMF exits can't do anything involving a WAIT, so the only thing the exit could do would be to post the information to another address space cross-memory. That address space (STARTED task?) could then act as a HTTPS client and send the data to the outboard server.

I have been reading about the "z/OS Client Web Enablement Toolkit". Would this work for the HTTPS client?

Whatever I create I need it to work in a vanilla z/OS environment V2.1 or after (no CICS etc.).

Does this sound like I am on track? Or is there a better way?

Note: only events for selected files/jobs would be transmitted. Files would need to match a list of patterns, jobs to match a list of USERS. So it would not be a high volume interface.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Charles Mills]

There are a number of commercial products that will send z/OS operational and security events to Linux or Windows. I am, ahem, the principal author of one:
https://correlog.com/mainframe-security-solutions/zdefender-for-zos/

There are a number of competitive products but I'll let you do your own Google searches. <g>

The SMF exits could be a starting point but they are not for the feint-hearted. Cross-memory, locks, SRB, Key 0, that sort of thing. https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.ieaa800/condiscon.htm

A better starting point if you wanted to roll your own is "SMF log streams." More of a problem state, get-a-record sort of interface.

You might also take a look at IBM Common Data Provider.

Charles


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of John Roberts
Sent: Tuesday, October 9, 2018 3:08 PM
To: IBM-***@LISTSERV.UA.EDU
Subject: Sending z/OS Events to Windows Server

I have a need to recognize certain events (creation of a file, completion of a JOB, etc) and send basic information about each event to a Windows process near real-time.

The Windows process would be a RESTful web server deployed on the Intranet alongside the z/OS box.

I have been out of the SYSPROG loop for quite a while, but I seem to recall that the SMF exits are a good place to detect these events. But I know that the SMF exits can't do anything involving a WAIT, so the only thing the exit could do would be to post the information to another address space cross-memory. That address space (STARTED task?) could then act as a HTTPS client and send the data to the outboard server.

I have been reading about the "z/OS Client Web Enablement Toolkit". Would this work for the HTTPS client?

Whatever I create I need it to work in a vanilla z/OS environment V2.1 or after (no CICS etc.).

Does this sound like I am on track? Or is there a better way?

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Rob Schramm]

I think there is someone on the list that already has some SMF 83/84/85
that will capture everything. You would have to modify for your purposes.
I was looking for Unix file creation trigger a couple years ago...should be
in archives.

Rob Schramm

Post by Charles Mills
There are a number of commercial products that will send z/OS operational
and security events to Linux or Windows. I am, ahem, the principal author
https://correlog.com/mainframe-security-solutions/zdefender-for-zos/
There are a number of competitive products but I'll let you do your own
Google searches. <g>
The SMF exits could be a starting point but they are not for the
feint-hearted. Cross-memory, locks, SRB, Key 0, that sort of thing.
https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.ieaa800/condiscon.htm
A better starting point if you wanted to roll your own is "SMF log
streams." More of a problem state, get-a-record sort of interface.
You might also take a look at IBM Common Data Provider.
Charles
-----Original Message-----
Behalf Of John Roberts
Sent: Tuesday, October 9, 2018 3:08 PM
Subject: Sending z/OS Events to Windows Server
I have a need to recognize certain events (creation of a file, completion
of a JOB, etc) and send basic information about each event to a Windows
process near real-time.
The Windows process would be a RESTful web server deployed on the Intranet
alongside the z/OS box.
I have been out of the SYSPROG loop for quite a while, but I seem to
recall that the SMF exits are a good place to detect these events. But I
know that the SMF exits can't do anything involving a WAIT, so the only
thing the exit could do would be to post the information to another address
space cross-memory. That address space (STARTED task?) could then act as a
HTTPS client and send the data to the outboard server.
I have been reading about the "z/OS Client Web Enablement Toolkit". Would
this work for the HTTPS client?
Whatever I create I need it to work in a vanilla z/OS environment V2.1 or
after (no CICS etc.).
Does this sound like I am on track? Or is there a better way?
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

[Timothy Sipples]

If you're looking for a fairly simple, straightforward notification of a
job completion (and results), z/OS 2.3 introduced the ability to send
e-mail notifications when a batch job completes. Here's an article
discussing how it works:

https://www.ibm.com/developerworks/community/blogs/43ea8e78-acbe-49f5-9290-379e4f4569cb/entry/Implementation_of_the_JES2_EMAIL_function_in_z_OS_2_3?lang=en

What's the list of events that you want to capture? Ought it be
bi-directional (when event X happens on Windows Server, notify z/OS)? How
quickly and reliably does the notification need to be sent/received?
(Latency, OK to lose any event notifications?)

--------------------------------------------------------------------------------------------------------
Timothy Sipples
IT Architect Executive, Industry Solutions, IBM Z & LinuxONE,
Multi-Geography
E-Mail: ***@sg.ibm.com

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[John Roberts]

Thanks Charles, Rob, Timothy, and Christopher for some excellent ideas to
research.

My needs are limited to jobstep completion, file creation, and file
deletion. Needs to be near real time, within a few ms of the occurence.

Email notification is ok, but Web service is preferred. One-way - z to
Windows.

I would like to steer clear of requiring prerequisite products, since it
would happen on client installations, not mine. For the same reason, it is
best not to require the latest zOS release.

Re IXGCONN and IXGBRSE, are we certain these can be used to read SMF
logstream records as they are being written?

Post by Timothy Sipples
If you're looking for a fairly simple, straightforward notification of a
job completion (and results), z/OS 2.3 introduced the ability to send
e-mail notifications when a batch job completes. Here's an article
https://www.ibm.com/developerworks/community/blogs/43ea8e78-acbe-49f5-9290-379e4f4569cb/entry/Implementation_of_the_JES2_EMAIL_function_in_z_OS_2_3?lang=en
What's the list of events that you want to capture? Ought it be
bi-directional (when event X happens on Windows Server, notify z/OS)? How
quickly and reliably does the notification need to be sent/received?
(Latency, OK to lose any event notifications?)
--------------------------------------------------------------------------------------------------------
Timothy Sipples
IT Architect Executive, Industry Solutions, IBM Z & LinuxONE,
Multi-Geography
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Charles Mills]

Post by John Roberts
Re IXGCONN and IXGBRSE, are we certain these can be used to read SMF
logstream records as they are being written?

That's what IBM says.

We have had internal philosophical discussions here about what constitutes "real time." There is a competitive solution that runs FTP as often as the customer wants (every five minutes?). Is that "real time"? Our sales folks say No. Well, we delay sending events by tens of microseconds or more. Is that "real time"? If only zero delay is "real time," then no solution is "real time."

But yes, I have not used the above but my impression is that you get SMF records "right away" (whatever that means) as they are produced.

BTW, by "file creation and deletion" do you mean traditional datasets, PDS members, or USS files? They are three different SMF records with radically different formats.

If I make sneak just one more tiny plug in here, the commercial products have already solved these problems and more.

Charles


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of John Roberts
Sent: Thursday, October 11, 2018 9:50 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Sending z/OS Events to Windows Server

Thanks Charles, Rob, Timothy, and Christopher for some excellent ideas to
research.

My needs are limited to jobstep completion, file creation, and file
deletion. Needs to be near real time, within a few ms of the occurence.

Email notification is ok, but Web service is preferred. One-way - z to
Windows.

I would like to steer clear of requiring prerequisite products, since it
would happen on client installations, not mine. For the same reason, it is
best not to require the latest zOS release.

Re IXGCONN and IXGBRSE, are we certain these can be used to read SMF
logstream records as they are being written?

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Ed Jaffe]

Post by Charles Mills
We have had internal philosophical discussions here about what constitutes "real time." There is a competitive solution that runs FTP as often as the customer wants (every five minutes?). Is that "real time"? Our sales folks say No. Well, we delay sending events by tens of microseconds or more. Is that "real time"? If only zero delay is "real time," then no solution is "real time."

In JES3, everything ids event driven. However, in JES2 environments with
multiple systems comprising a JESplex (aka MAS), we have observed delays
in customer sites of  upwards of five seconds before certain events are
recognized by programs accessing JES2's checkpoint data.

IMHO, anything shorter than that should be considered "real" time...
--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
https://www.phoenixsoftware.com/


--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Charles Mills]

Post by Ed Jaffe
anything shorter than that should be considered "real" time...

It's like everything else, right? The way WE do it is perfect. Anything less misses the bar. <g>

Charles


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-***@LISTSERV.UA.EDU] On Behalf Of Ed Jaffe
Sent: Thursday, October 11, 2018 10:20 AM
To: IBM-***@LISTSERV.UA.EDU
Subject: Re: Sending z/OS Events to Windows Server

Post by Ed Jaffe
We have had internal philosophical discussions here about what constitutes "real time." There is a competitive solution that runs FTP as often as the customer wants (every five minutes?). Is that "real time"? Our sales folks say No. Well, we delay sending events by tens of microseconds or more. Is that "real time"? If only zero delay is "real time," then no solution is "real time."

In JES3, everything ids event driven. However, in JES2 environments with
multiple systems comprising a JESplex (aka MAS), we have observed delays
in customer sites of upwards of five seconds before certain events are
recognized by programs accessing JES2's checkpoint data.

IMHO, anything shorter than that should be considered "real" time...
--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
https://www.phoenixsoftware.com/


--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN

[Brian Westerman]

Sorry for the marketing entry, but Syzygy has a product called SyzMPF/z which (among lots of other stuff) will allow you to automatically send email (with the step condition codes and the job related information from the steps, programs ect.) at the end of any job or task (or anywhere during the task) without any JCL changes to any job or task. If you have the DSN monitor turned on, it will do the same thing for DSN creation (or deletes) based on the message(s) generated to syslog. It can be set up to look for specific DSNs or prefixes, or generic names.

You can then have SyzMPF/z send any information (you build it dynamically at the time it happens), including the contents of the file you just identified to anywhere you want via SMTP or SMS text, or FTP. Currently you can only send the contents of Sequential or PDS (or PDS/e) members, but there are plans for the next release (due in November) to send VSAM file contents.

Again, sorry for the marketing response, but I figured it might be what you're looking for.

Brian Westerman

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to ***@listserv.ua.edu with the message: INFO IBM-MAIN